Dog Breed Categorization

Q: Can you use DataRobot to determine dog breeds? Based on their images?
A: Yes. Read below.

This article is intended as an overview of the two more interesting aspects of this application - DataRobot Visual AI, which lets you combine image and tabular data, and classifying large number of categories.

This guide assumes some familiarity with DataRobot APIs. If you are not familiar with the basics of DataRobot and it's APIs, we recommend you check out the Quickstart guide.

The app

You can check out the app at the following link: For best results, open it on a smartphone.
If you have your phone handy, you can also scan the QR code below, which will take you to the app 👇

You can also check out the source code on GitHub:

Project overview

This is a mobile friendly app that lets you classify dog breeds, based on their photo and some descriptive data you provide! It shows you 5 most similar breeds.

The app is a NextJS app, deployed on Vercel.
It is written in TypeScript, which is a typed subset of JavaScript. If you can read JavaScript, you can read TypeScript as well. :smiley:

Preparing data 🐶

In this example we have already prepared the data in advance.
To get the training dataset of dogs we gathered the information about dog breeds from AKC - American Kennel Club, and the dog images are taken from Wikipedia.

You can find the script used to obtain and prepare data in the GitHub repository - scripts/

Dog data

We are interested in the following attributes of dogs, in order to categorize them in breeds:

  • Photo
  • Height
  • Weight
  • Behavioral traits

The photo is self explanatory - for this we use Visual AI to automatically determine the "interesting" parts of the photograph.
Height and weight are initially numerical values, in inches/centimeters and pounds/kilograms. We created categories out of these as well - from x-small to x-large, in order to make estimating easier for users.
Behavioral traits are a set of behaviors that can describe a dog. A labrador retriever for example will be friendly, and a doberman might be protective. We are observing 6 traits: friendly, playful, protective, fast learner, athletic, and graceful.

Visual AI datasets 🖼

DataRobot supports multi-modal datasets. In our case, the dataset will consist of an image, dog size, weight, and traits that describe its behavior.

To define an image feature, you need to pass its relative path in the dataset.
The dataset you upload needs to be in a compressed file format, which contains a CSV and an image.

For example:


The entire folder structure looks like in the image below 👇

Directory structure for Visual AIDirectory structure for Visual AI

Directory structure for Visual AI

Other than you need to compress the folder containing both the images and tabular data, the upload and project creation process works exactly the same.
If you are interested in using the API for this, check out either the Quickstart or the Finance Finder guides.

Predicting on images


Note: this is currently an undocumented behaviour. The image upload API might change in the future. If this happens, we will update this guide accordingly.

If an image comes in via browser APIs, you get a DataURI returned, and the image will already be Base64 encoded, which will let us encode an image into a text format, easy to transmit over the network in a POST request.

In this case, the API call will pass in the image data as image_path just like this in a regular POST request payload:

let predictionPayload: Payload = {
    height: req.body.height,
    image_path: BASE_64_ENCODED_IMAGE,
    is_friendly: req.body.is_friendly || false,
    is_playful: req.body.is_playful || false,
    is_protective: req.body.is_protective || false,
    is_fast_learner: req.body.is_fast_learner || false,
    is_athletic: req.body.is_athletic || false,
    is_graceful: req.body.is_graceful || false
await, {
          headers: {
            "Authorization": `Bearer ${API_KEY}`,
            "datarobot-key": DATAROBOT_KEY
          json: [predictionPayload],
          responseType: 'json'

The entire DataRobot API integration code is in the file pages/api/payload.ts.

Predicting many categories 💯

DataRobot lets you perform predictions on up to 100 classes. That is that there are 100 unique possible values that it can predict in a model.

With dog breeds however, there are between 250 and 300 recognized dog breeds out there (depending on who you ask). So the question is - how do we go about that?

In our case we needed to split the dataset into multiple datasets, and decide in our app which one to call / require. One approach to this is splitting models in a different feature-category.
As dog breeds tend to be standardized in their size and weight, we can use one of these features to split. In our case, we used weight, to split datasets into 5 files, which are available in 5 deployments:

const deployments = {
  x_small: process.env.DEPLOYMENT_00_XSMALL || "",
  small: process.env.DEPLOYMENT_01_SMALL || "",
  medium: process.env.DEPLOYMENT_02_MEDIUM || "",
  large: process.env.DEPLOYMENT_03_LARGE || "",
  x_large: process.env.DEPLOYMENT_04_XLARGE || "",

Once we have 5 deployments, we could then trigger requests just to the deployment that matches our dog's weight estimate. To introduce a bit more tolerance to the system we are actually making requests to the selected and the adjacent sizes. For example, if small is selected, we are also interested in x-small and medium. See the example below:

const adjacentDeploymentsFromWeight: (weight: string) => string[] = (weight) => {
    case 'x-small': return [deployments.x_small, deployments.small]
    case 'small': return [deployments.x_small, deployments.small, deployments.medium ]
    case 'medium': return [deployments.small, deployments.medium, deployments.large ]
    case 'large': return [deployments.medium, deployments.large, deployments.x_large ]
    case 'x-large': return [deployments.large, deployments.x_large ]
    default: return [] //This is unreachable

After we get all the responses back from DataRobot prediction API, we just need to sort them by prediction probability (value).

    let allPredictions: PredictedBreed[] = []
    let selectedDeployments = adjacentDeploymentsFromWeight(weight)

    let predictionResponses = selectedDeployments
      .map(async deploymentId => 
        await makeRequest() 

      for (const predictionResponse of predictionResponses) {        
        let predictionValues = (await predictionResponse)[0].predictionValues

      // Sort all results by prediction probability
      allPredictions.sort( (first, second) => second.value - first.value)

This lets us effectively make predictions a large number of categories.


In this article we have reviewed a fun little project that classifies dog breeds based on their weight, height, personality traits, as well as their images. DataRobot makes this easy and straightforward.
You learned how to use datasets that combine multiple types of data, and how to overcome the 100 category limit in DataRobot multiclass categorization, to determine dog breeds.

In the repository you can explore the datasets and train the models yourself. You will also find the steps to run and deploy the application.

Did this page help you?