Prevent inaccurate predictions with Humble AI

A guide on how to write Humble AI rules to override uncertain predictions with DataRobot MLOps

ML Models aren't always right.

A model is only as good as the degree to which the data used for training aligns with the data used for predictions. If the predicted data is substantially different to what the model has been trained on, then there's going to be data drift (see Measuring Data Drift).

By default, a model will make a prediction regardless of how certain it is about it. This may not be what you want in your application; in fact, you might only want to consider the prediction when you're absolutely certain of its quality.

You can change this behavior with MLOps by tweaking how your models behave outside the expected parameters. The ability to customize model behavior is part of the DataRobot Humility feature.

Understanding uncertain predictions

There are some obvious indicators of what an uncertain prediction means.

Let's take the simplest option as an example: binary classification. Here the predicted value can be either 0 or 1. If you think of a prediction in terms of probability it will always fall somewhere between 0 and 1, and the predicted value will be whichever value it's closest to. This also means that the closer a value is to 0.5 (i.e., the halfway point), the less certain the prediction.

Setting humility rules

Humility settings let you apply various types of rules for predictions and inputs.

To set one, head to the Humility tab of your deployment and select Enable humility.
Then, explore what options you have available for your model; note that options available are dependent on the dataset.

In the example below, we are setting a humility rule for a binary classification problem from the Movie ratings—sentiment guide.

Throw an error when predicted value is not certainThrow an error when predicted value is not certain

Throw an error when predicted value is not certain

Handling uncertain predictions

Let's say we set a humility rule in our Auto MPG example from the Quickstart Guide for outlying model years. We didn't have many cars older than 1961 and newer than 1990, so we only want to accept cars that were made within that time frame.

Outlying input trigger setOutlying input trigger set

Outlying input trigger set

After we add and submit the rule, we can make a new prediction request, and pass in an outlying input (one that falls outside the desired values/time frame), to see the error in action:

import requests

DATAROBOT_API_URL='https://app2.datarobot.com/api/v2' # Replace with your endpoint if different
DEPLOYMENT_ID='YOUR_DEPLOYMENT_ID' # ID of your Auto MPG model deployment
API_KEY = 'YOUR_API_KEY'

payload = [
    {
        "cylinders": 4,
        "displacement": 119.0,
        "horsepower": 82.00,
        "weight": 2720.0,
        "acceleration": 19.4,
        "model year": 99,
        "origin": 1,
    }
]

prediction_response = requests.post(
    f"{DATAROBOT_API_URL}/deployments/{DEPLOYMENT_ID}/predictions/",
    headers = {
        'Authorization': f"Bearer {YOUR_API_KEY}"
    },
    json = payload,
)

print(prediction_response.status_code) 
# 480
print(prediction_response.json())
# {'message': 'Humility ReturnError action triggered.'}

As set in the Humility tab, the new rule will trigger when the input isn't in valid bounds, and the API will return a 480 status code with a humility error message.

If we were to set a humility rule that would instead override the response, then no error would be triggered and instead we'd just handle the response as any other prediction.

To summarize, humility rules are a great way to add some custom logic to models and elegantly handle situations where a model might not be making accurate predictions. This lets you architect more resilient ML-powered applications.


Did this page help you?