Batch Inference#

A function for inferring given input through a given model while producing a Result Set and performing Data Drift Analysis.

In this notebook we will go over the function’s docs and outputs and see an end-to-end example of running it.

  1. Documentation

  2. Results Prediction

  3. Data Drift Analysis

  4. End-to-end Demo

1. Documentation#

Perform a prediction on a given dataset with the given model. Can perform drift analysis between the sample set statistics stored in the model to the current input data. The drift rule is the value per-feature mean of the TVD and Hellinger scores according to the thresholds configures here.

1.1. Parameters:#

  • context: mlrun.MLClientCtx

    An MLRun context.

  • model: str

    The model Store path, a logged model URI.

  • dataset: Union[mlrun.DataItem, list, dict, pd.DataFrame, pd.Series, np.ndarray]

    The dataset to infer through the model.

    • Can be passed in inputs as either a Dataset artifact / Feature vector URI.

    • Or, in parameters as a list, dictionary or numpy array.

  • drop_columns: Union[str, List[str], int, List[int]] = None

    A string / integer or a list of strings / integers that represent the column names / indices to drop. When the dataset is a list or a numpy array this parameter must be represented by integers.

  • label_columns: Union[str, List[str]] = None

    The target label(s) of the column(s) in the dataset. These names will be used as the column names for the predictions. The label column can be accessed from the model object, or the feature vector provided if available. The default name is "predicted_label_i" for the i column.

  • feature_columns: Union[str, List[str]] = None

    List of feature columns that will be used to build the dataframe when dataset is from type list or numpy array.

  • log_result_set: str = True

    Whether to log the result set - a DataFrame of the given inputs concatenated with the predictions. Defaulted to True.

  • result_set_name: str = "prediction"

    The db key to set name of the prediction result and the filename. Defaulted to "prediction".

  • batch_id: str = None

    The ID of the given batch (inference dataset). If None, it will be generated. Will be logged as a result of the run.

  • perform_drift_analysis: bool = None

    Whether to perform drift analysis between the sample set of the model object to the dataset given. By default, None, which means it will perform drift analysis if the model has a sample set statistics.

  • sample_set: Union[mlrun.DataItem, list, dict, pd.DataFrame, pd.Series, np.ndarray]

    A sample dataset to give to compare the inputs in the drift analysis. The default chosen sample set will always be the one who is set in the model artifact itself.

    • Can be passed in inputs as either a Dataset artifact / Feature vector URI.

    • Or, in parameters as a list, dictionary or numpy array.

  • drift_threshold: float = 0.7

    The threshold of which to mark drifts. Defaulted to 0.7.

  • possible_drift_threshold: float = 0.5

    The threshold of which to mark possible drifts. Defaulted to 0.5.

  • inf_capping: float = 10.0

    The value to set for when it reached infinity. Defaulted to 10.0.

  • artifacts_tag: str = ""

    Tag to use for all the artifacts resulted from the function. Defaulted to no tag.

1.2. Outputs#

The outputs are split to two actions the functions can perform:

  • Results Prediction - Will log:

    • A dataset artifact named by the result_set_name parameter.

    • A str result named "batch_id" of the given / generated batch ID.

  • Data Drift Analysis - Will log:

    • A plotly artifact named "data_drift_table" with a visualization of the drifts results and histograms.

    • A json artifact named "features_drift_results" with all the features metric values.

    • A bool result named "drift_status" of the overall drift status (True if there was a drift and False otherwise).

    • A float result named "drift_score" of the overall drift metric score.

For more details, see the next chapters.

2. Results Prediction#

The result set is a concatenated dataset of the inputs ($X$) provided and the predictions ($Y$) yielded by the model, so it will be $X | Y$.

For example, if the dataset given as inputs was:

x1

x2

x3

x4

x5

…

…

…

…

…

…

…

…

…

…

…

…

…

…

…

And the outputs yielded by the model’s prediction was:

y1

y2

…

…

…

…

…

…

Then the result set will be:

x1

x2

x3

x4

x5

y1

y2

…

…

…

…

…

…

…

…

…

…

…

…

…

…

…

…

…

…

…

…

…

In case the parameter log_result_set is True, the outputs of the results prediction will be:

  • The result set as described above.

  • The batch ID result - batch_id: str - a hashing result that is given by the user or generated randomly in case it was not provided to represent the batch that was being inferred.

    {
    "batch_id": "884a0cb00d8ae16d132dd8259aac29aa78f50a9245d0e4bd58cfbf77",
}

3. Data Drift Analysis#

The data drift analysis is done per feature using two distance measure metrics for probability distributions.

Let us mark our sample set as $S$ and our inputs as $I$. We will look at one feature $x$ out of $n$ features. Assuming the histograms of feature $x$ is split into 20 bins: $b_1,b_2,…,b_{20}$, we will match the feature $x$ histogram of the inputs $I$ ($x_I$) into the same bins (meaning to $x_S$) and compare their distributions using:

  • Total Variance Distance: $TVD(x_S,x_I) = \frac{1}{2}\sum_{b_1}^{b_{20}} {|x_S - x_I|}$

  • Hellinger Distance: $H(x_S,x_I) = \sqrt{1-{\sum_{b_1}^{b_{20}}\sqrt{x_S \cdot x_I}}}$

Our rule then is calculating for each $x\in S: \frac{H(x_S,x_I)+TVD(x_S,x_I)}{2} < $ given thresholds.

In case the parameter perform_drift_analysis is True, the outputs of the analysis will be:

  • Drift table plot - The results are presented in a plotly table artifact named "drift_table_plot" that shows each feature’s statistics and its TVD, Hellinger and KLD (Kullback–Leibler divergence) results as follows:

Count

Mean

Std

Min

Max

Tvd

Hellinger

Kld

Histograms

Sample

Input

Sample

Input

Sample

Input

Sample

Input

Sample

Input

x1

…

…

…

…

…

…

…

…

…

…

…

…

…

…

x2

…

…

…

…

…

…

…

…

…

…

…

…

…

…

x3

…

…

…

…

…

…

…

…

…

…

…

…

…

…

  • Features drift results - A rule metric per feature dictionary is saved in a json file named "features_drift_results" where each key is a feature and its value is the feature’s metric value: Dict[str, float]

    {
    "x1": 0.12,
    "x2": 0.345,
    "x3": 0.00678,
    ...
}

  • In addition, two results are being added to summarize the drift analysis:

    • drift_status: bool - A boolean value indicating whether a drift was found.

    • drift_metric: float - The mean of all the features drift metric value (the rule above): for $n$ features and metric rule $M(x_S,x_I)=\frac{H(x_S,x_I)+TVD(x_S,x_I)}{2}$, drift_metric $=\frac{1}{n}\sum_{x\in S}M(x_S,x_I)$

    {
    "drift_status": True,
    "drift_metric": 0.81234
}

4. End-to-end Demo#

We will see an end-to-end example that follows the steps below:

  1. Generate data.

  2. Train a model.

  3. Infer data through the model using batch_predict and review the outputs.

4.1. Code review#

We are using a very simple example of training a decision tree on a binary classification problem. For that we wrote two functions:

  • generate_data - Generate a binary classification data. The data will be split into a training set and data for prediction. The data for prediction will be drifted in half of its features to showcase the plot later on.

  • train - Train a decision tree classifier on a given data.

# mlrun: start-code

# upload environment variables from env file if exists
import os,mlrun

# Specify path
path = "/tmp/examples_ci.env"

if os.path.exists(path):
    env_dict = mlrun.set_env_from_file(path, return_dict=True)

import numpy as np
import pandas as pd

from sklearn.datasets import make_classification
from sklearn.tree import DecisionTreeClassifier

import mlrun
from mlrun.frameworks.sklearn import apply_mlrun


@mlrun.handler(outputs=["training_set", "prediction_set"])
def generate_data(n_samples: int = 5000, n_features: int = 20):
    # Generate a classification data:
    x, y = make_classification(
        n_samples=n_samples, n_features=n_features, n_classes=2
    )

    # Split the data into a training set and a prediction set:
    x_train, x_prediction = x[: n_samples // 2], x[n_samples // 2 :]
    y_train = y[: n_samples // 2]
    
    # Randomly drift some features:
    x_prediction += (
        np.random.uniform(low=2, high=4, size=x_train.shape) * 
        np.random.randint(low=0, high=2, size=x_train.shape[1], dtype=int)
    )
    
    # Initialize dataframes:
    features = [f"feature_{i}" for i in range(n_features)]
    training_set = pd.DataFrame(data=x_train, columns=features)
    training_set.insert(
        loc=n_features, column="label", value=y_train, allow_duplicates=True
    )
    prediction_set = pd.DataFrame(data=x_prediction, columns=features)

    return training_set, prediction_set


@mlrun.handler()
def train(training_set: pd.DataFrame):
    # Get the data into x, y:
    labels = pd.DataFrame(training_set["label"])
    training_set.drop(columns=["label"], inplace=True)

    # Initialize a model:
    model = DecisionTreeClassifier()

    # Apply MLRun:
    apply_mlrun(model=model, model_name="model")

    # Train:
    model.fit(training_set, labels)

# mlrun: end-code

4.2. Run the Example with MLRun#

First, we will prepare our MLRun functions:

  1. We will use mlrun.code_to_function to turn this demo notebook into an MLRun function we can run.

  2. We will use mlrun.import_function to import the batch_predict function .

# Create an MLRun function to run the notebook:
demo_function = mlrun.code_to_function(name="batch_inference_demo", kind="job")

# Import the `batch_predict` function from the marketplace:
batch_inference_function = mlrun.import_function("hub://batch_inference")

# Set the desired artifact path:
artifact_path = "./"

Now, we will follow the demo steps as discussed above:

# 1. Generate data:
generate_data_run = demo_function.run(
    handler="generate_data",
    artifact_path=artifact_path,
    local=True,
)

# 2. Train a model:
train_run = demo_function.run(
    handler="train",
    artifact_path=artifact_path,
    inputs={"training_set": generate_data_run.outputs["training_set"]},
    local=True,
)

# 3. Perform batch prediction:
batch_inference_run = batch_inference_function.run(
    handler="infer",
    artifact_path=artifact_path,
    inputs={"dataset": generate_data_run.outputs["prediction_set"]},
    params={
        "model": train_run.outputs["model"],
        "label_columns": "label",
    },
    local=True,
)

> 2022-09-13 09:54:59,693 [warning] artifact path is not defined or is local, artifacts will not be visible in the UI
> 2022-09-13 09:54:59,694 [info] starting run batch-predict-demo-generate_data uid=a5b1ca0a37d946e892b9305b9af833c3 DB=http://mlrun-api:8080
project uid iter start state name labels inputs parameters results artifacts
default 0 Sep 13 09:54:59 completed batch-predict-demo-generate_data
v3io_user=guyl
kind=
owner=guyl
host=jupyter-guyl-66857b7999-ffvsx
training_set
prediction_set
> to track results use the .show() or .logs() methods or click here to open in UI
> 2022-09-13 09:55:06,462 [info] run executed, status=completed
> 2022-09-13 09:55:06,464 [warning] artifact path is not defined or is local, artifacts will not be visible in the UI
> 2022-09-13 09:55:06,464 [info] starting run batch-predict-demo-train uid=384b36e84c4e4f91900e49e1f24ff1a6 DB=http://mlrun-api:8080
project uid iter start state name labels inputs parameters results artifacts
default 0 Sep 13 09:55:06 completed batch-predict-demo-train
v3io_user=guyl
kind=
owner=guyl
host=jupyter-guyl-66857b7999-ffvsx
training_set
model
> to track results use the .show() or .logs() methods or click here to open in UI
> 2022-09-13 09:55:07,367 [info] run executed, status=completed
> 2022-09-13 09:55:07,370 [warning] artifact path is not defined or is local, artifacts will not be visible in the UI
> 2022-09-13 09:55:07,370 [info] starting run batch-predict-predict uid=cf88e39d59704912a5ee41ceb539cd05 DB=http://mlrun-api:8080
> 2022-09-13 09:55:07,703 [info] Loading model...'
> 2022-09-13 09:55:07,753 [info] Calculating prediction...
> 2022-09-13 09:55:07,757 [info] Logging result set (x | prediction)...
> 2022-09-13 09:55:07,952 [info] Performing drift analysis...

divide by zero encountered in log
project uid iter start state name labels inputs parameters results artifacts
default 0 Sep 13 09:55:07 completed batch-predict-predict
v3io_user=guyl
kind=
owner=guyl
host=jupyter-guyl-66857b7999-ffvsx
dataset
model=store://artifacts/default/model:384b36e84c4e4f91900e49e1f24ff1a6
label_columns=label
drift_status=False
drift_metric=0.3880999515903545
prediction
drift_table_plot
features_drift_results
> to track results use the .show() or .logs() methods or click here to open in UI
> 2022-09-13 09:55:10,078 [info] run executed, status=completed

4.3. Review Outputs#

We will review the outputs as explained in the notebook above.

4.3.1. Results Prediction#

First we will showcase the Result Set. As we didn’t send any name, it’s default name will be "prediction":

batch_inference_run.artifact("prediction").as_df()
feature_0 feature_1 feature_2 feature_3 feature_4 feature_5 feature_6 feature_7 feature_8 feature_9 ... feature_11 feature_12 feature_13 feature_14 feature_15 feature_16 feature_17 feature_18 feature_19 label
0 6.319111 3.208210 0.793499 -0.613252 2.634766 1.208352 4.718735 3.557495 -2.116311 0.370145 ... 1.962901 0.581321 4.844532 1.408737 0.964987 2.111456 3.134610 -1.727937 0.335741 0
1 3.138378 1.633661 0.144557 0.687003 2.404279 1.405990 3.235892 1.804426 2.019980 1.719908 ... -1.486809 0.648228 3.039797 1.806766 1.201692 2.475974 1.448559 0.959266 1.158651 1
2 2.353026 4.642879 0.952097 0.605977 4.051640 -0.157584 1.218743 2.464738 1.706084 -0.250366 ... 0.559968 0.979378 2.411703 3.746830 2.252155 3.406102 3.263166 -0.236510 -0.313161 1
3 1.617202 4.568332 2.937961 2.501166 3.952541 0.671749 3.774594 4.042543 -2.173079 -0.983443 ... -0.839010 0.953698 3.033551 1.006891 2.398563 5.047382 5.291260 1.305584 0.843951 1
4 3.344291 4.538357 1.032059 -0.047931 3.118438 0.403812 4.472615 1.840558 -0.714775 0.287726 ... 1.598060 -0.805508 4.742032 4.608792 1.617717 4.514895 3.648923 -1.344024 0.610534 0
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
2495 2.319301 2.996941 1.337934 0.805649 2.303656 0.203069 5.575559 3.437790 0.709777 0.392013 ... -0.114619 -1.469797 4.538126 1.282498 5.686133 2.826973 2.445658 -0.145780 0.337803 0
2496 2.920678 2.144983 2.153517 -0.527295 2.612040 1.113704 2.438761 3.284425 1.093894 0.921599 ... -1.586852 0.409838 4.094763 2.636654 3.333414 3.251106 1.132976 1.072658 -1.240186 1
2497 4.256698 2.135673 -0.114491 0.329980 3.935633 -0.777958 2.543643 2.195111 -0.926822 -0.251254 ... -0.952889 0.687820 2.268043 5.077454 2.248259 3.469704 2.262900 0.687038 -0.614066 1
2498 4.738030 2.390842 -0.972329 1.471461 2.904280 -2.079088 2.570604 2.325262 -1.602976 -0.806244 ... 0.554399 0.027493 4.145728 3.782802 4.202006 3.272709 0.867462 -1.020029 2.013301 0
2499 2.047831 2.153813 0.392484 0.249010 3.846910 0.300846 3.005997 2.799457 -0.304962 -0.990622 ... -0.263473 0.110091 2.995411 2.582843 4.599535 3.219091 1.592652 -0.074851 -0.617769 1

2500 rows × 21 columns

4.3.2. Data Drift Analysis#

Second we will review the data drift table plot and the drift results:

batch_inference_run.artifact("drift_table_plot").show()

batch_inference_run.status.results

{'drift_status': False, 'drift_metric': 0.3880999515903545}