Batch Inference V2

A function for inferring given input through a given model while producing a Result Set and applying monitoring analysis.

In this notebook we will go over the function’s docs and outputs and see an end-to-end example of running it.

1. Documentation

2. Results Prediction

3. End-to-end Demo

4. Data Drift Analysis

1. Documentation

Perform a prediction on a given dataset with the given model. Using the default histogram data drift application, can perform drift analysis between the sample set statistics stored in the model to the current input data. The drift rule in this case is the value per-feature mean of the TVD and Hellinger scores. When §, this function either creates or update an existing model endpoint record (depends on the provided endpoint_id).

At the moment, this function is supported for mlrun>=1.5.0 versions.

1.1. Parameters:

1.1.1 Batch Infer Parameters:

• context: mlrun.MLClientCtx

  An MLRun context.

• dataset: Union[mlrun.DataItem, list, dict, pd.DataFrame, pd.Series, np.ndarray]

  The dataset to infer through the model.

  • Can be passed in inputs as a dataset artifact / Feature vector URI.

  • Or, in parameters as a list, dictionary or numpy array.

• model_path: Union[str, mlrun.DataItem]

  Model store uri (should start with store://). Provided as an input (DataItem). To generate a valid model store URI, please log the model before running this function. If endpoint_id of existing model endpoint is provided, make sure that it has a similar model store path, otherwise the drift analysis won’t be triggered.

• drop_columns: Union[str, List[str], int, List[int]] = None

  A string / integer or a list of strings / integers that represent the column names / indices to drop. When the dataset is a list or a numpy array this parameter must be represented by integers.

• label_columns: Union[str, List[str]] = None

  The target label(s) of the column(s) in the dataset. These names will be used as the column names for the predictions. The label column can be accessed from the model object, or the feature vector provided if available. The default name is "predicted_label_i" for the i column.

• feature_columns: Union[str, List[str]] = None

  List of feature columns that will be used to build the dataframe when dataset is from type list or numpy array.

• log_result_set: str = True

  Whether to log the result set - a DataFrame of the given inputs concatenated with the predictions. Defaulted to True.

• result_set_name: str = "prediction"

  The db key to set name of the prediction result and the filename. Defaulted to "prediction".

• batch_id: str = None

  The ID of the given batch (inference dataset). If None, it will be generated. Will be logged as a result of the run.

• artifacts_tag: str = ""

  Tag to use for all the artifacts resulted from the function. Defaulted to no tag.

1.1.2 Drift Analysis Parameters:

• perform_drift_analysis: bool = None

  Whether to perform drift analysis between the sample set of the model object to the dataset given. By default, None, which means it will perform drift analysis if the model already has feature stats that are considered as a reference sample set. Performing drift analysis on a new endpoint id will generate a new model endpoint record.

• endpoint_id: str = ""

  Model endpoint unique ID. If perform_drift_analysis was set, the endpoint_id will be used either to perform the analysis on existing model endpoint or to generate a new model endpoint record.

1.1.3 New Model Endpoint Parameters:

• model_endpoint_name: str = "batch-infer"

  If a new model endpoint is generated, the model name will be presented under this endpoint.

• model_endpoint_sample_set: Union[mlrun.DataItem, list, dict, pd.DataFrame, pd.Series, np.ndarray] = None

  A sample dataset to give to compare the inputs in the drift analysis. Can be provided as an input or as a parameter. The default chosen sample set will always be the one who is set in the model artifact itself.

1.2. Outputs

The outputs are split to two actions the functions can perform:

• Results Prediction - Will log:

  • A dataset artifact named by the result_set_name parameter.

  • A str result named "batch_id" of the given / generated batch ID.

• Data Drift Analysis - Will log:

  • A plotly artifact named "data_drift_table" with a visualization of the drifts results and histograms.

  • A json artifact named "features_drift_results" with all the features metric values.

For more details, see the next chapters.

2. Results Prediction

The result set is a concatenated dataset of the inputs ($X$) provided and the predictions ($Y$) yielded by the model, so it will be $X | Y$.

For example, if the dataset given as inputs was:

x1	x2	x3	x4	x5
…	…	…	…	…
…	…	…	…	…
…	…	…	…	…

And the outputs yielded by the model’s prediction was:

y1	y2
…	…
…	…
…	…

Then the result set will be:

x1	x2	x3	x4	x5	y1	y2
…	…	…	…	…	…	…
…	…	…	…	…	…	…
…	…	…	…	…	…	…

In case the parameter log_result_set is True, the outputs of the results prediction will be:

• The result set as described above.

• The batch ID result - batch_id: str - a hashing result that is given by the user or generated randomly in case it was not provided to represent the batch that was being inferred.

  {
      "batch_id": "884a0cb00d8ae16d132dd8259aac29aa78f50a9245d0e4bd58cfbf77",
  }
  

3. End-to-end Demo

We will see an end-to-end example that follows the steps below:

1. Generate data.

2. Train a model.

3. Infer data through the model using batch_predict and review the outputs.

3.1. Code review

We are using a very simple example of training a decision tree on a binary classification problem. For that we wrote two functions:

• generate_data - Generate a binary classification data. The data will be split into a training set and data for prediction. The data for prediction will be drifted in half of its features to showcase the plot later on.

• train - Train a decision tree classifier on a given data.

import mlrun

# Create MLRun project
project_name = "batch-infer-demo"
project = mlrun.get_or_create_project(project_name, context="./")

> 2024-10-08 10:23:13,060 [info] Loading project from path: {"path":"./","project_name":"batch-infer-demo","user_project":false}
> 2024-10-08 10:23:28,490 [info] Project loaded successfully: {"path":"./","project_name":"batch-infer-demo","stored_in_db":true}

# mlrun: start-code

# upload environment variables from env file if exists
import os

# Specify path
path = "/tmp/examples_ci.env"

if os.path.exists(path):
    env_dict = mlrun.set_env_from_file(path, return_dict=True)

import numpy as np
import pandas as pd

from sklearn.datasets import make_classification
from sklearn.tree import DecisionTreeClassifier

from mlrun.frameworks.sklearn import apply_mlrun


def generate_data(n_samples: int = 5000, n_features: int = 20):
    # Generate a classification data:
    x, y = make_classification(
        n_samples=n_samples, n_features=n_features, n_classes=2
    )

    # Split the data into a training set and a prediction set:
    x_train, x_prediction = x[: n_samples // 2], x[n_samples // 2 :]
    y_train = y[: n_samples // 2]
    
    # Randomly drift some features:
    x_prediction += (
        np.random.uniform(low=2, high=4, size=x_train.shape) * 
        np.random.randint(low=0, high=2, size=x_train.shape[1], dtype=int)
    )
    
    # Initialize dataframes:
    features = [f"feature_{i}" for i in range(n_features)]
    training_set = pd.DataFrame(data=x_train, columns=features)
    training_set.insert(
        loc=n_features, column="label", value=y_train, allow_duplicates=True
    )
    prediction_set = pd.DataFrame(data=x_prediction, columns=features)

    return training_set, prediction_set


def train(training_set: pd.DataFrame):
    # Get the data into x, y:
    labels = pd.DataFrame(training_set["label"])
    training_set.drop(columns=["label"], inplace=True)

    # Initialize a model:
    model = DecisionTreeClassifier()

    # Apply MLRun:
    apply_mlrun(model=model, model_name="model")

    # Train:
    model.fit(training_set, labels)

# mlrun: end-code

3.2. Run the Example with MLRun

First, we will prepare our MLRun functions:

1. We will use mlrun.code_to_function to turn this demo notebook into an MLRun function we can run.

2. We will use mlrun.import_function to import the batch_predict function .

# Create an MLRun function to run the notebook:
demo_function = mlrun.code_to_function(name="batch-inference-demo", kind="job")

# Import the `batch_inference_v2` function from the functions hub:
batch_inference_function = mlrun.import_function('hub://batch_inference_v2')
# you can import the function from the current directory as well: 
# batch_inference_function = mlrun.import_function("function.yaml")

Now, we will follow the demo steps as discussed above:

# 1. Generate data:
generate_data_run = demo_function.run(
    handler="generate_data",
    returns=["training_set : dataset", "prediction_set : dataset"],
)

# 2. Train a model:
train_run = demo_function.run(
    handler="train",
    inputs={"training_set": generate_data_run.outputs["training_set"]},
)

# 3. Perform batch prediction:
batch_inference_run = batch_inference_function.run(
    handler="infer",
    inputs={"dataset": generate_data_run.outputs["prediction_set"]},
    params={
        "model_path": train_run.outputs["model"],
        "label_columns": "label",
        "perform_drift_analysis": False,
    },
)

> 2024-10-08 10:23:40,584 [error] error getting build status: details: MLRunNotFoundError('Function tag not found batch-infer-demo/batch-inference-demo'), caused by: 404 Client Error: Not Found for url: http://mlrun-api:8080/api/v1/build/status?name=batch-inference-demo&project=batch-infer-demo&tag=&logs=no&offset=0&last_log_timestamp=0.0&verbose=no
> 2024-10-08 10:23:40,586 [info] Storing function: {"db":"http://mlrun-api:8080","name":"batch-inference-demo-generate-data","uid":"4f68ba3fd9084e3e941ab3872ceb3635"}
> 2024-10-08 10:23:40,881 [info] Job is running in the background, pod: batch-inference-demo-generate-data-52w8s
> 2024-10-08 10:23:46,954 [info] To track results use the CLI: {"info_cmd":"mlrun get run 4f68ba3fd9084e3e941ab3872ceb3635 -p batch-infer-demo","logs_cmd":"mlrun logs 4f68ba3fd9084e3e941ab3872ceb3635 -p batch-infer-demo"}
> 2024-10-08 10:23:46,954 [info] Or click for UI: {"ui_url":"https://dashboard.default-tenant.app.vmdev57.lab.iguazeng.com/mlprojects/batch-infer-demo/jobs/monitor/4f68ba3fd9084e3e941ab3872ceb3635/overview"}
> 2024-10-08 10:23:46,955 [info] Run execution finished: {"name":"batch-inference-demo-generate-data","status":"completed"}

project	uid	iter	start	state	kind	name	labels	inputs	parameters	results	artifacts
batch-infer-demo	...2ceb3635	0	Oct 08 10:23:44	completed	run	batch-inference-demo-generate-data	v3io_user=eyald kind=job owner=eyald mlrun/client_version=1.7.0-rc51 mlrun/client_python_version=3.9.18 host=batch-inference-demo-generate-data-52w8s				training_set prediction_set

Title ×

> to track results use the .show() or .logs() methods or click here to open in UI

> 2024-10-08 10:23:52,384 [info] Run execution finished: {"name":"batch-inference-demo-generate-data","status":"completed"}
> 2024-10-08 10:23:52,434 [info] Storing function: {"db":"http://mlrun-api:8080","name":"batch-inference-demo-train","uid":"1102e4aea9424149837a81aa214b9489"}
> 2024-10-08 10:23:52,714 [info] Job is running in the background, pod: batch-inference-demo-train-ldclf
> 2024-10-08 10:23:58,683 [info] To track results use the CLI: {"info_cmd":"mlrun get run 1102e4aea9424149837a81aa214b9489 -p batch-infer-demo","logs_cmd":"mlrun logs 1102e4aea9424149837a81aa214b9489 -p batch-infer-demo"}
> 2024-10-08 10:23:58,684 [info] Or click for UI: {"ui_url":"https://dashboard.default-tenant.app.vmdev57.lab.iguazeng.com/mlprojects/batch-infer-demo/jobs/monitor/1102e4aea9424149837a81aa214b9489/overview"}
> 2024-10-08 10:23:58,684 [info] Run execution finished: {"name":"batch-inference-demo-train","status":"completed"}

project	uid	iter	start	state	kind	name	labels	inputs	parameters	results	artifacts
batch-infer-demo	...214b9489	0	Oct 08 10:23:56	completed	run	batch-inference-demo-train	v3io_user=eyald kind=job owner=eyald mlrun/client_version=1.7.0-rc51 mlrun/client_python_version=3.9.18 host=batch-inference-demo-train-ldclf	training_set			model

Title ×

> to track results use the .show() or .logs() methods or click here to open in UI

> 2024-10-08 10:24:01,923 [info] Run execution finished: {"name":"batch-inference-demo-train","status":"completed"}
> 2024-10-08 10:24:01,953 [info] Storing function: {"db":"http://mlrun-api:8080","name":"batch-inference-v2-infer","uid":"27dd151299ff42bc9301ed268bab8b5b"}
> 2024-10-08 10:24:02,241 [info] Job is running in the background, pod: batch-inference-v2-infer-lt4w9
> 2024-10-08 10:24:06,348 [info] Loading model...
> 2024-10-08 10:24:07,569 [info] Loading data...
> 2024-10-08 10:24:07,631 [info] Calculating prediction...
> 2024-10-08 10:24:07,636 [info] Logging result set (x | prediction)...
> 2024-10-08 10:24:08,468 [info] To track results use the CLI: {"info_cmd":"mlrun get run 27dd151299ff42bc9301ed268bab8b5b -p batch-infer-demo","logs_cmd":"mlrun logs 27dd151299ff42bc9301ed268bab8b5b -p batch-infer-demo"}
> 2024-10-08 10:24:08,469 [info] Or click for UI: {"ui_url":"https://dashboard.default-tenant.app.vmdev57.lab.iguazeng.com/mlprojects/batch-infer-demo/jobs/monitor/27dd151299ff42bc9301ed268bab8b5b/overview"}
> 2024-10-08 10:24:08,469 [info] Run execution finished: {"name":"batch-inference-v2-infer","status":"completed"}

project	uid	iter	start	state	kind	name	labels	inputs	parameters	results	artifacts
batch-infer-demo	...8bab8b5b	0	Oct 08 10:24:06	completed	run	batch-inference-v2-infer	v3io_user=eyald kind=job owner=eyald mlrun/client_version=1.7.0-rc51 mlrun/client_python_version=3.9.18 host=batch-inference-v2-infer-lt4w9	dataset	model_path=store://models/batch-infer-demo/model:latest@1102e4aea9424149837a81aa214b9489 label_columns=label perform_drift_analysis=False	batch_id=bdf589be041d04464671f41842278989c9f1ca72dcd5e3ed5cdf5495	prediction

Title ×

> to track results use the .show() or .logs() methods or click here to open in UI

> 2024-10-08 10:24:12,530 [info] Run execution finished: {"name":"batch-inference-v2-infer","status":"completed"}

3.3. Review Outputs

We will review the outputs as explained in the notebook above.

3.3.1. Results Prediction

First we will showcase the Result Set. As we didn’t send any name, it’s default name will be "prediction":

batch_inference_run.artifact("prediction").as_df()

	feature_0	feature_1	feature_2	feature_3	feature_4	feature_5	feature_6	feature_7	feature_8	feature_9	...	feature_11	feature_12	feature_13	feature_14	feature_15	feature_16	feature_17	feature_18	feature_19	label
0	6.301344	0.270723	0.810890	1.916672	1.480180	-1.111209	3.623056	2.151706	-0.581654	0.469363	...	1.162767	1.631004	-0.382130	1.307247	-0.851273	3.321264	0.697387	2.381824	-0.070941	1
1	1.482197	-1.242443	0.837535	2.904502	-1.605309	0.632422	2.955928	-0.693749	-0.326387	-0.897179	...	4.941789	4.033131	-0.009700	-0.585573	-1.503230	1.074927	0.803923	3.804727	-0.028967	0
2	3.493216	0.731723	-2.769300	1.533892	-1.341591	2.544158	2.855936	0.826364	-1.093561	0.303124	...	0.848919	1.783395	-0.644753	1.994629	2.166190	4.072446	2.121466	3.146668	1.574255	1
3	1.243322	-1.185999	1.510705	2.280017	-0.139123	-1.333367	4.703854	-1.238155	0.659352	-0.514514	...	2.731269	3.631489	0.175608	0.754273	-0.783163	4.356327	0.350378	3.347941	-1.111401	1
4	3.368511	0.294880	1.098486	2.401271	1.178792	0.050650	1.631508	0.915017	0.002918	-0.217509	...	3.671178	2.873755	0.352395	-1.290841	-0.773709	1.938640	0.776364	3.676751	0.867656	1
...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
2495	2.006662	-0.377155	-0.674967	3.425157	0.774268	-0.288493	3.312510	-0.935283	0.898680	1.028747	...	1.511337	2.026637	-1.371283	-0.162678	2.033294	0.387519	-1.778998	4.347185	0.066542	0
2496	4.624801	-0.435007	-1.340297	3.709764	1.303919	-0.913823	2.927538	1.808688	-0.461729	-0.309318	...	4.194667	0.308783	-0.455863	0.333040	0.302472	0.571399	-0.664472	3.399752	1.068366	0
2497	5.299738	1.273677	-1.801943	3.892334	1.255739	0.095366	2.051728	1.577475	-0.348716	0.714247	...	2.760161	4.798985	0.862472	-1.737803	1.263144	3.620031	-0.426575	3.938336	-0.700078	1
2498	0.866703	-1.056071	1.670582	2.334009	-1.333572	-0.048753	2.157949	-0.995259	-0.026593	1.162342	...	2.564957	2.874556	0.443551	1.508644	-1.702975	4.317128	-1.488132	3.223802	0.036243	0
2499	2.827924	-0.391928	0.139046	0.273091	-1.208622	-0.401865	2.850371	0.857199	-1.050983	1.923212	...	4.897142	2.617614	1.056777	1.459342	-0.002913	2.095362	0.309448	3.965130	-0.366275	1

2500 rows × 21 columns

4. Data Drift Analysis

The data drift analysis in this demo is based on the histogram data drift application, and it is done per feature using two distance measure metrics for probability distributions.

Let us mark our sample set as $S$ and our inputs as $I$. We will look at one feature $x$ out of $n$ features. Assuming the histograms of feature $x$ is split into 20 bins: $b_1,b_2,…,b_{20}$, we will match the feature $x$ histogram of the inputs $I$ ($x_I$) into the same bins (meaning to $x_S$) and compare their distributions using:

• Total Variance Distance: $TVD(x_S,x_I) = \frac{1}{2}\sum_{b_1}^{b_{20}} {|x_S - x_I|}$

• Hellinger Distance: $H(x_S,x_I) = \sqrt{1-{\sum_{b_1}^{b_{20}}\sqrt{x_S \cdot x_I}}}$

Our rule then is calculating for each $x\in S: \frac{H(x_S,x_I)+TVD(x_S,x_I)}{2} < $ given thresholds.

The outputs of the analysis will be:

• Drift table plot - The results are presented in a plotly table artifact named "drift_table_plot" that shows each feature’s statistics and its TVD, Hellinger and KLD (Kullback–Leibler divergence) results as follows:

	Count		Mean		Std		Min		Max		Tvd	Hellinger	Kld	Histograms
	Sample	Input	Sample	Input	Sample	Input	Sample	Input	Sample	Input
x1	…	…	…	…	…	…	…	…	…	…	…	…	…	…
x2	…	…	…	…	…	…	…	…	…	…	…	…	…	…
x3	…	…	…	…	…	…	…	…	…	…	…	…	…	…

• Features drift results - A rule metric per feature dictionary is saved in a json file named "features_drift_results" where each key is a feature and its value is the feature’s metric value: Dict[str, float]

  {
      "x1": 0.12,
      "x2": 0.345,
      "x3": 0.00678,
      ...
  }
  

• In addition, two results are being added to summarize the drift analysis:

  • drift_status: bool - A boolean value indicating whether a drift was found.

  • drift_metric: float - The mean of all the features drift metric value (the rule above): for $n$ features and metric rule $M(x_S,x_I)=\frac{H(x_S,x_I)+TVD(x_S,x_I)}{2}$, drift_metric $=\frac{1}{n}\sum_{x\in S}M(x_S,x_I)$

  {
      "drift_status": True,
      "drift_metric": 0.81234
  }
  

4.1. Enable Model Monitoring

To calculate data drift and generate artifact results, we first need to define the model monitoring configuration. Next, we will deploy the model monitoring infrastructure, which includes the default histogram-based data drift application.

In this demo, we will use V3IO resources, but you can also provide your own. For more information on supported resources, please visit https://docs.mlrun.org/en/latest/model-monitoring/index.html#selecting-the-streaming-and-tsdb-platforms

4.1.1 Set credentials and deploy model monitoring infrastructure

project.set_model_monitoring_credentials(
    endpoint_store_connection="v3io",
    tsdb_connection="v3io",
    stream_path="v3io")

# Deploy model monitoring infrastructure
project.enable_model_monitoring(wait_for_deployment=True, base_period=1)

> 2024-10-08 10:24:25,053 [warning] enable_model_monitoring: 'base_period' < 10 minutes is not supported in production environments: {"project":"batch-infer-demo"}
2024-10-08 10:24:30  (info) Deploying function
2024-10-08 10:24:30  (info) Building
2024-10-08 10:24:30  (info) Staging files and preparing base images
2024-10-08 10:24:30  (warn) Using user provided base image, runtime interpreter version is provided by the base image
2024-10-08 10:24:30  (info) Building processor image
2024-10-08 10:26:01  (info) Build complete
2024-10-08 10:26:11  (info) Function deploy complete
2024-10-08 10:24:25  (info) Deploying function
2024-10-08 10:24:25  (info) Building
2024-10-08 10:24:25  (info) Staging files and preparing base images
2024-10-08 10:24:25  (warn) Using user provided base image, runtime interpreter version is provided by the base image
2024-10-08 10:24:25  (info) Building processor image
2024-10-08 10:26:06  (info) Build complete
2024-10-08 10:27:07  (info) Function deploy complete
2024-10-08 10:24:26  (info) Deploying function
2024-10-08 10:24:26  (info) Building
2024-10-08 10:24:28  (info) Staging files and preparing base images
2024-10-08 10:24:28  (warn) Using user provided base image, runtime interpreter version is provided by the base image
2024-10-08 10:24:28  (info) Building processor image
2024-10-08 10:26:01  (info) Build complete
2024-10-08 10:26:10  (info) Function deploy complete
2024-10-08 10:24:32  (info) Deploying function
2024-10-08 10:24:32  (info) Building
2024-10-08 10:24:32  (info) Staging files and preparing base images
2024-10-08 10:24:32  (warn) Using user provided base image, runtime interpreter version is provided by the base image
2024-10-08 10:24:32  (info) Building processor image
2024-10-08 10:26:01  (info) Build complete
2024-10-08 10:26:12  (info) Function deploy complete

4.1.2 Rerun batch infer with model monitoring

# Perform batch prediction, this time enable drift analysis
batch_inference_run = batch_inference_function.run(
    handler="infer",
    inputs={"dataset": generate_data_run.outputs["prediction_set"]},
    params={
        "model_path": train_run.outputs["model"],
        "label_columns": "label",
        "perform_drift_analysis": True,
        "model_endpoint_name": "my_cool_endpoint" # a display name for the model endpoint
    },
)

> 2024-10-08 10:27:29,954 [info] Storing function: {"db":"http://mlrun-api:8080","name":"batch-inference-v2-infer","uid":"ca26cf51bb984354aa2b6c42540af1f7"}
> 2024-10-08 10:27:30,317 [info] Job is running in the background, pod: batch-inference-v2-infer-qmszk
> 2024-10-08 10:27:34,169 [info] Loading model...
> 2024-10-08 10:27:35,237 [info] Loading data...
> 2024-10-08 10:27:35,277 [info] Calculating prediction...
> 2024-10-08 10:27:35,281 [info] Logging result set (x | prediction)...
> 2024-10-08 10:27:35,773 [info] Performing drift analysis...
> 2024-10-08 10:27:37,335 [info] Bumping model endpoint last request time (EP without serving): {"bumped_last_request":"2024-10-08T10:29:35.991331+00:00","current_request":"2024-10-08T10:27:35.991331+00:00","endpoint_id":"cc285767a39c0455d050a8c0d7b0421c19394aba","last_request":"2024-10-08T10:27:35.806824+00:00","project":"batch-infer-demo"}
> 2024-10-08 10:27:37,586 [info] To track results use the CLI: {"info_cmd":"mlrun get run ca26cf51bb984354aa2b6c42540af1f7 -p batch-infer-demo","logs_cmd":"mlrun logs ca26cf51bb984354aa2b6c42540af1f7 -p batch-infer-demo"}
> 2024-10-08 10:27:37,586 [info] Or click for UI: {"ui_url":"https://dashboard.default-tenant.app.vmdev57.lab.iguazeng.com/mlprojects/batch-infer-demo/jobs/monitor/ca26cf51bb984354aa2b6c42540af1f7/overview"}
> 2024-10-08 10:27:37,587 [info] Run execution finished: {"name":"batch-inference-v2-infer","status":"completed"}

project	uid	iter	start	state	kind	name	labels	inputs	parameters	results	artifacts
batch-infer-demo	...540af1f7	0	Oct 08 10:27:34	completed	run	batch-inference-v2-infer	v3io_user=eyald kind=job owner=eyald mlrun/client_version=1.7.0-rc51 mlrun/client_python_version=3.9.18 host=batch-inference-v2-infer-qmszk	dataset	model_path=store://models/batch-infer-demo/model:latest@1102e4aea9424149837a81aa214b9489 label_columns=label perform_drift_analysis=True model_endpoint_name=my_cool_endpoint	batch_id=286df90faa9d3edf774974db7b6d601c5412caad635bccb56e7e8598	prediction

Title ×

> to track results use the .show() or .logs() methods or click here to open in UI

> 2024-10-08 10:27:41,580 [info] Run execution finished: {"name":"batch-inference-v2-infer","status":"completed"}

4.2 Overview drift results

Please note that all results are available in the UI, including time-based metrics and drift results for each model endpoint.

4.2.1 List artifacts

# Wait until the monitoring application is triggered
import time
time.sleep(60)

# Retrieve the model endpoint
from mlrun.model_monitoring.api import get_or_create_model_endpoint
endpoint = get_or_create_model_endpoint(project=project.name, model_endpoint_name="my_cool_endpoint")

# Validate that the artifacts were logged in the project
artifacts = project.list_artifacts(
    labels={
        "mlrun/producer-type": "model-monitoring-app",
        "mlrun/app-name": "histogram-data-drift",
        "mlrun/endpoint-id": endpoint.metadata.uid,
    }
)

artifacts

[{'kind': 'plotly',
  'metadata': {'key': 'drift_table_plot',
   'project': 'batch-infer-demo',
   'iter': 0,
   'tree': '0f9bc30c-e223-4161-a4b3-7ea93a2b200d',
   'hash': 'a9cbb5cd3532a7de394134c94d7ffabcf337e25f',
   'uid': 'ffab65fdff6bfb6b1e53003f6f135a39d94933fc',
   'updated': '2024-10-08 10:29:12.088626+00:00',
   'labels': {'mlrun/runner-pod': 'nuclio-batch-infer-demo-histogram-data-drift-5bcfd66559-c58k5',
    'mlrun/producer-type': 'model-monitoring-app',
    'mlrun/app-name': 'histogram-data-drift',
    'mlrun/endpoint-id': 'cc285767a39c0455d050a8c0d7b0421c19394aba'},
   'created': '2024-10-08 10:29:12.088665+00:00',
   'tag': 'latest'},
  'spec': {'target_path': 'v3io:///projects/batch-infer-demo/artifacts/drift_table_plot.html',
   'viewer': 'plotly',
   'size': 4637273,
   'license': '',
   'producer': {'kind': 'project',
    'name': 'batch-infer-demo',
    'tag': '0f9bc30c-e223-4161-a4b3-7ea93a2b200d',
    'owner': 'eyald'},
   'format': 'html',
   'db_key': 'drift_table_plot'},
  'status': {'state': 'created'},
  'project': 'batch-infer-demo'},
 {'kind': 'artifact',
  'metadata': {'key': 'features_drift_results',
   'project': 'batch-infer-demo',
   'iter': 0,
   'tree': '8636f959-4738-4c36-9e86-1fbb2a30cb7c',
   'hash': '8021f8f05337970dfaf19803f7c106efca3cefae',
   'uid': 'a0a361f92fb9aa0d70bf3c0b8373eeff97b8d86a',
   'updated': '2024-10-08 10:29:10.747465+00:00',
   'labels': {'mlrun/runner-pod': 'nuclio-batch-infer-demo-histogram-data-drift-5bcfd66559-c58k5',
    'mlrun/producer-type': 'model-monitoring-app',
    'mlrun/app-name': 'histogram-data-drift',
    'mlrun/endpoint-id': 'cc285767a39c0455d050a8c0d7b0421c19394aba'},
   'created': '2024-10-08 10:29:10.747499+00:00',
   'tag': 'latest'},
  'spec': {'target_path': 'v3io:///projects/batch-infer-demo/artifacts/features_drift_results.json',
   'size': 530,
   'license': '',
   'producer': {'kind': 'project',
    'name': 'batch-infer-demo',
    'tag': '8636f959-4738-4c36-9e86-1fbb2a30cb7c',
    'owner': 'eyald'},
   'format': 'json',
   'db_key': 'features_drift_results'},
  'status': {'state': 'created'},
  'project': 'batch-infer-demo'}]

4.2.2 Plot artifacts

# Plot the drift results per feature and label
features_drift_results_artifact = project.get_artifact("features_drift_results")
features_drift_results_artifact.to_dataitem().show()

<IPython.core.display.JSON object>

# Plot the drift table artifact
drift_table_plot_artifact = project.get_artifact("drift_table_plot")
drift_table_plot_artifact.to_dataitem().show()