Count Events Demo#
Overview#
This notebook walks through a simple example of how to monitor a real-time serving function and how to add your a custom monitoring application from the hub. For simplicity, we’ll use the Count Events application, which calculates the number of requests in each time window. If you’d like to create your own model monitoring application (which can later be added to the hub), follow these instructions:https://docs.mlrun.org/en/stable/model-monitoring/applications.html
To add a model monitoring application to your project from the hub, you can choose one of two approaches:
Set it directly – the application will be deployed as is.
Import it as a module – this lets you test and modify the application code before deploying it.
Demo#
Create a project#
import mlrun
project = mlrun.get_or_create_project("count-events-demo",'./')
> 2025-11-05 15:33:39,611 [warning] Failed resolving version info. Ignoring and using defaults
> 2025-11-05 15:33:43,049 [warning] Server or client version is unstable. Assuming compatible: {"client_version":"0.0.0+unstable","server_version":"1.11.0"}
> 2025-11-05 15:33:58,614 [info] Created and saved project: {"context":"./","from_template":null,"name":"count-events-demo","overwrite":false,"save":true}
> 2025-11-05 15:33:58,616 [info] Project created successfully: {"project_name":"count-events-demo","stored_in_db":true}
Generate datastore profiles for model monitoring#
Before you enable model monitoring, you must configure datastore profiles for TSDB and streaming endpoints. A datastore profile holds all the information required to address an external data source, including credentials. Model monitoring supports Kafka and V3IO as streaming platforms, and TDEngine and V3IO as TSDB platforms.
In this example we will use V3IO for both streaming and TSDB platforms.
from mlrun.datastore.datastore_profile import (
DatastoreProfileV3io
)
v3io_profile = DatastoreProfileV3io(name="v3io_profile", v3io_access_key=mlrun.mlconf.get_v3io_access_key())
project.register_datastore_profile(v3io_profile)
project.set_model_monitoring_credentials(stream_profile_name=v3io_profile.name, tsdb_profile_name=v3io_profile.name)
Deploy model monitoring infrastructure#
Once you’ve provided the model monitoring credentials, you can enable monitoring capabilities for your project. Visit MLRun’s Model Monitoring Architecture to read more.
project.enable_model_monitoring(base_period=10,
deploy_histogram_data_drift_app=False, # built-in monitoring application for structured data
wait_for_deployment=True)
2025-11-05 15:41:01 (info) Deploying function
2025-11-05 15:41:01 (info) Building
2025-11-05 15:41:01 (info) Staging files and preparing base images
2025-11-05 15:41:01 (warn) Using user provided base image, runtime interpreter version is provided by the base image
2025-11-05 15:41:02 (info) Building processor image
2025-11-05 15:42:57 (info) Build complete
2025-11-05 15:43:07 (info) Function deploy complete
2025-11-05 15:40:57 (info) Deploying function
2025-11-05 15:40:57 (info) Building
2025-11-05 15:40:58 (info) Staging files and preparing base images
2025-11-05 15:40:58 (warn) Using user provided base image, runtime interpreter version is provided by the base image
2025-11-05 15:40:58 (info) Building processor image
2025-11-05 15:42:53 (info) Build complete
2025-11-05 15:43:12 (info) Function deploy complete
2025-11-05 15:40:59 (info) Deploying function
2025-11-05 15:40:59 (info) Building
2025-11-05 15:40:59 (info) Staging files and preparing base images
2025-11-05 15:40:59 (warn) Using user provided base image, runtime interpreter version is provided by the base image
2025-11-05 15:41:00 (info) Building processor image
2025-11-05 15:42:55 (info) Build complete
2025-11-05 15:43:03 (info) Function deploy complete
Log Models#
We’ll generate some dummy classification models and log them to the project.
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
import pickle
import pandas as pd
# Prepare a model and generate training set
X,y = make_classification(n_samples=200,n_features=5,random_state=42)
X_train,X_test,y_train,y_test = train_test_split(X,y,train_size=0.8,test_size=0.2,random_state=42)
model = LinearRegression()
model.fit(X_train,y_train)
X_test = pd.DataFrame(X_test,columns=[f"column_{i}" for i in range(5)])
y_test = pd.DataFrame(y_test,columns=["label"])
training_set = pd.concat([X_test,y_test],axis=1)
# Log your models
for i in range(5):
project.log_model(key=f"model_{i}",body=pickle.dumps(model),model_file=f'model.pkl',training_set=training_set,label_column="label")
Deploy Serving Function#
We’ll use a basic serving function and enrich it with the logged models.
Note that if you want to monitor a serving function along with its associated models, you must enable tracking by calling set_tracking(). Otherwise, the serving function’s requests won’t be monitored.
# Define the serving
serving = mlrun.new_function('serving-model-v1',kind='serving')
graph = serving.set_topology("router", engine="sync")
# Apply monitoring
serving.set_tracking()
# Add models to your serving
models_uri = [model.uri for model in project.list_models(tag="latest")]
i=0
from tqdm import tqdm
for uri in tqdm(models_uri):
serving.add_model(key=f'model_{i}',model_path=uri,class_name='mlrun.frameworks.sklearn.SKLearnModelServer')
i+=1
100%|██████████| 5/5 [00:00<00:00, 22052.07it/s]
# Deploy serving
serving_function = project.deploy_function(serving)
> 2025-11-05 15:55:08,989 [info] Starting remote function deploy
2025-11-05 15:55:09 (info) Deploying function
2025-11-05 15:55:09 (info) Building
2025-11-05 15:55:09 (info) Staging files and preparing base images
2025-11-05 15:55:09 (warn) Using user provided base image, runtime interpreter version is provided by the base image
2025-11-05 15:55:09 (info) Building processor image
2025-11-05 15:56:54 (info) Build complete
2025-11-05 15:57:06 (info) Function deploy complete
> 2025-11-05 15:57:10,181 [info] Model endpoint creation task completed with state succeeded
> 2025-11-05 15:57:10,181 [info] Successfully deployed function: {"external_invocation_urls":["count-events-demo-serving-model-v1.default-tenant.app.vmdev211.lab.iguazeng.com/"],"internal_invocation_urls":["nuclio-count-events-demo-serving-model-v1.default-tenant.svc.cluster.local:8080"]}
Invoke Serving#
Let’s generate some dummy data and invoke our serving function.
serving = project.get_function("serving-model-v1")
inputs = [[-0.51,0.051,0.6287761723991921,-0.8751269647375463,-1.0660002219502747], [-0.51,0.051,0.6287761723991921,-0.8751269647375463,-1.0660002219502747], [-0.51,0.051,0.6287761723991921,-0.8751269647375463,-1.0660002219502747], [-0.51,0.051,0.6287761723991921,-0.8751269647375463,-1.0660002219502747], [-0.51,0.051,0.6287761723991921,-0.8751269647375463,-1.0660002219502747]]
import time
for i in range(5):
for j in range(100):
serving.invoke(f"/v2/models/model_{i}/infer", {"inputs": inputs})
Evaluate App#
Before deploying the Count Events application, let’s first test it to make sure it works as expected. We’ll import it as a module, which downloads the module file to your local filesystem, and then run it as a job using the evaluate mechanism.
# Import count events from the hub
count_events_app = mlrun.import_module("hub://count_events")
# Run the app as a job
res = count_events_app.CountApp.evaluate(func_path="count_events.py",
run_local=False,
sample_data=pd.DataFrame({"col": [1, 2, 3, 4]}),
image=image,
endpoints=["model_0"])
> 2025-11-05 15:57:37,746 [info] Changing function name - adding `"-batch"` suffix: {"func_name":"countapp-batch"}
> 2025-11-05 15:57:37,927 [info] Storing function: {"db":"http://mlrun-api:8080","name":"countapp-batch--handler","uid":"b7c240fd99ed4c9b940db6a587a53b80"}
> 2025-11-05 15:57:38,202 [info] Job is running in the background, pod: countapp-batch--handler-469fm
> 2025-11-05 15:57:42,390 [info] Counted events for model endpoint window: {"count":4,"end":"NaT","model_endpoint_name":"model_0","start":"NaT"}
> 2025-11-05 15:57:42,498 [info] To track results use the CLI: {"info_cmd":"mlrun get run b7c240fd99ed4c9b940db6a587a53b80 -p count-events-demo","logs_cmd":"mlrun logs b7c240fd99ed4c9b940db6a587a53b80 -p count-events-demo"}
> 2025-11-05 15:57:42,498 [info] Or click for UI: {"ui_url":"https://dashboard.default-tenant.app.vmdev211.lab.iguazeng.com/mlprojects/count-events-demo/jobs/monitor-jobs/countapp-batch--handler/b7c240fd99ed4c9b940db6a587a53b80/overview"}
> 2025-11-05 15:57:42,499 [info] Run execution finished: {"name":"countapp-batch--handler","status":"completed"}
| project | uid | iter | start | end | state | kind | name | labels | inputs | parameters | results |
|---|---|---|---|---|---|---|---|---|---|---|---|
| count-events-demo | 0 | Nov 05 15:57:41 | 2025-11-05 15:57:42.474376+00:00 | completed | run | countapp-batch--handler | v3io_user=iguazio kind=job owner=iguazio mlrun/client_version=0.0.0+unstable mlrun/client_python_version=3.11.12 host=countapp-batch--handler-469fm |
sample_data |
endpoints=['model_0'] write_output=False existing_data_handling=fail_on_overlap stream_profile=None |
model_0-d25a6714a19b4027b9bccfe8adca8ddc_NaT_NaT={'metric_name': 'count', 'metric_value': 4.0} |
> 2025-11-05 15:57:46,373 [info] Run execution finished: {"name":"countapp-batch--handler","status":"completed"}
res.outputs
{'model_0-d25a6714a19b4027b9bccfe8adca8ddc_NaT_NaT': {'metric_name': 'count',
'metric_value': 4.0}}
Now that the application is available on your filesystem, you can register and deploy it just like any other custom application.
fn = project.set_model_monitoring_function(
func="count_events.py",
application_class="CountApp",
name="CountEventsFromFile",
image=image,
)
project.deploy_function(fn)
> 2025-11-05 16:09:48,293 [info] Starting remote function deploy
2025-11-05 16:09:48 (info) Deploying function
2025-11-05 16:09:48 (info) Building
2025-11-05 16:09:48 (info) Staging files and preparing base images
2025-11-05 16:09:48 (warn) Using user provided base image, runtime interpreter version is provided by the base image
2025-11-05 16:09:48 (info) Building processor image
2025-11-05 16:11:33 (info) Build complete
2025-11-05 16:11:41 (info) Function deploy complete
> 2025-11-05 16:11:49,604 [info] Model endpoint creation task completed with state succeeded
> 2025-11-05 16:11:49,605 [info] Successfully deployed function: {"external_invocation_urls":[],"internal_invocation_urls":["nuclio-count-events-demo-counteventsfromfile.default-tenant.svc.cluster.local:8080"]}
DeployStatus(state=ready, outputs={'endpoint': 'http://nuclio-count-events-demo-counteventsfromfile.default-tenant.svc.cluster.local:8080', 'name': 'count-events-demo-counteventsfromfile'})
Set Application from Hub#
As mentioned, you can set the application directly from the hub by providing a valid hub path (hub://<app_name>).
fn = project.set_model_monitoring_function(
func="hub://count_events",
application_class="CountApp",
name="CountEvents",
)
project.deploy_function(fn)
> 2025-11-05 15:57:58,659 [info] Starting remote function deploy
2025-11-05 15:57:59 (info) Deploying function
2025-11-05 15:57:59 (info) Building
2025-11-05 15:57:59 (info) Staging files and preparing base images
2025-11-05 15:57:59 (warn) Using user provided base image, runtime interpreter version is provided by the base image
2025-11-05 15:57:59 (info) Building processor image
2025-11-05 15:59:34 (info) Build complete
2025-11-05 15:59:42 (info) Function deploy complete
> 2025-11-05 15:59:49,826 [info] Model endpoint creation task completed with state succeeded
> 2025-11-05 15:59:49,827 [info] Successfully deployed function: {"external_invocation_urls":[],"internal_invocation_urls":["nuclio-count-events-demo-countevents.default-tenant.svc.cluster.local:8080"]}
DeployStatus(state=ready, outputs={'endpoint': 'http://nuclio-count-events-demo-countevents.default-tenant.svc.cluster.local:8080', 'name': 'count-events-demo-countevents'})