Live Model Server Testing#

Test model server via HTTP calls.
To see how the model is trained or how the data-set is generated, check out sklearn_classifier function from the function marketplace repository

Steps#

Setup function parameters
Importing the function
Running the function locally
Running the function remotely

import warnings
warnings.filterwarnings("ignore")

Setup function parameters#

# Declaring the model's path that we are going to test
models_path = 'https://s3.wasabisys.com/iguazio/models/function-marketplace-models/test_classifier/RandomForestClassifier.pkl'
# Testing with dataset
data_path = 'https://s3.wasabisys.com/iguazio/data/function-marketplace-data/sklearn_classifier/iris_dataset.csv'

import mlrun
mlrun.set_environment(project='function-marketplace')

# deploying v2_model_server in order to get an http endpoint to test
# importing the function from the hub
fn = mlrun.import_function("hub://v2_model_server")
fn.apply(mlrun.auto_mount())

# Adding the model 
fn.add_model(key='RandomForestClassifier', model_path=models_path ,class_name='ClassifierModel')
address = fn.deploy()

> 2021-10-18 05:03:08,020 [info] loaded project function-marketplace from MLRun DB
> 2021-10-18 05:03:08,791 [info] Starting remote function deploy
2021-10-18 05:03:08  (info) Deploying function
2021-10-18 05:03:08  (info) Building
2021-10-18 05:03:08  (info) Staging files and preparing base images
2021-10-18 05:03:08  (info) Building processor image
2021-10-18 05:03:10  (info) Build complete
2021-10-18 05:03:18  (info) Function deploy complete
> 2021-10-18 05:03:18,932 [info] successfully deployed function: {'internal_invocation_urls': ['nuclio-function-marketplace-v2-model-server.default-tenant.svc.cluster.local:8080'], 'external_invocation_urls': ['default-tenant.app.dev39.lab.iguazeng.com:31003']}

Importing the function#

# Importing the function from the hub
fn = mlrun.import_function("hub://v2_model_tester")
fn.apply(mlrun.auto_mount())

<mlrun.runtimes.kubejob.KubejobRuntime at 0x7f6ec6f06790>

Running the function locally#

tester_run = fn.run(name='model_server_tester',
                    handler = 'model_server_tester', # default value is 'handler', therefore specifying function entry point.
                    params={'addr': address, 'model': 'RandomForestClassifier'},
                    inputs={'table': data_path},
                    local=True)

> 2021-10-18 05:10:09,850 [info] starting run model_server_tester uid=c5773ef3ebc544d390c3b66cce7d7665 DB=http://mlrun-api:8080
> 2021-10-18 05:10:11,386 [info] testing with dataset against http://default-tenant.app.dev39.lab.iguazeng.com:31003, model: RandomForestClassifier
{'id': '674dc477-0281-4aaa-b98c-e74998ac295d', 'model_name': 'RandomForestClassifier', 'outputs': [0]}
{'id': '8ff10778-834b-4920-a455-2d638d125116', 'model_name': 'RandomForestClassifier', 'outputs': [1]}
{'id': 'c4a03a43-2715-4772-b88b-6db8a222f8d4', 'model_name': 'RandomForestClassifier', 'outputs': [1]}
{'id': '03d3b15c-843d-4ba3-af90-ec5aaf5ccec4', 'model_name': 'RandomForestClassifier', 'outputs': [2]}
{'id': '10caf431-7375-4d0a-840c-6d4bd479d2e3', 'model_name': 'RandomForestClassifier', 'outputs': [0]}
{'id': '7b9c8753-6ff7-46f4-a77c-e91a2f49826d', 'model_name': 'RandomForestClassifier', 'outputs': [2]}
{'id': 'e08f322b-f732-4d88-9911-a6b4d55b6b51', 'model_name': 'RandomForestClassifier', 'outputs': [0]}
{'id': '2eb7912a-d464-45c0-a4d7-c884613f9a93', 'model_name': 'RandomForestClassifier', 'outputs': [2]}
{'id': 'a03751af-e2c5-4123-b75f-7d03618cff30', 'model_name': 'RandomForestClassifier', 'outputs': [0]}
{'id': '22a2abfa-cab6-44c3-8253-8f598b01ab5b', 'model_name': 'RandomForestClassifier', 'outputs': [0]}
{'id': '881e7014-8eeb-4e44-a88e-f8fa2bd865e8', 'model_name': 'RandomForestClassifier', 'outputs': [1]}
{'id': '400ead78-7335-4037-89b6-e19b5fef8e7b', 'model_name': 'RandomForestClassifier', 'outputs': [2]}
{'id': 'de3f8418-6ac6-482d-876d-8a12964f3c15', 'model_name': 'RandomForestClassifier', 'outputs': [1]}
{'id': '80d4adf7-c44f-450f-83a7-bb9f22f5600e', 'model_name': 'RandomForestClassifier', 'outputs': [2]}
{'id': '38fbb00f-dee5-4f48-a3b5-422c9100bf70', 'model_name': 'RandomForestClassifier', 'outputs': [1]}
{'id': '2fb61f97-ab33-4bfe-aa7a-cb65dac9f525', 'model_name': 'RandomForestClassifier', 'outputs': [0]}
{'id': '803731c6-d1a1-4ec1-a1da-91b0915596a2', 'model_name': 'RandomForestClassifier', 'outputs': [0]}
{'id': '92c7a186-e714-4dc4-860a-50b17696cf86', 'model_name': 'RandomForestClassifier', 'outputs': [1]}
{'id': 'f6fd25a4-b056-4fc7-a37a-244f01268c36', 'model_name': 'RandomForestClassifier', 'outputs': [0]}
{'id': '4e37f305-ed1e-4594-8289-28353898a2e5', 'model_name': 'RandomForestClassifier', 'outputs': [0]}
> 2021-10-18 05:10:11,984 [info] run 20 tests, 0 errors and 9 match expected value

project	uid	iter	start	state	name	labels	inputs	parameters	results	artifacts
function-marketplace	...ce7d7665	0	Oct 18 05:10:10	completed	model_server_tester	v3io_user=dani kind= owner=dani host=jupyter-dani-6bfbd76d96-zxx6f	table	addr=http://default-tenant.app.dev39.lab.iguazeng.com:31003 model=RandomForestClassifier	total_tests=20 errors=0 match=9 avg_latency=26833 min_latency=19420 max_latency=152063	latency

> to track results use the .show() or .logs() methods or click here to open in UI

> 2021-10-18 05:10:12,045 [info] run executed, status=completed

tester_run.artifact('latency').show()

Running the function remotely#

tester_run = fn.run(name='model_server_tester',
                    handler = 'model_server_tester', # default value is 'handler', therefore specifying function entry point.
                    params={'addr': address, 'model': 'RandomForestClassifier'},
                    inputs={'table': data_path},
                    local=False)

> 2021-10-18 05:12:30,250 [info] starting run model_server_tester uid=a3ffac1155564332a828961fa66ab30c DB=http://mlrun-api:8080
> 2021-10-18 05:12:30,380 [info] Job is running in the background, pod: model-server-tester-thhll
> 2021-10-18 05:12:36,763 [info] testing with dataset against http://default-tenant.app.dev39.lab.iguazeng.com:31003, model: RandomForestClassifier
{'id': '4318bfe9-ec16-43ad-9990-d637a87275a7', 'model_name': 'RandomForestClassifier', 'outputs': [2]}
{'id': '71ba6a72-dd4c-4cb0-a877-39dc8d739377', 'model_name': 'RandomForestClassifier', 'outputs': [2]}
{'id': 'c5359476-b0b1-4dd8-8ce7-562addb02929', 'model_name': 'RandomForestClassifier', 'outputs': [0]}
{'id': '0679c92f-9f92-46bc-a773-8df73de2c5de', 'model_name': 'RandomForestClassifier', 'outputs': [2]}
{'id': '2c272335-ee43-458e-b662-c236d29f62dc', 'model_name': 'RandomForestClassifier', 'outputs': [2]}
{'id': 'c9e600b5-36ef-4366-b771-519a3037bb2e', 'model_name': 'RandomForestClassifier', 'outputs': [1]}
{'id': '84ea193f-ff47-4431-b689-5ad29284e3cc', 'model_name': 'RandomForestClassifier', 'outputs': [1]}
{'id': 'dea37518-ceff-487c-a416-d7f4eb69a448', 'model_name': 'RandomForestClassifier', 'outputs': [0]}
{'id': 'c35b9f63-95b5-4c3a-af3f-1f47a9427285', 'model_name': 'RandomForestClassifier', 'outputs': [2]}
{'id': '7df9a5f2-5cd7-40d3-bc44-60ba8a87d349', 'model_name': 'RandomForestClassifier', 'outputs': [0]}
{'id': '58d7fa40-f094-49c0-bf94-626571aab0fd', 'model_name': 'RandomForestClassifier', 'outputs': [1]}
{'id': 'b1ced717-5ddd-4148-8a6a-c98e737e2649', 'model_name': 'RandomForestClassifier', 'outputs': [0]}
{'id': '9cec3bb4-4770-4679-abaf-4f3fec2302da', 'model_name': 'RandomForestClassifier', 'outputs': [2]}
{'id': '5971197e-1d33-4338-a114-9cf79349d69b', 'model_name': 'RandomForestClassifier', 'outputs': [0]}
{'id': 'd844c786-c799-429c-9192-0bf71c63099b', 'model_name': 'RandomForestClassifier', 'outputs': [1]}
{'id': '383ee50e-e8be-4744-8cad-ab35be13acd9', 'model_name': 'RandomForestClassifier', 'outputs': [1]}
{'id': '627d7683-cf8e-4e1f-832c-a8da929b1ce5', 'model_name': 'RandomForestClassifier', 'outputs': [2]}
{'id': '35759e5a-33a7-443d-a7e2-11f081347cc6', 'model_name': 'RandomForestClassifier', 'outputs': [1]}
{'id': 'bdb5571f-116d-4744-93e3-a2e5f983112f', 'model_name': 'RandomForestClassifier', 'outputs': [2]}
{'id': '978496b0-22bf-4372-a77f-90091f96388a', 'model_name': 'RandomForestClassifier', 'outputs': [0]}
> 2021-10-18 05:12:37,462 [info] run 20 tests, 0 errors and 6 match expected value
> 2021-10-18 05:12:37,553 [info] run executed, status=completed
final state: completed

project	uid	iter	start	state	name	labels	inputs	parameters	results	artifacts
function-marketplace	...a66ab30c	0	Oct 18 05:12:35	completed	model_server_tester	v3io_user=dani kind=job owner=dani host=model-server-tester-thhll	table	addr=http://default-tenant.app.dev39.lab.iguazeng.com:31003 model=RandomForestClassifier	total_tests=20 errors=0 match=6 avg_latency=29752 min_latency=27708 max_latency=33304	latency