Image Classification Model - Serving Function#
This notebook demonstrates how to deploy a Tensorflow model using MLRun & Nuclio.
In this notebook you will:
Write a Tensorflow-Model class to load and predict on the incoming data
Deploy the model as a serverless function
Invoke the serving endpoint with data as:
URLs to images hosted on S3
Direct image send
Steps:
Define Nuclio Function#
To use the magic commands for deploying this jupyter notebook as a nuclio function we must first import nuclio
Since we do not want to import nuclio in the actual function, the comment annotation nuclio: ignore is used. This marks the cell for nuclio, telling it to ignore the cell’s values when building the function.
# nuclio: ignore
import nuclio
Install dependencies and set config#
Note: Since tensorflow is being pulled from the baseimage it is not directly installed as a build command. If it is not installed on your system please uninstall and install using the line:
pip install tensorflow
%nuclio config kind="nuclio:serving"
%nuclio env MODEL_CLASS=TF2Model
# tensorflow 2 use the default serving image (or the mlrun/ml-models for a faster build)
%nuclio config spec.build.baseImage = "mlrun/mlrun"
%nuclio: setting kind to 'nuclio:serving'
%nuclio: setting 'MODEL_CLASS' environment variable
%nuclio: setting spec.build.baseImage to 'mlrun/mlrun'
Since we are using packages which are not surely installed on our baseimage, or want to verify that a specific version of the package will be installed we use the %nuclio cmd annotation.
%nuclio cmdworks both locally and during deployment by default, but can be set with-cflag to only run the commands while deploying or-lto set the variable for the local environment only.
%%nuclio cmd -c
pip install tensorflow>=2.1
pip install requests pillow
Function Code#
import warnings
warnings.simplefilter(action="ignore", category=FutureWarning)
import json
import numpy as np
import requests
from tensorflow import keras
from tensorflow.keras.models import load_model
from tensorflow.keras.preprocessing import image
from tensorflow.keras.preprocessing.image import load_img
from os import environ, path
from PIL import Image
from io import BytesIO
from urllib.request import urlopen
import mlrun
Model Serving Class#
We define the TFModel class which we will use to define data handling and prediction of our model.
The class should consist of:
__init__(name, model_dir)- Setup the internal parametersload(self)- How to load the model and broadcast it’s ready for predictionpreprocess(self, body)- How to handle the incoming event, forming the request to an{'instances': [<samples>]}dictionary as requested by the protocolpredict(self, data)- Receives and{'instances': [<samples>]}and returns the model’s prediction as a listpostprocess(self, data)- Does any additional processing needed on the predictions.
class TFModel(mlrun.runtimes.MLModelServer):
def __init__(self, name: str, model_dir: str):
super().__init__(name, model_dir)
self.IMAGE_WIDTH = int(environ.get('IMAGE_WIDTH', '128'))
self.IMAGE_HEIGHT = int(environ.get('IMAGE_HEIGHT', '128'))
try:
with open(environ['classes_map'], 'r') as f:
self.classes = json.load(f)
except:
self.classes = None
def load(self):
model_file, extra_data = self.get_model('.h5')
self.model = load_model(model_file)
def preprocess(self, body):
try:
output = {'instances': []}
instances = body.get('instances', [])
for byte_image in instances:
img = Image.open(byte_image)
img = img.resize((self.IMAGE_WIDTH, self.IMAGE_HEIGHT))
# Load image
x = image.img_to_array(img)
x = np.expand_dims(x, axis=0)
output['instances'].append(x)
# Format instances list
output['instances'] = [np.vstack(output['instances'])]
return output
except:
raise Exception(f'received: {body}')
def predict(self, data):
images = data.get('instances', [])
# Predict
predicted_probability = self.model.predict(images)
# return prediction
return predicted_probability
def postprocess(self, predicted_probability):
if self.classes:
predicted_classes = np.around(predicted_probability, 1).tolist()[0]
predicted_probabilities = predicted_probability.tolist()[0]
return {
'prediction': [self.classes[str(int(cls))] for cls in predicted_classes],
f'{self.classes["1"]}-probability': predicted_probabilities
}
else:
return predicted_probability.tolist()[0]
To let our nuclio builder know that our function code ends at this point we will use the comment annotation nuclio: end-code.
Any new cell from now on will be treated as if a nuclio: ignore comment was set, and will not be added to the funcion.
# nuclio: end-code
Test the function locally#
Make sure your local TF / Keras version is the same as pulled in the nuclio image for accurate testing
Set the served models and their file paths using: SERVING_MODEL_<name> = <model file path>
Note: this notebook assumes the model and categories are under /User/mlrun/examples/
from PIL import Image
from io import BytesIO
import matplotlib.pyplot as plt
import os
Define test parameters#
# Testing event
cat_image_url = 'https://s3.amazonaws.com/iguazio-sample-data/images/catanddog/cat.102.jpg'
response = requests.get(cat_image_url)
cat_image = response.content
img = Image.open(BytesIO(cat_image))
print('Test image:')
plt.imshow(img)
Test image:
<matplotlib.image.AxesImage at 0x7f8ef06357f0>
Define Function specifications#
import os
from mlrun import mlconf
# Model Server variables
model_class = 'TFModel'
model_name = 'cat_vs_dog_tfv2' # Define for later use in tests
models = {model_name: os.path.join(mlconf.artifact_path, 'tf2/cats_n_dogs.h5')}
# Specific model variables
function_envs = {
'IMAGE_HEIGHT': 128,
'IMAGE_WIDTH': 128,
'classes_map': '/User/artifacts/categories_map.json',
}
Deploy the serving function to the cluster#
from mlrun import new_model_server, mount_v3io
# Setup the model server function
fn = new_model_server('tf2-serving',
model_class=model_class,
models=models)
fn.set_envs(function_envs)
fn.spec.description = "tf2 image classification server"
fn.metadata.categories = ['serving', 'dl']
fn.metadata.labels = {'author': 'yaronh'}
fn.export("function.yaml")
[mlrun] 2020-05-04 22:34:16,419 function spec saved to path: function.yaml
<mlrun.runtimes.function.RemoteRuntime at 0x7fb5190c9908>
if "V3IO_HOME" in list(os.environ):
from mlrun import mount_v3io
fn.apply(mount_v3io())
else:
# is you set up mlrun using the instructions at
# https://github.com/mlrun/mlrun/blob/master/hack/local/README.md
from mlrun.platforms import mount_pvc
fn.apply(mount_pvc('nfsvol', 'nfsvol', '/home/joyan/data'))
# Deploy the model server
addr = fn.deploy(project='cat-and-dog-servers')
[mlrun] 2020-04-30 20:56:50,173 deploy started
[nuclio] 2020-04-30 20:56:54,304 (info) Build complete
[nuclio] 2020-04-30 20:57:01,421 done updating tensorflow-v2-2layers, function address: 3.135.130.246:30031
Test the deployed function on the cluster#
Test the deployed function (with URL)#
# URL event
event_body = json.dumps({"data_url": cat_image_url})
print(f'Sending event: {event_body}')
headers = {'Content-type': 'application/json'}
response = requests.post(url=addr + f'/{model_name}/predict', data=event_body, headers=headers)
response.content
Sending event: {"data_url": "https://s3.amazonaws.com/iguazio-sample-data/images/catanddog/cat.102.jpg"}
b'[4.548141341568789e-27]'
Test the deployed function (with Jpeg Image)#
# URL event
event_body = cat_image
print(f'Sending image from {cat_image_url}')
plt.imshow(img)
headers = {'Content-type': 'image/jpeg'}
response = requests.post(url=addr + f'/{model_name}/predict/', data=event_body, headers=headers)
response.content
Sending image from https://s3.amazonaws.com/iguazio-sample-data/images/catanddog/cat.102.jpg
b'[4.548141341568789e-27]'