{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **V2 Model Server (SKLearn)**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Test one or more classifier models against held-out dataset.<br>\n",
    "Using held-out test features, evaluates the peformance of the estimated model.<br>\n",
    "Can be part of a kubeflow pipeline as a test step that is run post EDA and training/validation cycles.<br>\n",
    "This function is part of the [scikit-learn-pipeline demo.](https://github.com/mlrun/demos/tree/master/scikit-learn-pipeline)<br>\n",
    "To see how the model is trained or how the data-set is generated, check out `sklearn_classifier` function from the function marketplace repository\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **Steps**\n",
    "1. [Setup function parameters](#Setup-function-parameters)\n",
    "2. [Importing the function](#Importing-the-function)\n",
    "3. [Testing the function locally](#Testing-the-function-locally)\n",
    "4. [Testing the function remotely](#Testing-the-function-remotely)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **Setup function parameters**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_path = 'https://s3.wasabisys.com/iguazio/data/function-marketplace-data/sklearn_classifier/iris_dataset.csv'\n",
    "models_path = 'https://s3.wasabisys.com/iguazio/models/function-marketplace-models/test_classifier/RandomForestClassifier.pkl'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **Importing the function**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2021-10-17 14:04:23,167 [info] loaded project function-marketplace from MLRun DB\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<mlrun.serving.states.TaskStep at 0x7f95f58e5f50>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import mlrun\n",
    "mlrun.set_environment(project='function-marketplace')\n",
    "\n",
    "# Importing the function from the hub\n",
    "fn = mlrun.import_function(\"hub://v2_model_server\")\n",
    "fn.apply(mlrun.auto_mount())\n",
    "\n",
    "# Adding the model \n",
    "fn.add_model(key='RandomForestClassifier', model_path=models_path ,class_name='ClassifierModel')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **Testing the function locally**\n",
    "\n",
    "> Test against the iris dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2021-10-17 14:04:26,871 [info] model RandomForestClassifier was loaded\n",
      "> 2021-10-17 14:04:26,872 [info] Initializing endpoint records\n",
      "> 2021-10-17 14:04:26,899 [info] Loaded ['RandomForestClassifier']\n"
     ]
    }
   ],
   "source": [
    "# When mocking, class has to be present\n",
    "from v2_model_server import *\n",
    "\n",
    "# Mocking function\n",
    "server = fn.to_mock_server()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sepal length (cm)</th>\n",
       "      <th>sepal width (cm)</th>\n",
       "      <th>petal length (cm)</th>\n",
       "      <th>petal width (cm)</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>5.1</td>\n",
       "      <td>3.5</td>\n",
       "      <td>1.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4.9</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4.7</td>\n",
       "      <td>3.2</td>\n",
       "      <td>1.3</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4.6</td>\n",
       "      <td>3.1</td>\n",
       "      <td>1.5</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5.0</td>\n",
       "      <td>3.6</td>\n",
       "      <td>1.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   sepal length (cm)  sepal width (cm)  petal length (cm)  petal width (cm)  \\\n",
       "0                5.1               3.5                1.4               0.2   \n",
       "1                4.9               3.0                1.4               0.2   \n",
       "2                4.7               3.2                1.3               0.2   \n",
       "3                4.6               3.1                1.5               0.2   \n",
       "4                5.0               3.6                1.4               0.2   \n",
       "\n",
       "   label  \n",
       "0      0  \n",
       "1      0  \n",
       "2      0  \n",
       "3      0  \n",
       "4      0  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Getting the data\n",
    "import pandas as pd\n",
    "\n",
    "iris_dataset = pd.read_csv(data_path)\n",
    "iris_dataset.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# KFServing protocol event\n",
    "event_data = {\"inputs\": iris_dataset.drop(['label'],axis=1).values.tolist()}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = server.test(path='/v2/models/RandomForestClassifier/predict',body=event_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "When mocking to server, returned dict has the following fields : id, model_name, outputs\n",
      "model's accuracy 0.9733333333333334\n"
     ]
    }
   ],
   "source": [
    "print(f'When mocking to server, returned dict has the following fields : {\", \".join([x for x in response.keys()])}')\n",
    "print(f\"model's accuracy { sum(1 for x,y in zip(iris_dataset['label'],response['outputs']) if x == y) / len(response['outputs'])}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **Testing the function remotely**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2021-10-17 14:04:27,617 [info] Starting remote function deploy\n",
      "2021-10-17 14:04:27  (info) Deploying function\n",
      "2021-10-17 14:04:27  (info) Building\n",
      "2021-10-17 14:04:27  (info) Staging files and preparing base images\n",
      "2021-10-17 14:04:27  (info) Building processor image\n",
      "2021-10-17 14:04:29  (info) Build complete\n",
      "> 2021-10-17 14:04:39,180 [info] successfully deployed function: {'internal_invocation_urls': ['nuclio-function-marketplace-v2-model-server.default-tenant.svc.cluster.local:8080'], 'external_invocation_urls': ['default-tenant.app.dev39.lab.iguazeng.com:31003']}\n"
     ]
    }
   ],
   "source": [
    "address = fn.deploy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'{\"id\": \"ac6be063-b05f-4276-972b-5e0acb96dfd9\", \"model_name\": \"RandomForestClassifier\", \"outputs\": [0, 2]}'"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import json\n",
    "import requests\n",
    "\n",
    "# Made up data\n",
    "my_data = '''{\"inputs\":[[5.1, 3.5, 1.4, 0.2],[7.7, 3.8, 6.7, 2.2]]}'''\n",
    "\n",
    "# using requests to predict\n",
    "response = requests.put(address + \"/v2/models/RandomForestClassifier/predict\", json=json.dumps(my_data))\n",
    "response.text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Back to the top](#V2-Model-Server-(SKLearn))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:root] *",
   "language": "python",
   "name": "conda-root-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}