AzureML AutoML Demo
Contents
AzureML AutoML Demo#
MLRun function for using Azure AutoML, Including the following handlers:
init_experiment- Initialize workspace and experiment in Azure ML.init_compute- Initialize Azure ML compute target to run experiment.register_dataset- Register dataset object (can be also an Iguazio FeatureVector) in Azure ML.download_model- Download trained model from Azure ML to local filesystem.upload_model- Upload pre-trained model from local filesystem to Azure ML.submit_training_job- Submit training job to Azure AutoML and download trained model when completed.automl_train- Whole training flow for Azure AutoML: - Initializing workspace and experiment in Azure ML - Registers dataset/feature vector, - submits training job - downloads trained model
1. Setup MLRun Project#
Creating MLRun project
import mlrun
> 2022-02-02 18:28:06,840 [warning] Failed resolving version info. Ignoring and using defaults
> 2022-02-02 18:28:11,379 [warning] Server or client version is unstable. Assuming compatible: {'server_version': '0.0.0+unstable', 'client_version': '0.0.0+unstable'}
# Initialize the MLRun project object
project = mlrun.get_or_create_project('azureml', context="./", user_project=True)
> 2022-02-02 18:28:11,423 [info] loaded project azureml from MLRun DB
2. Preparing Dataset (Iris)#
Preparing training URI for the MLRun function
DATA_URL = "https://s3.wasabisys.com/iguazio/data/iris/iris_dataset.csv"
mlrun.get_dataitem(DATA_URL).as_df().head()
| sepal length (cm) | sepal width (cm) | petal length (cm) | petal width (cm) | label | |
|---|---|---|---|---|---|
| 0 | 5.1 | 3.5 | 1.4 | 0.2 | 0 |
| 1 | 4.9 | 3.0 | 1.4 | 0.2 | 0 |
| 2 | 4.7 | 3.2 | 1.3 | 0.2 | 0 |
| 3 | 4.6 | 3.1 | 1.5 | 0.2 | 0 |
| 4 | 5.0 | 3.6 | 1.4 | 0.2 | 0 |
3. Submit Azure AutoML Training Job#
Submit Azure Secrets#
For more information about working with secrets see: MLRun docs: Working with secrets
project.set_secrets(file_path="env")
Import azureml_utils from marketplace#
azureml_fn = mlrun.import_function('hub://azureml_utils')
azureml_fn.deploy()
Automl configuration & run parameters#
The
automl_settingsobject is the setup for Azure AutoML. It holds thetasktype, number of models to train -iterations, the desired metric -primary metric, the allowed types of modelsallowed_modelsand more.The
paramsare the parameters for the MLRun function, such as experiment (experiment_name) and cpu cluster (cpu_cluster_name) names in AzureML, dataset properties for registration, target label for training -label_column_name, number of models to downloadsave_n_modelsand more.
label_column_name = 'label' # target label
# Configure automl settings:
automl_settings = {
"task": 'classification',
"debug_log": 'automl_errors.log',
# "experiment_exit_score" : 0.9,
"enable_early_stopping": False,
"allowed_models": ['LogisticRegression', 'SGD', 'SVM'],
"iterations": 5,
"iteration_timeout_minutes": 2,
"max_concurrent_iterations": 2,
"max_cores_per_iteration": -1,
"n_cross_validations": 5,
"primary_metric": 'accuracy',
"featurization": 'off',
"model_explainability": False,
"enable_voting_ensemble": False,
"enable_stack_ensemble": False
}
# Setting params to azure_run function:
params = {
"experiment_name": 'azure-automl-test',
"cpu_cluster_name": 'azureml-cpu',
"dataset_name": 'iris',
"dataset_description": 'iris training data',
"label_column_name": label_column_name,
"create_new_version": True,
"register_model_name": "iris-model",
"save_n_models": 3,
"automl_settings": automl_settings
}
Run Azure AutoML train:#
This MLRun function will perform the following:
Initialize workspace and experiment in your AzureML
Register the dataset/feature vector to Iguazio and to AzureML.
Submit the training job to AzureML and print the live training results fro each model
Generate the top trained models.
azureml_run = azureml_fn.run(
handler="train",
inputs={"dataset": DATA_URL},
params=params,
)
> 2022-02-02 18:28:11,740 [info] Function is not deployed and auto_build flag is set, starting deploy...
> 2022-02-02 18:28:11,932 [info] Started building image: .mlrun/func-azureml-yonatan-azureml-utils:latest
INFO[0000] Retrieving image manifest python:3.7.9-slim
INFO[0000] Retrieving image python:3.7.9-slim from registry index.docker.io
INFO[0000] Built cross stage deps: map[]
INFO[0000] Retrieving image manifest python:3.7.9-slim
INFO[0000] Returning cached image manifest
INFO[0000] Executing 0 build triggers
INFO[0000] Unpacking rootfs as cmd RUN python -m pip install pip==21.2.4 requires it.
INFO[0002] RUN python -m pip install pip==21.2.4
INFO[0002] Taking snapshot of full filesystem...
INFO[0003] cmd: /bin/sh
INFO[0003] args: [-c python -m pip install pip==21.2.4]
INFO[0003] Running: [/bin/sh -c python -m pip install pip==21.2.4]
Collecting pip==21.2.4
Downloading pip-21.2.4-py3-none-any.whl (1.6 MB)
Installing collected packages: pip
Attempting uninstall: pip
Found existing installation: pip 21.0.1
Uninstalling pip-21.0.1:
Successfully uninstalled pip-21.0.1
Successfully installed pip-21.2.4
INFO[0006] Taking snapshot of full filesystem...
INFO[0006] RUN apt-get update && apt-get install -y --no-install-recommends git
INFO[0006] cmd: /bin/sh
INFO[0006] args: [-c apt-get update && apt-get install -y --no-install-recommends git]
INFO[0006] Running: [/bin/sh -c apt-get update && apt-get install -y --no-install-recommends git]
Get:1 http://security.debian.org/debian-security buster/updates InRelease [65.4 kB]
Get:2 http://deb.debian.org/debian buster InRelease [122 kB]
Get:3 http://deb.debian.org/debian buster-updates InRelease [51.9 kB]
Get:4 http://security.debian.org/debian-security buster/updates/main amd64 Packages [314 kB]
Get:5 http://deb.debian.org/debian buster/main amd64 Packages [7906 kB]
Get:6 http://deb.debian.org/debian buster-updates/main amd64 Packages [8792 B]
Fetched 8468 kB in 2s (5347 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
git-man libcurl3-gnutls liberror-perl libgdbm-compat4 libgssapi-krb5-2
libk5crypto3 libkeyutils1 libkrb5-3 libkrb5support0 libldap-2.4-2
libldap-common libnghttp2-14 libpcre2-8-0 libperl5.28 libpsl5 librtmp1
libsasl2-2 libsasl2-modules-db libssh2-1 perl perl-modules-5.28
Suggested packages:
gettext-base git-daemon-run | git-daemon-sysvinit git-doc git-el git-email
git-gui gitk gitweb git-cvs git-mediawiki git-svn krb5-doc krb5-user
sensible-utils perl-doc libterm-readline-gnu-perl
| libterm-readline-perl-perl make libb-debug-perl liblocale-codes-perl
Recommended packages:
patch less ssh-client krb5-locales publicsuffix libsasl2-modules
The following NEW packages will be installed:
git git-man libcurl3-gnutls liberror-perl libgdbm-compat4 libgssapi-krb5-2
libk5crypto3 libkeyutils1 libkrb5-3 libkrb5support0 libldap-2.4-2
libldap-common libnghttp2-14 libpcre2-8-0 libperl5.28 libpsl5 librtmp1
libsasl2-2 libsasl2-modules-db libssh2-1 perl perl-modules-5.28
0 upgraded, 22 newly installed, 0 to remove and 16 not upgraded.
Need to get 16.4 MB of archives.
After this operation, 90.1 MB of additional disk space will be used.
Get:1 http://deb.debian.org/debian buster/main amd64 perl-modules-5.28 all 5.28.1-6+deb10u1 [2873 kB]
Get:2 http://deb.debian.org/debian buster/main amd64 libgdbm-compat4 amd64 1.18.1-4 [44.1 kB]
Get:3 http://deb.debian.org/debian buster/main amd64 libperl5.28 amd64 5.28.1-6+deb10u1 [3894 kB]
Get:4 http://deb.debian.org/debian buster/main amd64 perl amd64 5.28.1-6+deb10u1 [204 kB]
Get:5 http://deb.debian.org/debian buster/main amd64 libkeyutils1 amd64 1.6-6 [15.0 kB]
Get:6 http://deb.debian.org/debian buster/main amd64 libkrb5support0 amd64 1.17-3+deb10u3 [65.8 kB]
Get:7 http://deb.debian.org/debian buster/main amd64 libk5crypto3 amd64 1.17-3+deb10u3 [122 kB]
Get:8 http://deb.debian.org/debian buster/main amd64 libkrb5-3 amd64 1.17-3+deb10u3 [370 kB]
Get:9 http://deb.debian.org/debian buster/main amd64 libgssapi-krb5-2 amd64 1.17-3+deb10u3 [158 kB]
Get:10 http://deb.debian.org/debian buster/main amd64 libsasl2-modules-db amd64 2.1.27+dfsg-1+deb10u1 [69.1 kB]
Get:11 http://deb.debian.org/debian buster/main amd64 libsasl2-2 amd64 2.1.27+dfsg-1+deb10u1 [106 kB]
Get:12 http://deb.debian.org/debian buster/main amd64 libldap-common all 2.4.47+dfsg-3+deb10u6 [90.0 kB]
Get:13 http://deb.debian.org/debian buster/main amd64 libldap-2.4-2 amd64 2.4.47+dfsg-3+deb10u6 [224 kB]
Get:14 http://deb.debian.org/debian buster/main amd64 libnghttp2-14 amd64 1.36.0-2+deb10u1 [85.0 kB]
Get:15 http://deb.debian.org/debian buster/main amd64 libpsl5 amd64 0.20.2-2 [53.7 kB]
Get:16 http://deb.debian.org/debian buster/main amd64 librtmp1 amd64 2.4+20151223.gitfa8646d.1-2 [60.5 kB]
Get:17 http://deb.debian.org/debian buster/main amd64 libssh2-1 amd64 1.8.0-2.1 [140 kB]
Get:18 http://deb.debian.org/debian buster/main amd64 libcurl3-gnutls amd64 7.64.0-4+deb10u2 [330 kB]
Get:19 http://deb.debian.org/debian buster/main amd64 libpcre2-8-0 amd64 10.32-5 [213 kB]
Get:20 http://deb.debian.org/debian buster/main amd64 liberror-perl all 0.17027-2 [30.9 kB]
Get:21 http://deb.debian.org/debian buster/main amd64 git-man all 1:2.20.1-2+deb10u3 [1620 kB]
Get:22 http://deb.debian.org/debian buster/main amd64 git amd64 1:2.20.1-2+deb10u3 [5633 kB]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 16.4 MB in 0s (62.3 MB/s)
Selecting previously unselected package perl-modules-5.28.
(Reading database ... 6840 files and directories currently installed.)
Preparing to unpack .../00-perl-modules-5.28_5.28.1-6+deb10u1_all.deb ...
Unpacking perl-modules-5.28 (5.28.1-6+deb10u1) ...
Selecting previously unselected package libgdbm-compat4:amd64.
Preparing to unpack .../01-libgdbm-compat4_1.18.1-4_amd64.deb ...
Unpacking libgdbm-compat4:amd64 (1.18.1-4) ...
Selecting previously unselected package libperl5.28:amd64.
Preparing to unpack .../02-libperl5.28_5.28.1-6+deb10u1_amd64.deb ...
Unpacking libperl5.28:amd64 (5.28.1-6+deb10u1) ...
Selecting previously unselected package perl.
Preparing to unpack .../03-perl_5.28.1-6+deb10u1_amd64.deb ...
Unpacking perl (5.28.1-6+deb10u1) ...
Selecting previously unselected package libkeyutils1:amd64.
Preparing to unpack .../04-libkeyutils1_1.6-6_amd64.deb ...
Unpacking libkeyutils1:amd64 (1.6-6) ...
Selecting previously unselected package libkrb5support0:amd64.
Preparing to unpack .../05-libkrb5support0_1.17-3+deb10u3_amd64.deb ...
Unpacking libkrb5support0:amd64 (1.17-3+deb10u3) ...
Selecting previously unselected package libk5crypto3:amd64.
Preparing to unpack .../06-libk5crypto3_1.17-3+deb10u3_amd64.deb ...
Unpacking libk5crypto3:amd64 (1.17-3+deb10u3) ...
Selecting previously unselected package libkrb5-3:amd64.
Preparing to unpack .../07-libkrb5-3_1.17-3+deb10u3_amd64.deb ...
Unpacking libkrb5-3:amd64 (1.17-3+deb10u3) ...
Selecting previously unselected package libgssapi-krb5-2:amd64.
Preparing to unpack .../08-libgssapi-krb5-2_1.17-3+deb10u3_amd64.deb ...
Unpacking libgssapi-krb5-2:amd64 (1.17-3+deb10u3) ...
Selecting previously unselected package libsasl2-modules-db:amd64.
Preparing to unpack .../09-libsasl2-modules-db_2.1.27+dfsg-1+deb10u1_amd64.deb ...
Unpacking libsasl2-modules-db:amd64 (2.1.27+dfsg-1+deb10u1) ...
Selecting previously unselected package libsasl2-2:amd64.
Preparing to unpack .../10-libsasl2-2_2.1.27+dfsg-1+deb10u1_amd64.deb ...
Unpacking libsasl2-2:amd64 (2.1.27+dfsg-1+deb10u1) ...
Selecting previously unselected package libldap-common.
Preparing to unpack .../11-libldap-common_2.4.47+dfsg-3+deb10u6_all.deb ...
Unpacking libldap-common (2.4.47+dfsg-3+deb10u6) ...
Selecting previously unselected package libldap-2.4-2:amd64.
Preparing to unpack .../12-libldap-2.4-2_2.4.47+dfsg-3+deb10u6_amd64.deb ...
Unpacking libldap-2.4-2:amd64 (2.4.47+dfsg-3+deb10u6) ...
Selecting previously unselected package libnghttp2-14:amd64.
Preparing to unpack .../13-libnghttp2-14_1.36.0-2+deb10u1_amd64.deb ...
Unpacking libnghttp2-14:amd64 (1.36.0-2+deb10u1) ...
Selecting previously unselected package libpsl5:amd64.
Preparing to unpack .../14-libpsl5_0.20.2-2_amd64.deb ...
Unpacking libpsl5:amd64 (0.20.2-2) ...
Selecting previously unselected package librtmp1:amd64.
Preparing to unpack .../15-librtmp1_2.4+20151223.gitfa8646d.1-2_amd64.deb ...
Unpacking librtmp1:amd64 (2.4+20151223.gitfa8646d.1-2) ...
Selecting previously unselected package libssh2-1:amd64.
Preparing to unpack .../16-libssh2-1_1.8.0-2.1_amd64.deb ...
Unpacking libssh2-1:amd64 (1.8.0-2.1) ...
Selecting previously unselected package libcurl3-gnutls:amd64.
Preparing to unpack .../17-libcurl3-gnutls_7.64.0-4+deb10u2_amd64.deb ...
Unpacking libcurl3-gnutls:amd64 (7.64.0-4+deb10u2) ...
Selecting previously unselected package libpcre2-8-0:amd64.
Preparing to unpack .../18-libpcre2-8-0_10.32-5_amd64.deb ...
Unpacking libpcre2-8-0:amd64 (10.32-5) ...
Selecting previously unselected package liberror-perl.
Preparing to unpack .../19-liberror-perl_0.17027-2_all.deb ...
Unpacking liberror-perl (0.17027-2) ...
Selecting previously unselected package git-man.
Preparing to unpack .../20-git-man_1%3a2.20.1-2+deb10u3_all.deb ...
Unpacking git-man (1:2.20.1-2+deb10u3) ...
Selecting previously unselected package git.
Preparing to unpack .../21-git_1%3a2.20.1-2+deb10u3_amd64.deb ...
Unpacking git (1:2.20.1-2+deb10u3) ...
Setting up perl-modules-5.28 (5.28.1-6+deb10u1) ...
Setting up libkeyutils1:amd64 (1.6-6) ...
Setting up libpsl5:amd64 (0.20.2-2) ...
Setting up libnghttp2-14:amd64 (1.36.0-2+deb10u1) ...
Setting up libldap-common (2.4.47+dfsg-3+deb10u6) ...
Setting up libkrb5support0:amd64 (1.17-3+deb10u3) ...
Setting up libsasl2-modules-db:amd64 (2.1.27+dfsg-1+deb10u1) ...
Setting up librtmp1:amd64 (2.4+20151223.gitfa8646d.1-2) ...
Setting up libgdbm-compat4:amd64 (1.18.1-4) ...
Setting up libpcre2-8-0:amd64 (10.32-5) ...
Setting up libk5crypto3:amd64 (1.17-3+deb10u3) ...
Setting up libsasl2-2:amd64 (2.1.27+dfsg-1+deb10u1) ...
Setting up libperl5.28:amd64 (5.28.1-6+deb10u1) ...
Setting up git-man (1:2.20.1-2+deb10u3) ...
Setting up libssh2-1:amd64 (1.8.0-2.1) ...
Setting up libkrb5-3:amd64 (1.17-3+deb10u3) ...
Setting up libldap-2.4-2:amd64 (2.4.47+dfsg-3+deb10u6) ...
Setting up perl (5.28.1-6+deb10u1) ...
Setting up libgssapi-krb5-2:amd64 (1.17-3+deb10u3) ...
Setting up libcurl3-gnutls:amd64 (7.64.0-4+deb10u2) ...
Setting up liberror-perl (0.17027-2) ...
Setting up git (1:2.20.1-2+deb10u3) ...
Processing triggers for libc-bin (2.28-10) ...
INFO[0012] Taking snapshot of full filesystem...
INFO[0015] RUN python -m pip install azureml-core==1.33.0 azureml-train-automl-client==1.33.0
INFO[0015] cmd: /bin/sh
INFO[0015] args: [-c python -m pip install azureml-core==1.33.0 azureml-train-automl-client==1.33.0]
INFO[0015] Running: [/bin/sh -c python -m pip install azureml-core==1.33.0 azureml-train-automl-client==1.33.0]
Collecting azureml-core==1.33.0
Downloading azureml_core-1.33.0-py3-none-any.whl (2.2 MB)
Collecting azureml-train-automl-client==1.33.0
Downloading azureml_train_automl_client-1.33.0-py3-none-any.whl (128 kB)
Collecting pytz
Downloading pytz-2021.3-py2.py3-none-any.whl (503 kB)
Collecting urllib3<=1.26.5,>=1.23
Downloading urllib3-1.26.5-py2.py3-none-any.whl (138 kB)
Collecting pathspec<1.0.0
Downloading pathspec-0.9.0-py2.py3-none-any.whl (31 kB)
Collecting azure-mgmt-containerregistry>=2.0.0
Downloading azure_mgmt_containerregistry-9.0.0-py3-none-any.whl (937 kB)
Collecting contextlib2<1.0.0
Downloading contextlib2-0.6.0.post1-py2.py3-none-any.whl (9.8 kB)
Collecting SecretStorage<4.0.0
Downloading SecretStorage-3.3.1-py3-none-any.whl (15 kB)
Collecting backports.tempfile
Downloading backports.tempfile-1.0-py2.py3-none-any.whl (4.4 kB)
Collecting docker<5.0.0
Downloading docker-4.4.4-py2.py3-none-any.whl (147 kB)
Collecting msrestazure<=0.6.4,>=0.4.33
Downloading msrestazure-0.6.4-py2.py3-none-any.whl (40 kB)
Collecting cryptography!=1.9,!=2.0.*,!=2.1.*,!=2.2.*,<4.0.0
Downloading cryptography-3.4.8-cp36-abi3-manylinux_2_24_x86_64.whl (3.0 MB)
Collecting msrest<1.0.0,>=0.5.1
Downloading msrest-0.6.21-py2.py3-none-any.whl (85 kB)
Collecting requests<3.0.0,>=2.19.1
Downloading requests-2.27.1-py2.py3-none-any.whl (63 kB)
Collecting adal<=1.2.7,>=1.2.0
Downloading adal-1.2.7-py2.py3-none-any.whl (55 kB)
Collecting jsonpickle<3.0.0
Downloading jsonpickle-2.1.0-py2.py3-none-any.whl (38 kB)
Collecting azure-common<2.0.0,>=1.1.12
Downloading azure_common-1.1.27-py2.py3-none-any.whl (12 kB)
Collecting python-dateutil<3.0.0,>=2.7.3
Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
Collecting PyJWT<3.0.0
Downloading PyJWT-2.3.0-py3-none-any.whl (16 kB)
Collecting azure-mgmt-authorization<1.0.0,>=0.40.0
Downloading azure_mgmt_authorization-0.61.0-py2.py3-none-any.whl (94 kB)
Collecting pyopenssl<21.0.0
Downloading pyOpenSSL-20.0.1-py2.py3-none-any.whl (54 kB)
Collecting jmespath<1.0.0
Downloading jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
Collecting ndg-httpsclient<=0.5.1
Downloading ndg_httpsclient-0.5.1-py3-none-any.whl (34 kB)
Collecting azure-mgmt-storage<16.0.0,>=1.5.0
Downloading azure_mgmt_storage-11.2.0-py2.py3-none-any.whl (547 kB)
Collecting azure-mgmt-keyvault<10.0.0,>=0.40.0
Downloading azure_mgmt_keyvault-9.3.0-py2.py3-none-any.whl (412 kB)
Collecting ruamel.yaml<0.17.5,>=0.15.35
Downloading ruamel.yaml-0.17.4-py3-none-any.whl (101 kB)
Collecting azure-graphrbac<1.0.0,>=0.40.0
Downloading azure_graphrbac-0.61.1-py2.py3-none-any.whl (141 kB)
Collecting azure-mgmt-resource<15.0.0,>=1.2.1
Downloading azure_mgmt_resource-13.0.0-py2.py3-none-any.whl (1.3 MB)
Collecting azureml-dataset-runtime~=1.33.0
Downloading azureml_dataset_runtime-1.33.0-py3-none-any.whl (3.5 kB)
Collecting azureml-telemetry~=1.33.0
Downloading azureml_telemetry-1.33.0-py3-none-any.whl (30 kB)
Collecting azureml-automl-core~=1.33.0
Downloading azureml_automl_core-1.33.1-py3-none-any.whl (214 kB)
Collecting azure-mgmt-core<2.0.0,>=1.3.0
Downloading azure_mgmt_core-1.3.0-py2.py3-none-any.whl (25 kB)
Collecting azure-core<2.0.0,>=1.15.0
Downloading azure_core-1.21.1-py2.py3-none-any.whl (178 kB)
Collecting six>=1.11.0
Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting pyarrow<4.0.0,>=0.17.0
Downloading pyarrow-3.0.0-cp37-cp37m-manylinux2014_x86_64.whl (20.7 MB)
Collecting azureml-dataprep<2.21.0a,>=2.20.0a
Downloading azureml_dataprep-2.20.1-py3-none-any.whl (39.4 MB)
Collecting numpy!=1.19.3
Downloading numpy-1.21.5-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (15.7 MB)
Collecting azure-identity<1.5.0,>=1.2.0
Downloading azure_identity-1.4.1-py2.py3-none-any.whl (86 kB)
Collecting dotnetcore2<3.0.0,>=2.1.14
Downloading dotnetcore2-2.1.23-py3-none-manylinux1_x86_64.whl (29.3 MB)
Collecting azureml-dataprep-native<39.0.0,>=38.0.0
Downloading azureml_dataprep_native-38.0.0-cp37-cp37m-manylinux1_x86_64.whl (1.3 MB)
Collecting azureml-dataprep-rslex<1.19.0a,>=1.18.0dev0
Downloading azureml_dataprep_rslex-1.18.2-cp37-cp37m-manylinux1_x86_64.whl (10.4 MB)
Collecting cloudpickle<2.0.0,>=1.1.0
Downloading cloudpickle-1.6.0-py3-none-any.whl (23 kB)
Collecting msal-extensions~=0.2.2
Downloading msal_extensions-0.2.2-py2.py3-none-any.whl (15 kB)
Collecting msal<2.0.0,>=1.3.0
Downloading msal-1.16.0-py2.py3-none-any.whl (78 kB)
Collecting applicationinsights
Downloading applicationinsights-0.11.10-py2.py3-none-any.whl (55 kB)
Collecting cffi>=1.12
Downloading cffi-1.15.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (427 kB)
Collecting pycparser
Downloading pycparser-2.21-py2.py3-none-any.whl (118 kB)
Collecting websocket-client>=0.32.0
Downloading websocket_client-1.2.3-py3-none-any.whl (53 kB)
Collecting distro>=1.2.0
Downloading distro-1.6.0-py2.py3-none-any.whl (19 kB)
Collecting importlib-metadata
Downloading importlib_metadata-4.10.1-py3-none-any.whl (17 kB)
Collecting portalocker~=1.0
Downloading portalocker-1.7.1-py2.py3-none-any.whl (10 kB)
Collecting requests-oauthlib>=0.5.0
Downloading requests_oauthlib-1.3.1-py2.py3-none-any.whl (23 kB)
Collecting isodate>=0.6.0
Downloading isodate-0.6.1-py2.py3-none-any.whl (41 kB)
Collecting certifi>=2017.4.17
Downloading certifi-2021.10.8-py2.py3-none-any.whl (149 kB)
Collecting pyasn1>=0.1.1
Downloading pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting charset-normalizer~=2.0.0
Downloading charset_normalizer-2.0.11-py3-none-any.whl (39 kB)
Collecting idna<4,>=2.5
Downloading idna-3.3-py3-none-any.whl (61 kB)
Collecting oauthlib>=3.0.0
Downloading oauthlib-3.2.0-py3-none-any.whl (151 kB)
Collecting ruamel.yaml.clib>=0.1.2
Downloading ruamel.yaml.clib-0.2.6-cp37-cp37m-manylinux1_x86_64.whl (546 kB)
Collecting jeepney>=0.6
Downloading jeepney-0.7.1-py3-none-any.whl (54 kB)
Collecting backports.weakref
Downloading backports.weakref-1.0.post1-py2.py3-none-any.whl (5.2 kB)
Collecting zipp>=0.5
Downloading zipp-3.7.0-py3-none-any.whl (5.3 kB)
Collecting typing-extensions>=3.6.4
Downloading typing_extensions-4.0.1-py3-none-any.whl (22 kB)
Installing collected packages: pycparser, urllib3, idna, charset-normalizer, cffi, certifi, six, requests, PyJWT, oauthlib, cryptography, requests-oauthlib, python-dateutil, isodate, zipp, typing-extensions, portalocker, msrest, msal, azure-core, adal, websocket-client, ruamel.yaml.clib, pyopenssl, pyasn1, msrestazure, msal-extensions, jeepney, importlib-metadata, distro, backports.weakref, azure-mgmt-core, azure-common, SecretStorage, ruamel.yaml, pytz, pathspec, numpy, ndg-httpsclient, jsonpickle, jmespath, dotnetcore2, docker, contextlib2, cloudpickle, backports.tempfile, azureml-dataprep-rslex, azureml-dataprep-native, azure-mgmt-storage, azure-mgmt-resource, azure-mgmt-keyvault, azure-mgmt-containerregistry, azure-mgmt-authorization, azure-identity, azure-graphrbac, pyarrow, azureml-dataprep, azureml-core, applicationinsights, azureml-telemetry, azureml-dataset-runtime, azureml-automl-core, azureml-train-automl-client
Successfully installed PyJWT-2.3.0 SecretStorage-3.3.1 adal-1.2.7 applicationinsights-0.11.10 azure-common-1.1.27 azure-core-1.21.1 azure-graphrbac-0.61.1 azure-identity-1.4.1 azure-mgmt-authorization-0.61.0 azure-mgmt-containerregistry-9.0.0 azure-mgmt-core-1.3.0 azure-mgmt-keyvault-9.3.0 azure-mgmt-resource-13.0.0 azure-mgmt-storage-11.2.0 azureml-automl-core-1.33.1 azureml-core-1.33.0 azureml-dataprep-2.20.1 azureml-dataprep-native-38.0.0 azureml-dataprep-rslex-1.18.2 azureml-dataset-runtime-1.33.0 azureml-telemetry-1.33.0 azureml-train-automl-client-1.33.0 backports.tempfile-1.0 backports.weakref-1.0.post1 certifi-2021.10.8 cffi-1.15.0 charset-normalizer-2.0.11 cloudpickle-1.6.0 contextlib2-0.6.0.post1 cryptography-3.4.8 distro-1.6.0 docker-4.4.4 dotnetcore2-2.1.23 idna-3.3 importlib-metadata-4.10.1 isodate-0.6.1 jeepney-0.7.1 jmespath-0.10.0 jsonpickle-2.1.0 msal-1.16.0 msal-extensions-0.2.2 msrest-0.6.21 msrestazure-0.6.4 ndg-httpsclient-0.5.1 numpy-1.21.5 oauthlib-3.2.0 pathspec-0.9.0 portalocker-1.7.1 pyarrow-3.0.0 pyasn1-0.4.8 pycparser-2.21 pyopenssl-20.0.1 python-dateutil-2.8.2 pytz-2021.3 requests-2.27.1 requests-oauthlib-1.3.1 ruamel.yaml-0.17.4 ruamel.yaml.clib-0.2.6 six-1.16.0 typing-extensions-4.0.1 urllib3-1.26.5 websocket-client-1.2.3 zipp-3.7.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
WARNING: You are using pip version 21.2.4; however, version 22.0.2 is available.
You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
INFO[0040] Taking snapshot of full filesystem...
INFO[0059] RUN python -m pip install "mlrun[complete] @ git+https://github.com/mlrun/mlrun@development"
INFO[0059] cmd: /bin/sh
INFO[0059] args: [-c python -m pip install "mlrun[complete] @ git+https://github.com/mlrun/mlrun@development"]
INFO[0059] Running: [/bin/sh -c python -m pip install "mlrun[complete] @ git+https://github.com/mlrun/mlrun@development"]
Collecting mlrun[complete]@ git+https://github.com/mlrun/mlrun@development
Cloning https://github.com/mlrun/mlrun (to revision development) to /tmp/pip-install-yegk19ip/mlrun_b9fdf6ee5c8c4e2d9de3a37d0807b6c5
Running command git clone -q https://github.com/mlrun/mlrun /tmp/pip-install-yegk19ip/mlrun_b9fdf6ee5c8c4e2d9de3a37d0807b6c5
Resolved https://github.com/mlrun/mlrun to commit 832a07b11f3198b844d30b4a80db12a45c6e8948
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'done'
Preparing wheel metadata: started
Preparing wheel metadata: finished with status 'done'
Collecting pymysql~=1.0
Downloading PyMySQL-1.0.2-py3-none-any.whl (43 kB)
Requirement already satisfied: pyarrow<6,>=1 in /usr/local/lib/python3.7/site-packages (from mlrun[complete]@ git+https://github.com/mlrun/mlrun@development) (3.0.0)
Collecting fsspec~=2021.8.1
Downloading fsspec-2021.8.1-py3-none-any.whl (119 kB)
Collecting v3io~=0.5.13
Downloading v3io-0.5.15-py3-none-any.whl (49 kB)
Collecting ipykernel~=5.0
Downloading ipykernel-5.5.6-py3-none-any.whl (121 kB)
Collecting click~=7.0
Downloading click-7.1.2-py2.py3-none-any.whl (82 kB)
Collecting sqlalchemy~=1.3
Downloading SQLAlchemy-1.4.31-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.6 MB)
Collecting dask~=2021.11.2
Downloading dask-2021.11.2-py3-none-any.whl (1.0 MB)
Collecting distributed~=2021.11.2
Downloading distributed-2021.11.2-py3-none-any.whl (802 kB)
Collecting chardet<4.0,>=3.0.2
Downloading chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Collecting nuclio-jupyter~=0.8.22
Downloading nuclio_jupyter-0.8.22-py3-none-any.whl (49 kB)
Collecting pydantic~=1.5
Downloading pydantic-1.9.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (10.9 MB)
Collecting v3iofs~=0.1.7
Downloading v3iofs-0.1.10-py3-none-any.whl (13 kB)
Collecting kfp~=1.8.0
Downloading kfp-1.8.11.tar.gz (298 kB)
Collecting pandas~=1.2
Downloading pandas-1.3.5-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (11.3 MB)
Collecting pyyaml~=5.1
Downloading PyYAML-5.4.1-cp37-cp37m-manylinux1_x86_64.whl (636 kB)
Collecting orjson<3.4,>=3
Downloading orjson-3.3.1-cp37-cp37m-manylinux2014_x86_64.whl (208 kB)
Collecting inflection~=0.5.0
Downloading inflection-0.5.1-py2.py3-none-any.whl (9.5 kB)
Collecting aiohttp~=3.8
Downloading aiohttp-3.8.1-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.1 MB)
Collecting humanfriendly~=8.2
Downloading humanfriendly-8.2-py2.py3-none-any.whl (86 kB)
Collecting ipython~=7.0
Downloading ipython-7.31.1-py3-none-any.whl (792 kB)
Collecting fastapi~=0.67.0
Downloading fastapi-0.67.0-py3-none-any.whl (51 kB)
Requirement already satisfied: requests~=2.22 in /usr/local/lib/python3.7/site-packages (from mlrun[complete]@ git+https://github.com/mlrun/mlrun@development) (2.27.1)
Collecting tabulate~=0.8.6
Downloading tabulate-0.8.9-py3-none-any.whl (25 kB)
Collecting nest-asyncio~=1.0
Downloading nest_asyncio-1.5.4-py3-none-any.whl (5.1 kB)
Collecting storey~=0.10.4
Downloading storey-0.10.4-py3-none-any.whl (116 kB)
Requirement already satisfied: urllib3<1.27,>=1.25.4 in /usr/local/lib/python3.7/site-packages (from mlrun[complete]@ git+https://github.com/mlrun/mlrun@development) (1.26.5)
Collecting GitPython~=3.0
Downloading GitPython-3.1.26-py3-none-any.whl (180 kB)
Collecting mergedeep~=1.3
Downloading mergedeep-1.3.4-py3-none-any.whl (6.4 kB)
Collecting semver~=2.13
Downloading semver-2.13.0-py2.py3-none-any.whl (12 kB)
Collecting cryptography<3.4,~=3.0
Downloading cryptography-3.3.2-cp36-abi3-manylinux2010_x86_64.whl (2.6 MB)
Collecting v3io-frames~=0.10.2
Downloading v3io_frames-0.10.2-py3-none-any.whl (35 kB)
Collecting deepdiff~=5.0
Downloading deepdiff-5.7.0-py3-none-any.whl (68 kB)
Requirement already satisfied: numpy<1.22.0,>=1.16.5 in /usr/local/lib/python3.7/site-packages (from mlrun[complete]@ git+https://github.com/mlrun/mlrun@development) (1.21.5)
Collecting python-dotenv~=0.17.0
Downloading python_dotenv-0.17.1-py2.py3-none-any.whl (18 kB)
Collecting alembic<1.6.0,~=1.4
Downloading alembic-1.5.8-py2.py3-none-any.whl (159 kB)
Collecting kubernetes~=12.0
Downloading kubernetes-12.0.1-py2.py3-none-any.whl (1.7 MB)
Collecting google-auth<2.0dev,>=1.25.0
Downloading google_auth-1.35.0-py2.py3-none-any.whl (152 kB)
Collecting azure-identity~=1.5
Downloading azure_identity-1.7.1-py2.py3-none-any.whl (129 kB)
Collecting aiobotocore~=1.4.0
Downloading aiobotocore-1.4.2.tar.gz (52 kB)
Collecting boto3<1.17.107,~=1.9
Downloading boto3-1.17.106-py2.py3-none-any.whl (131 kB)
Collecting azure-keyvault-secrets~=4.2
Downloading azure_keyvault_secrets-4.3.0-py2.py3-none-any.whl (233 kB)
Collecting s3fs~=2021.8.1
Downloading s3fs-2021.8.1-py3-none-any.whl (26 kB)
Collecting plotly~=5.4
Downloading plotly-5.5.0-py2.py3-none-any.whl (26.5 MB)
Collecting botocore<1.20.107,>=1.20.106
Downloading botocore-1.20.106-py2.py3-none-any.whl (7.7 MB)
Collecting adlfs~=2021.8.1
Downloading adlfs-2021.8.2.tar.gz (38 kB)
Collecting gcsfs~=2021.8.1
Downloading gcsfs-2021.8.1-py2.py3-none-any.whl (23 kB)
Collecting azure-storage-blob~=12.0
Downloading azure_storage_blob-12.9.0-py2.py3-none-any.whl (356 kB)
Requirement already satisfied: azure-core>=1.7.0 in /usr/local/lib/python3.7/site-packages (from adlfs~=2021.8.1->mlrun[complete]@ git+https://github.com/mlrun/mlrun@development) (1.21.1)
Collecting azure-datalake-store<0.1,>=0.0.46
Downloading azure_datalake_store-0.0.52-py2.py3-none-any.whl (61 kB)
Collecting wrapt>=1.10.10
Downloading wrapt-1.13.3-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (79 kB)
Collecting aioitertools>=0.5.1
Downloading aioitertools-0.8.0-py3-none-any.whl (21 kB)
Collecting frozenlist>=1.1.1
Downloading frozenlist-1.3.0-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (144 kB)
Collecting yarl<2.0,>=1.0
Downloading yarl-1.7.2-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (271 kB)
Collecting multidict<7.0,>=4.5
Downloading multidict-6.0.2-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (94 kB)
Collecting aiosignal>=1.1.2
Downloading aiosignal-1.2.0-py3-none-any.whl (8.2 kB)
Requirement already satisfied: typing-extensions>=3.7.4 in /usr/local/lib/python3.7/site-packages (from aiohttp~=3.8->mlrun[complete]@ git+https://github.com/mlrun/mlrun@development) (4.0.1)
Collecting async-timeout<5.0,>=4.0.0a3
Downloading async_timeout-4.0.2-py3-none-any.whl (5.8 kB)
Collecting asynctest==0.13.0
Downloading asynctest-0.13.0-py3-none-any.whl (26 kB)
Collecting attrs>=17.3.0
Downloading attrs-21.4.0-py2.py3-none-any.whl (60 kB)
Requirement already satisfied: charset-normalizer<3.0,>=2.0 in /usr/local/lib/python3.7/site-packages (from aiohttp~=3.8->mlrun[complete]@ git+https://github.com/mlrun/mlrun@development) (2.0.11)
Collecting Mako
Downloading Mako-1.1.6-py2.py3-none-any.whl (75 kB)
Collecting python-editor>=0.3
Downloading python_editor-1.0.4-py3-none-any.whl (4.9 kB)
Requirement already satisfied: python-dateutil in /usr/local/lib/python3.7/site-packages (from alembic<1.6.0,~=1.4->mlrun[complete]@ git+https://github.com/mlrun/mlrun@development) (2.8.2)
Requirement already satisfied: six>=1.11.0 in /usr/local/lib/python3.7/site-packages (from azure-core>=1.7.0->adlfs~=2021.8.1->mlrun[complete]@ git+https://github.com/mlrun/mlrun@development) (1.16.0)
Requirement already satisfied: cffi in /usr/local/lib/python3.7/site-packages (from azure-datalake-store<0.1,>=0.0.46->adlfs~=2021.8.1->mlrun[complete]@ git+https://github.com/mlrun/mlrun@development) (1.15.0)
Requirement already satisfied: adal>=0.4.2 in /usr/local/lib/python3.7/site-packages (from azure-datalake-store<0.1,>=0.0.46->adlfs~=2021.8.1->mlrun[complete]@ git+https://github.com/mlrun/mlrun@development) (1.2.7)
Requirement already satisfied: PyJWT<3,>=1.0.0 in /usr/local/lib/python3.7/site-packages (from adal>=0.4.2->azure-datalake-store<0.1,>=0.0.46->adlfs~=2021.8.1->mlrun[complete]@ git+https://github.com/mlrun/mlrun@development) (2.3.0)
Requirement already satisfied: msal<2.0.0,>=1.12.0 in /usr/local/lib/python3.7/site-packages (from azure-identity~=1.5->mlrun[complete]@ git+https://github.com/mlrun/mlrun@development) (1.16.0)
Collecting msal-extensions~=0.3.0
Downloading msal_extensions-0.3.1-py2.py3-none-any.whl (18 kB)
Requirement already satisfied: azure-common~=1.1 in /usr/local/lib/python3.7/site-packages (from azure-keyvault-secrets~=4.2->mlrun[complete]@ git+https://github.com/mlrun/mlrun@development) (1.1.27)
Requirement already satisfied: msrest>=0.6.21 in /usr/local/lib/python3.7/site-packages (from azure-keyvault-secrets~=4.2->mlrun[complete]@ git+https://github.com/mlrun/mlrun@development) (0.6.21)
Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /usr/local/lib/python3.7/site-packages (from boto3<1.17.107,~=1.9->mlrun[complete]@ git+https://github.com/mlrun/mlrun@development) (0.10.0)
Collecting s3transfer<0.5.0,>=0.4.0
Downloading s3transfer-0.4.2-py2.py3-none-any.whl (79 kB)
Requirement already satisfied: pycparser in /usr/local/lib/python3.7/site-packages (from cffi->azure-datalake-store<0.1,>=0.0.46->adlfs~=2021.8.1->mlrun[complete]@ git+https://github.com/mlrun/mlrun@development) (2.21)
Collecting partd>=0.3.10
Downloading partd-1.2.0-py3-none-any.whl (19 kB)
Collecting toolz>=0.8.2
Downloading toolz-0.11.2-py3-none-any.whl (55 kB)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.7/site-packages (from dask~=2021.11.2->mlrun[complete]@ git+https://github.com/mlrun/mlrun@development) (1.6.0)
Collecting packaging>=20.0
Downloading packaging-21.3-py3-none-any.whl (40 kB)
Collecting ordered-set==4.0.2
Downloading ordered-set-4.0.2.tar.gz (10 kB)
Collecting tornado>=5
Downloading tornado-6.1-cp37-cp37m-manylinux2010_x86_64.whl (428 kB)
Collecting msgpack>=0.6.0
Downloading msgpack-1.0.3-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (299 kB)
Collecting zict>=0.1.3
Downloading zict-2.0.0-py3-none-any.whl (10 kB)
Requirement already satisfied: setuptools in /usr/local/lib/python3.7/site-packages (from distributed~=2021.11.2->mlrun[complete]@ git+https://github.com/mlrun/mlrun@development) (53.0.0)
Collecting psutil>=5.0
Downloading psutil-5.9.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (280 kB)
Collecting jinja2
Downloading Jinja2-3.0.3-py3-none-any.whl (133 kB)
Collecting tblib>=1.6.0
Downloading tblib-1.7.0-py2.py3-none-any.whl (12 kB)
Collecting sortedcontainers!=2.0.0,!=2.0.1
Downloading sortedcontainers-2.4.0-py2.py3-none-any.whl (29 kB)
Collecting starlette==0.14.2
Downloading starlette-0.14.2-py3-none-any.whl (60 kB)
Collecting decorator
Downloading decorator-5.1.1-py3-none-any.whl (9.1 kB)
Collecting google-auth-oauthlib
Downloading google_auth_oauthlib-0.4.6-py2.py3-none-any.whl (18 kB)
Collecting gitdb<5,>=4.0.1
Downloading gitdb-4.0.9-py3-none-any.whl (63 kB)
Collecting smmap<6,>=3.0.1
Downloading smmap-5.0.0-py3-none-any.whl (24 kB)
Collecting pyasn1-modules>=0.2.1
Downloading pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting rsa<5,>=3.1.4
Downloading rsa-4.8-py3-none-any.whl (39 kB)
Collecting cachetools<5.0,>=2.0.0
Downloading cachetools-4.2.4-py3-none-any.whl (10 kB)
Collecting jupyter-client
Downloading jupyter_client-7.1.2-py3-none-any.whl (130 kB)
Collecting ipython-genutils
Downloading ipython_genutils-0.2.0-py2.py3-none-any.whl (26 kB)
Collecting traitlets>=4.1.0
Downloading traitlets-5.1.1-py3-none-any.whl (102 kB)
Collecting prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0
Downloading prompt_toolkit-3.0.26-py3-none-any.whl (375 kB)
Collecting jedi>=0.16
Downloading jedi-0.18.1-py2.py3-none-any.whl (1.6 MB)
Collecting matplotlib-inline
Downloading matplotlib_inline-0.1.3-py3-none-any.whl (8.2 kB)
Collecting pickleshare
Downloading pickleshare-0.7.5-py2.py3-none-any.whl (6.9 kB)
Collecting pygments
Downloading Pygments-2.11.2-py3-none-any.whl (1.1 MB)
Collecting backcall
Downloading backcall-0.2.0-py2.py3-none-any.whl (11 kB)
Collecting pexpect>4.3
Downloading pexpect-4.8.0-py2.py3-none-any.whl (59 kB)
Collecting parso<0.9.0,>=0.8.0
Downloading parso-0.8.3-py2.py3-none-any.whl (100 kB)
Collecting absl-py<2,>=0.9
Downloading absl_py-1.0.0-py3-none-any.whl (126 kB)
Collecting google-cloud-storage<2,>=1.20.0
Downloading google_cloud_storage-1.44.0-py2.py3-none-any.whl (106 kB)
Collecting google-api-python-client<2,>=1.7.8
Downloading google_api_python_client-1.12.10-py2.py3-none-any.whl (61 kB)
Collecting requests-toolbelt<1,>=0.8.0
Downloading requests_toolbelt-0.9.1-py2.py3-none-any.whl (54 kB)
Collecting cloudpickle>=1.1.1
Downloading cloudpickle-2.0.0-py3-none-any.whl (25 kB)
Collecting kfp-server-api<2.0.0,>=1.1.2
Downloading kfp-server-api-1.7.1.tar.gz (52 kB)
Collecting jsonschema<4,>=3.0.1
Downloading jsonschema-3.2.0-py2.py3-none-any.whl (56 kB)
Collecting Deprecated<2,>=1.2.7
Downloading Deprecated-1.2.13-py2.py3-none-any.whl (9.6 kB)
Collecting strip-hints<1,>=0.1.8
Downloading strip-hints-0.1.10.tar.gz (29 kB)
Collecting docstring-parser<1,>=0.7.3
Downloading docstring_parser-0.13.tar.gz (23 kB)
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'done'
Preparing wheel metadata: started
Preparing wheel metadata: finished with status 'done'
Collecting kfp-pipeline-spec<0.2.0,>=0.1.13
Downloading kfp_pipeline_spec-0.1.13-py3-none-any.whl (18 kB)
Collecting fire<1,>=0.3.1
Downloading fire-0.4.0.tar.gz (87 kB)
Collecting protobuf<4,>=3.13.0
Downloading protobuf-3.19.4-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.1 MB)
Collecting uritemplate<4,>=3.0.1
Downloading uritemplate-3.0.1-py2.py3-none-any.whl (15 kB)
Collecting typer<1.0,>=0.3.2
Downloading typer-0.4.0-py3-none-any.whl (27 kB)
Collecting typing-extensions>=3.7.4
Downloading typing_extensions-3.10.0.2-py3-none-any.whl (26 kB)
Collecting termcolor
Downloading termcolor-1.1.0.tar.gz (3.9 kB)
Collecting httplib2<1dev,>=0.15.0
Downloading httplib2-0.20.2-py3-none-any.whl (96 kB)
Collecting google-api-core<3dev,>=1.21.0
Downloading google_api_core-2.4.0-py2.py3-none-any.whl (111 kB)
Collecting google-auth-httplib2>=0.0.3
Downloading google_auth_httplib2-0.1.0-py2.py3-none-any.whl (9.3 kB)
Collecting googleapis-common-protos<2.0dev,>=1.52.0
Downloading googleapis_common_protos-1.54.0-py2.py3-none-any.whl (207 kB)
Collecting google-cloud-core<3.0dev,>=1.6.0
Downloading google_cloud_core-2.2.2-py2.py3-none-any.whl (29 kB)
Collecting google-resumable-media<3.0dev,>=1.3.0
Downloading google_resumable_media-2.1.0-py2.py3-none-any.whl (75 kB)
Collecting google-crc32c<2.0dev,>=1.0
Downloading google_crc32c-1.3.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (38 kB)
Collecting pyparsing!=3.0.0,!=3.0.1,!=3.0.2,!=3.0.3,<4,>=2.4.2
Downloading pyparsing-3.0.7-py3-none-any.whl (98 kB)
Requirement already satisfied: importlib-metadata in /usr/local/lib/python3.7/site-packages (from jsonschema<4,>=3.0.1->kfp~=1.8.0->mlrun[complete]@ git+https://github.com/mlrun/mlrun@development) (4.10.1)
Collecting pyrsistent>=0.14.0
Downloading pyrsistent-0.18.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (117 kB)
Requirement already satisfied: certifi in /usr/local/lib/python3.7/site-packages (from kfp-server-api<2.0.0,>=1.1.2->kfp~=1.8.0->mlrun[complete]@ git+https://github.com/mlrun/mlrun@development) (2021.10.8)
Requirement already satisfied: requests-oauthlib in /usr/local/lib/python3.7/site-packages (from kubernetes~=12.0->mlrun[complete]@ git+https://github.com/mlrun/mlrun@development) (1.3.1)
Requirement already satisfied: websocket-client!=0.40.0,!=0.41.*,!=0.42.*,>=0.32.0 in /usr/local/lib/python3.7/site-packages (from kubernetes~=12.0->mlrun[complete]@ git+https://github.com/mlrun/mlrun@development) (1.2.3)
Requirement already satisfied: portalocker<3,>=1.0 in /usr/local/lib/python3.7/site-packages (from msal-extensions~=0.3.0->azure-identity~=1.5->mlrun[complete]@ git+https://github.com/mlrun/mlrun@development) (1.7.1)
Requirement already satisfied: isodate>=0.6.0 in /usr/local/lib/python3.7/site-packages (from msrest>=0.6.21->azure-keyvault-secrets~=4.2->mlrun[complete]@ git+https://github.com/mlrun/mlrun@development) (0.6.1)
Collecting notebook>=5.2.0
Downloading notebook-6.4.8-py3-none-any.whl (9.9 MB)
Collecting nbconvert>=5.4
Downloading nbconvert-6.4.1-py3-none-any.whl (557 kB)
Collecting nbformat>=4.4
Downloading nbformat-5.1.3-py3-none-any.whl (178 kB)
Collecting mistune<2,>=0.8.1
Downloading mistune-0.8.4-py2.py3-none-any.whl (16 kB)
Collecting defusedxml
Downloading defusedxml-0.7.1-py2.py3-none-any.whl (25 kB)
Collecting nbclient<0.6.0,>=0.5.0
Downloading nbclient-0.5.10-py3-none-any.whl (69 kB)
Collecting jupyter-core
Downloading jupyter_core-4.9.1-py3-none-any.whl (86 kB)
Collecting testpath
Downloading testpath-0.5.0-py3-none-any.whl (84 kB)
Collecting pandocfilters>=1.4.1
Downloading pandocfilters-1.5.0-py2.py3-none-any.whl (8.7 kB)
Collecting bleach
Downloading bleach-4.1.0-py2.py3-none-any.whl (157 kB)
Collecting jupyterlab-pygments
Downloading jupyterlab_pygments-0.1.2-py2.py3-none-any.whl (4.6 kB)
Collecting entrypoints>=0.2.2
Downloading entrypoints-0.3-py2.py3-none-any.whl (11 kB)
Collecting MarkupSafe>=2.0
Downloading MarkupSafe-2.0.1-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (31 kB)
Collecting pyzmq>=13
Downloading pyzmq-22.3.0-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl (1.1 MB)
Collecting terminado>=0.8.3
Downloading terminado-0.13.1-py3-none-any.whl (14 kB)
Collecting argon2-cffi
Downloading argon2_cffi-21.3.0-py3-none-any.whl (14 kB)
Collecting Send2Trash>=1.8.0
Downloading Send2Trash-1.8.0-py3-none-any.whl (18 kB)
Collecting prometheus-client
Downloading prometheus_client-0.13.1-py3-none-any.whl (57 kB)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.7/site-packages (from pandas~=1.2->mlrun[complete]@ git+https://github.com/mlrun/mlrun@development) (2021.3)
Collecting locket
Downloading locket-0.2.1-py2.py3-none-any.whl (4.1 kB)
Collecting ptyprocess>=0.5
Downloading ptyprocess-0.7.0-py2.py3-none-any.whl (13 kB)
Collecting tenacity>=6.2.0
Downloading tenacity-8.0.1-py3-none-any.whl (24 kB)
Collecting wcwidth
Downloading wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /usr/local/lib/python3.7/site-packages (from pyasn1-modules>=0.2.1->google-auth<2.0dev,>=1.25.0->mlrun[complete]@ git+https://github.com/mlrun/mlrun@development) (0.4.8)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.7/site-packages (from requests~=2.22->mlrun[complete]@ git+https://github.com/mlrun/mlrun@development) (3.3)
Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.7/site-packages (from requests-oauthlib->kubernetes~=12.0->mlrun[complete]@ git+https://github.com/mlrun/mlrun@development) (3.2.0)
Collecting greenlet!=0.4.17
Downloading greenlet-1.1.2-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (150 kB)
Collecting grpcio-tools<1.42,>1.34.0
Downloading grpcio_tools-1.41.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.4 MB)
Collecting grpcio<1.42,>1.34.0
Downloading grpcio-1.41.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.9 MB)
Requirement already satisfied: wheel in /usr/local/lib/python3.7/site-packages (from strip-hints<1,>=0.1.8->kfp~=1.8.0->mlrun[complete]@ git+https://github.com/mlrun/mlrun@development) (0.36.2)
Collecting ujson>=3.0.0
Downloading ujson-5.1.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (43 kB)
Collecting future>=0.18.2
Downloading future-0.18.2.tar.gz (829 kB)
Collecting heapdict
Downloading HeapDict-1.0.1-py3-none-any.whl (3.9 kB)
Collecting argon2-cffi-bindings
Downloading argon2_cffi_bindings-21.2.0-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (86 kB)
Collecting webencodings
Downloading webencodings-0.5.1-py2.py3-none-any.whl (11 kB)
Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/site-packages (from importlib-metadata->jsonschema<4,>=3.0.1->kfp~=1.8.0->mlrun[complete]@ git+https://github.com/mlrun/mlrun@development) (3.7.0)
Building wheels for collected packages: adlfs, aiobotocore, ordered-set, kfp, docstring-parser, fire, kfp-server-api, strip-hints, future, mlrun, termcolor
Building wheel for adlfs (setup.py): started
Building wheel for adlfs (setup.py): finished with status 'done'
Created wheel for adlfs: filename=adlfs-2021.8.2-py3-none-any.whl size=21466 sha256=d3076fc23e1d05f49958ba450b9913a80c23c755d3347327d1d2150656fa3185
Stored in directory: /root/.cache/pip/wheels/0d/88/1d/e06072abb7fb4d59b5cf94e194e53017dfa2dc47af4dec88b7
Building wheel for aiobotocore (setup.py): started
Building wheel for aiobotocore (setup.py): finished with status 'done'
Created wheel for aiobotocore: filename=aiobotocore-1.4.2-py3-none-any.whl size=49910 sha256=d630bfe25d72229a76e207cb2de8dd29839368673c27edc12229b314660a7e69
Stored in directory: /root/.cache/pip/wheels/33/e7/d9/b297a9aa9c43d56bc2463e6e2771655ff638f30b30f0b61fcb
Building wheel for ordered-set (setup.py): started
Building wheel for ordered-set (setup.py): finished with status 'done'
Created wheel for ordered-set: filename=ordered_set-4.0.2-py2.py3-none-any.whl size=8210 sha256=d0a6fcb9c69107866ca516cb90c3a6a1c0dd00c2f55e981d69b476e92fe85c0a
Stored in directory: /root/.cache/pip/wheels/73/2b/f6/26e9f84153c25050fe7c09e88f8e32a6be3c7034a38c418319
Building wheel for kfp (setup.py): started
Building wheel for kfp (setup.py): finished with status 'done'
Created wheel for kfp: filename=kfp-1.8.11-py3-none-any.whl size=414450 sha256=62bc86dbc4fbb6d431756182a297f87d5d9f08edfb8c2bab347b81fe0654cad3
Stored in directory: /root/.cache/pip/wheels/85/1e/ee/a14b49663bddf9e72d1c269cbe53970167bfabb53cadbbea3a
Building wheel for docstring-parser (PEP 517): started
Building wheel for docstring-parser (PEP 517): finished with status 'done'
Created wheel for docstring-parser: filename=docstring_parser-0.13-py3-none-any.whl size=31866 sha256=6ab7172ddcc24d27d93d31af6438a00053464e76585907654ec90dfb3ecf1886
Stored in directory: /root/.cache/pip/wheels/bd/88/3c/d1aa049309f7945178cac9fbe6561a86424f432da57c18ca0f
Building wheel for fire (setup.py): started
Building wheel for fire (setup.py): finished with status 'done'
Created wheel for fire: filename=fire-0.4.0-py2.py3-none-any.whl size=115928 sha256=6704c4bed4908d06fbe2e61f97718607c4ef9dee5a573d09acf9882e6e620757
Stored in directory: /root/.cache/pip/wheels/8a/67/fb/2e8a12fa16661b9d5af1f654bd199366799740a85c64981226
Building wheel for kfp-server-api (setup.py): started
Building wheel for kfp-server-api (setup.py): finished with status 'done'
Created wheel for kfp-server-api: filename=kfp_server_api-1.7.1-py3-none-any.whl size=92618 sha256=b156a487ea471572b7c0d0dc85826bb5f6554ce5d097c3d16cdd75e36a93100a
Stored in directory: /root/.cache/pip/wheels/68/3f/d5/734c0278dd6c8969cef359edcf059505a61452c5eb0e2760e1
Building wheel for strip-hints (setup.py): started
Building wheel for strip-hints (setup.py): finished with status 'done'
Created wheel for strip-hints: filename=strip_hints-0.1.10-py2.py3-none-any.whl size=22279 sha256=ebdf8455bb18636c3ffe99dc1ffe7262298094e000e46a15df3ef1b809e3770f
Stored in directory: /root/.cache/pip/wheels/5e/14/c3/6e44e9b2545f2d570b03f5b6d38c00b7534aa8abb376978363
Building wheel for future (setup.py): started
Building wheel for future (setup.py): finished with status 'done'
Created wheel for future: filename=future-0.18.2-py3-none-any.whl size=491059 sha256=b71c260b8cae9faa06e701eb03743481d68f427d7ed0886bddf8eec6fab17927
Stored in directory: /root/.cache/pip/wheels/56/b0/fe/4410d17b32f1f0c3cf54cdfb2bc04d7b4b8f4ae377e2229ba0
Building wheel for mlrun (PEP 517): started
Building wheel for mlrun (PEP 517): finished with status 'done'
Created wheel for mlrun: filename=mlrun-0.0.0+unstable-py3-none-any.whl size=799835 sha256=a37564dbf60ba19531146c0404fa02216cea4492894cbafd9738bb199ce45775
Stored in directory: /tmp/pip-ephem-wheel-cache-xz3ex0pi/wheels/cd/42/82/13965317128ea26acc3fb21b24cc254077454998599db6f161
Building wheel for termcolor (setup.py): started
Building wheel for termcolor (setup.py): finished with status 'done'
Created wheel for termcolor: filename=termcolor-1.1.0-py3-none-any.whl size=4829 sha256=09ed568b0b6ea586b107bc3decf5a736c95331052246548b5a560c69a88e9414
Stored in directory: /root/.cache/pip/wheels/3f/e3/ec/8a8336ff196023622fbcb36de0c5a5c218cbb24111d1d4c7f2
Successfully built adlfs aiobotocore ordered-set kfp docstring-parser fire kfp-server-api strip-hints future mlrun termcolor
Installing collected packages: typing-extensions, traitlets, pyrsistent, attrs, wcwidth, tornado, rsa, pyzmq, pyparsing, pyasn1-modules, ptyprocess, protobuf, parso, nest-asyncio, jupyter-core, jsonschema, ipython-genutils, entrypoints, cachetools, webencodings, pygments, prompt-toolkit, pickleshare, pexpect, packaging, nbformat, matplotlib-inline, MarkupSafe, jupyter-client, jedi, googleapis-common-protos, google-auth, decorator, cryptography, backcall, ujson, toolz, testpath, pandocfilters, nbclient, multidict, mistune, locket, jupyterlab-pygments, jinja2, ipython, httplib2, grpcio, google-crc32c, google-api-core, future, frozenlist, defusedxml, botocore, bleach, argon2-cffi-bindings, yarl, wrapt, v3io, uritemplate, terminado, termcolor, smmap, Send2Trash, s3transfer, pyyaml, prometheus-client, partd, pandas, nbconvert, ipykernel, heapdict, grpcio-tools, greenlet, google-resumable-media, google-cloud-core, google-auth-httplib2, fsspec, cloudpickle, click, asynctest, async-timeout, argon2-cffi, aiosignal, zict, v3iofs, v3io-frames, typer, tblib, tabulate, strip-hints, starlette, sqlalchemy, sortedcontainers, requests-toolbelt, python-editor, pydantic, psutil, ordered-set, notebook, msgpack, msal-extensions, Mako, kubernetes, kfp-server-api, kfp-pipeline-spec, google-cloud-storage, google-api-python-client, gitdb, fire, docstring-parser, Deprecated, dask, boto3, aioitertools, aiohttp, absl-py, tenacity, storey, semver, python-dotenv, pymysql, orjson, nuclio-jupyter, mergedeep, kfp, inflection, humanfriendly, google-auth-oauthlib, GitPython, fastapi, distributed, deepdiff, chardet, azure-storage-blob, azure-identity, azure-datalake-store, alembic, aiobotocore, s3fs, plotly, mlrun, gcsfs, azure-keyvault-secrets, adlfs
Attempting uninstall: typing-extensions
Found existing installation: typing-extensions 4.0.1
Uninstalling typing-extensions-4.0.1:
Successfully uninstalled typing-extensions-4.0.1
Attempting uninstall: cryptography
Found existing installation: cryptography 3.4.8
Uninstalling cryptography-3.4.8:
Successfully uninstalled cryptography-3.4.8
Attempting uninstall: cloudpickle
Found existing installation: cloudpickle 1.6.0
Uninstalling cloudpickle-1.6.0:
Successfully uninstalled cloudpickle-1.6.0
Attempting uninstall: msal-extensions
Found existing installation: msal-extensions 0.2.2
Uninstalling msal-extensions-0.2.2:
Successfully uninstalled msal-extensions-0.2.2
Attempting uninstall: azure-identity
Found existing installation: azure-identity 1.4.1
Uninstalling azure-identity-1.4.1:
Successfully uninstalled azure-identity-1.4.1
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
azureml-dataprep 2.20.1 requires azure-identity<1.5.0,>=1.2.0, but you have azure-identity 1.7.1 which is incompatible.
azureml-dataprep 2.20.1 requires cloudpickle<2.0.0,>=1.1.0, but you have cloudpickle 2.0.0 which is incompatible.
Successfully installed Deprecated-1.2.13 GitPython-3.1.26 Mako-1.1.6 MarkupSafe-2.0.1 Send2Trash-1.8.0 absl-py-1.0.0 adlfs-2021.8.2 aiobotocore-1.4.2 aiohttp-3.8.1 aioitertools-0.8.0 aiosignal-1.2.0 alembic-1.5.8 argon2-cffi-21.3.0 argon2-cffi-bindings-21.2.0 async-timeout-4.0.2 asynctest-0.13.0 attrs-21.4.0 azure-datalake-store-0.0.52 azure-identity-1.7.1 azure-keyvault-secrets-4.3.0 azure-storage-blob-12.9.0 backcall-0.2.0 bleach-4.1.0 boto3-1.17.106 botocore-1.20.106 cachetools-4.2.4 chardet-3.0.4 click-7.1.2 cloudpickle-2.0.0 cryptography-3.3.2 dask-2021.11.2 decorator-5.1.1 deepdiff-5.7.0 defusedxml-0.7.1 distributed-2021.11.2 docstring-parser-0.13 entrypoints-0.3 fastapi-0.67.0 fire-0.4.0 frozenlist-1.3.0 fsspec-2021.8.1 future-0.18.2 gcsfs-2021.8.1 gitdb-4.0.9 google-api-core-2.4.0 google-api-python-client-1.12.10 google-auth-1.35.0 google-auth-httplib2-0.1.0 google-auth-oauthlib-0.4.6 google-cloud-core-2.2.2 google-cloud-storage-1.44.0 google-crc32c-1.3.0 google-resumable-media-2.1.0 googleapis-common-protos-1.54.0 greenlet-1.1.2 grpcio-1.41.1 grpcio-tools-1.41.1 heapdict-1.0.1 httplib2-0.20.2 humanfriendly-8.2 inflection-0.5.1 ipykernel-5.5.6 ipython-7.31.1 ipython-genutils-0.2.0 jedi-0.18.1 jinja2-3.0.3 jsonschema-3.2.0 jupyter-client-7.1.2 jupyter-core-4.9.1 jupyterlab-pygments-0.1.2 kfp-1.8.11 kfp-pipeline-spec-0.1.13 kfp-server-api-1.7.1 kubernetes-12.0.1 locket-0.2.1 matplotlib-inline-0.1.3 mergedeep-1.3.4 mistune-0.8.4 mlrun-0.0.0+unstable msal-extensions-0.3.1 msgpack-1.0.3 multidict-6.0.2 nbclient-0.5.10 nbconvert-6.4.1 nbformat-5.1.3 nest-asyncio-1.5.4 notebook-6.4.8 nuclio-jupyter-0.8.22 ordered-set-4.0.2 orjson-3.3.1 packaging-21.3 pandas-1.3.5 pandocfilters-1.5.0 parso-0.8.3 partd-1.2.0 pexpect-4.8.0 pickleshare-0.7.5 plotly-5.5.0 prometheus-client-0.13.1 prompt-toolkit-3.0.26 protobuf-3.19.4 psutil-5.9.0 ptyprocess-0.7.0 pyasn1-modules-0.2.8 pydantic-1.9.0 pygments-2.11.2 pymysql-1.0.2 pyparsing-3.0.7 pyrsistent-0.18.1 python-dotenv-0.17.1 python-editor-1.0.4 pyyaml-5.4.1 pyzmq-22.3.0 requests-toolbelt-0.9.1 rsa-4.8 s3fs-2021.8.1 s3transfer-0.4.2 semver-2.13.0 smmap-5.0.0 sortedcontainers-2.4.0 sqlalchemy-1.4.31 starlette-0.14.2 storey-0.10.4 strip-hints-0.1.10 tabulate-0.8.9 tblib-1.7.0 tenacity-8.0.1 termcolor-1.1.0 terminado-0.13.1 testpath-0.5.0 toolz-0.11.2 tornado-6.1 traitlets-5.1.1 typer-0.4.0 typing-extensions-3.10.0.2 ujson-5.1.0 uritemplate-3.0.1 v3io-0.5.15 v3io-frames-0.10.2 v3iofs-0.1.10 wcwidth-0.2.5 webencodings-0.5.1 wrapt-1.13.3 yarl-1.7.2 zict-2.0.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
WARNING: You are using pip version 21.2.4; however, version 22.0.2 is available.
You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
INFO[0128] Taking snapshot of full filesystem...
INFO[0148] Pushing image to docker-registry.default-tenant.app.yh41.iguazio-cd1.com/mlrun/func-azureml-yonatan-azureml-utils:latest
INFO[0153] Pushed image to 1 destinations
> 2022-02-02 18:30:56,789 [info] starting run azureml-utils-train uid=48dbbe26a2a34b5baaec5ca8aba3de5e DB=http://mlrun-api:8080
> 2022-02-02 18:30:56,988 [info] Job is running in the background, pod: azureml-utils-train-7pp86
> 2022-02-02 18:31:30,311 [warning] Failed resolving version info. Ignoring and using defaults
> 2022-02-02 18:31:32,893 [warning] Server or client version is unstable. Assuming compatible: {'server_version': '0.0.0+unstable', 'client_version': '0.0.0+unstable'}
Failure while loading azureml_run_type_providers. Failed to load entrypoint automl = azureml.train.automl.run:AutoMLRun._from_run_dto with exception (cloudpickle 2.0.0 (/usr/local/lib/python3.7/site-packages), Requirement.parse('cloudpickle<2.0.0,>=1.1.0'), {'azureml-dataprep'}).
> 2022-02-02 18:31:34,680 [info] Loading AzureML Workspace
> 2022-02-02 18:31:36,956 [info] Initializing AzureML experiment azure-automl-test
> 2022-02-02 18:31:40,005 [info] Initializing AzureML compute target azureml-cpu
> 2022-02-02 18:31:40,206 [info] Found existing cluster, will use it.
Succeeded
AmlCompute wait for completion finished
Minimum number of nodes requested have been provisioned
> 2022-02-02 18:31:40,322 [info] Connecting to AzureML experiment default datastore
> 2022-02-02 18:31:41,624 [info] Retrieving feature vector and uploading to Azure blob storage: az://azureml-blobstore-27f8977b-4946-4ca0-bdc5-5a685d2fe8d7/iris.csv
> 2022-02-02 18:31:41,912 [info] Registering dataset iris in Azure ML
> 2022-02-02 18:31:41,912 [info] OpenSSL version must be 1.1. Overriding the OpenSSL version to 1.1
> 2022-02-02 18:31:49,558 [info] Setting up experiment parameters
> 2022-02-02 18:31:49,812 [info] Submitting and running experiment
Submitting remote run.
Parent Run ID: AutoML_35c51a81-98fd-44fb-aa23-3192c3aca08d
https://ml.azure.com/runs/AutoML_35c51a81-98fd-44fb-aa23-3192c3aca08d?wsid=/subscriptions/8d81bc0b-6abd-4395-be83-000251d9fdbe/resourcegroups/nick/workspaces/NickAzureML&tid=af053911-a8b7-450d-9f58-0c08567d4769
Current status: ModelSelection. Beginning model selection.
****************************************************************************************************
DATA GUARDRAILS:
TYPE: Class balancing detection
STATUS: PASSED
DESCRIPTION: Your inputs were analyzed, and all classes are balanced in your training data.
Learn more about imbalanced data: https://aka.ms/AutomatedMLImbalancedData
****************************************************************************************************
****************************************************************************************************
ITERATION: The iteration being evaluated.
PIPELINE: A summary description of the pipeline being evaluated.
DURATION: Time taken for the current iteration.
METRIC: The result of computing score on the fitted pipeline.
BEST: The best observed score thus far.
****************************************************************************************************
ITERATION PIPELINE DURATION METRIC BEST
0 RobustScaler LogisticRegression 0:00:21 0.9667 0.9667
1 StandardScalerWrapper SVM 0:00:18 0.9533 0.9667
2 StandardScalerWrapper LogisticRegression 0:00:22 0.9733 0.9733
3 MaxAbsScaler LogisticRegression 0:00:22 0.9533 0.9733
4 MaxAbsScaler LogisticRegression 0:00:21 0.9733 0.9733
********************************************************************************************
> 2022-02-02 18:38:28,144 [info] Registering model
> 2022-02-02 18:38:29,495 [info] Registered model with name 'iris-model', id 'iris-model:178', version '178'
> 2022-02-02 18:38:29,495 [info] Downloading model iris-model:178
> 2022-02-02 18:38:34,083 [info] Logging model_1_standardscaler_logisticregression model to MLRun
> 2022-02-02 18:38:34,621 [info] Registering model
> 2022-02-02 18:38:35,519 [info] Registered model with name 'iris-model', id 'iris-model:179', version '179'
> 2022-02-02 18:38:35,519 [info] Downloading model iris-model:179
> 2022-02-02 18:38:39,972 [info] Logging model_2_maxabsscaler_logisticregression model to MLRun
> 2022-02-02 18:38:40,317 [info] Registering model
> 2022-02-02 18:38:41,087 [info] Registered model with name 'iris-model', id 'iris-model:180', version '180'
> 2022-02-02 18:38:41,087 [info] Downloading model iris-model:180
> 2022-02-02 18:38:46,299 [info] Logging model_3_robustscaler_logisticregression model to MLRun
> 2022-02-02 18:38:47,615 [info] run executed, status=completed
final state: completed
| project | uid | iter | start | state | name | labels | inputs | parameters | results | artifacts |
|---|---|---|---|---|---|---|---|---|---|---|
| azureml-yonatan | 0 | Feb 02 18:31:32 | completed | azureml-utils-train | v3io_user=yonatan kind=job owner=yonatan host=azureml-utils-train-7pp86 |
dataset |
experiment_name=azure-automl-test cpu_cluster_name=azureml-cpu dataset_name=iris dataset_description=iris training data label_column_name=label create_new_version=True register_model_name=iris-model save_n_models=3 automl_settings={'task': 'classification', 'debug_log': 'automl_errors.log', 'enable_early_stopping': False, 'allowed_models': ['LogisticRegression', 'SGD', 'SVM'], 'iterations': 5, 'iteration_timeout_minutes': 2, 'max_concurrent_iterations': 2, 'max_cores_per_iteration': -1, 'n_cross_validations': 5, 'primary_metric': 'accuracy', 'featurization': 'off', 'model_explainability': False, 'enable_voting_ensemble': False, 'enable_stack_ensemble': False} |
dataset_blob_path=az://azureml-blobstore-27f8977b-4946-4ca0-bdc5-5a685d2fe8d7/iris.csv best_iteration=1 auc_macro=0.9973298059964726 auc_micro=0.9979999999999999 norm_macro_recall=0.9594444444444443 balanced_accuracy=0.9729629629629629 f1_score_macro=0.9721779225097302 weighted_accuracy=0.9739694654594448 average_precision_score_weighted=0.9953861693861693 f1_score_weighted=0.9730901151988108 precision_score_micro=0.9733333333333334 matthews_correlation=0.9613232982405628 recall_score_macro=0.9729629629629629 precision_score_weighted=0.9767380952380952 recall_score_micro=0.9733333333333334 precision_score_macro=0.9754761904761905 average_precision_score_macro=0.9954059829059829 accuracy=0.9733333333333334 auc_weighted=0.9972857142857142 recall_score_weighted=0.9733333333333334 f1_score_micro=0.9733333333333334 average_precision_score_micro=0.9962096994520057 log_loss=0.07548089806904337 |
model iteration_results parallel_coordinates |
> to track results use the .show() or .logs() methods or click here to open in UI
> 2022-02-02 18:38:48,608 [info] run executed, status=completed
View the run result: (more details in the UI)
azureml_run.artifact('iteration_results').show()
| state | iter | param.data_trans_class_name | param.data_trans_module | param.data_trans_spec_class | param.train_class_name | param.train_module | param.train_param_kwargs_C | param.train_param_kwargs_class_weight | param.train_spec_class | ... | output.precision_score_weighted | output.recall_score_micro | output.precision_score_macro | output.average_precision_score_macro | output.accuracy | output.auc_weighted | output.recall_score_weighted | output.f1_score_micro | output.average_precision_score_micro | output.log_loss | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | completed | 1 | StandardScaler | sklearn.preprocessing | preproc | LogisticRegression | sklearn.linear_model | 16.768329 | NaN | sklearn | ... | 0.976738 | 0.973333 | 0.975476 | 0.995406 | 0.973333 | 0.997286 | 0.973333 | 0.973333 | 0.996210 | 0.075481 |
| 1 | completed | 2 | MaxAbsScaler | sklearn.preprocessing | preproc | LogisticRegression | sklearn.linear_model | 719.685673 | NaN | sklearn | ... | 0.976738 | 0.973333 | 0.975476 | 0.995227 | 0.973333 | 0.997286 | 0.973333 | 0.973333 | 0.996211 | 0.072493 |
| 2 | completed | 3 | RobustScaler | sklearn.preprocessing | preproc | LogisticRegression | sklearn.linear_model | 1048.113134 | balanced | sklearn | ... | 0.968359 | 0.966667 | 0.966410 | 0.994217 | 0.966667 | 0.996598 | 0.966667 | 0.966667 | 0.995595 | 0.086160 |
3 rows × 31 columns
4. Deploy Model Serving#
# Importing serving function from marketplace:
serving_fn = mlrun.new_function("serving", kind="serving", image="yhaviv/mlrun:dev")
serving_fn.with_code(body=" ")
serving_fn.with_requirements("./requirements.txt")
# Set the real-time pipeline topology
serving_fn.set_topology(
'router',
'mlrun.serving.routers.VotingEnsemble'
)
# Add the trained models:
artifacts = mlrun.get_run_db().list_artifacts(project=project.name)
models = {f"{model['algorithm']}{i}" :f"{model['db_key']}#{model['iter']}"
for i, model in enumerate(artifacts) if model["kind"]=="model"}
for name, path in models.items():
serving_fn.add_model(
name,
class_name="mlrun.frameworks.sklearn.PickleModelServer",
model_path=project.get_artifact_uri(path))
serving_fn.spec.graph.plot()
Building and Deploying the Serving Function
function_address = serving_fn.deploy()
> 2022-02-02 18:38:48,785 [info] Starting remote function deploy
5. Using the Live Model-Serving Function#
print (f'The address for the function is {function_address} \n')
!curl $function_address
# Data for testing:
source_df = mlrun.get_dataitem(DATA_URL).as_df()
test_vector = source_df.sample(5).drop('label', axis=1).values.tolist()
test_vector
After deploying the serving function with the required model we can make prediction:
serving_fn.invoke(f'/v2/models/infer', {"inputs": test_vector})
6. Clean up#
For cleaning up AzureML resources see: https://docs.microsoft.com/en-us/azure/machine-learning/tutorial-auto-train-models#clean-up-resources