03_H2O_AutoML_with_MLflow.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# End-to-End AutoML for Insurance Cross-Sell\n",
    "\n",
    "## Part 3 - H2O AutoML with MLflow\n",
    "\n",
    "### Contents\n",
    "[Part 1 - Initial Setup](#setup)  \n",
    "[Part 2 - H2O AutoML Training with MLflow Tracking](#automl)  \n",
    "[Part 3 - Predict with H2O AutoML Best Model](#predict)  \n",
    "[Part 4 - H2O Model Explainability](#explain)  \n",
    "[Part 5 - References](#references)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___\n",
    "<a name=\"setup\"></a>\n",
    "## (1) Initial Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Install pre-requisite dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# !pip install requests\n",
    "# !pip install tabulate\n",
    "# !pip install future"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Install H2O in Python"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# !pip install -f http://h2o-release.s3.amazonaws.com/h2o/latest_stable_Py.html h2o"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Install MLflow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# !pip install mlflow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import dependencies and datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import libraries\n",
    "import h2o\n",
    "from h2o.automl import H2OAutoML, get_leaderboard\n",
    "\n",
    "import mlflow\n",
    "import mlflow.h2o\n",
    "from mlflow.tracking import MlflowClient\n",
    "from mlflow.entities import ViewType\n",
    "\n",
    "import pandas as pd\n",
    "import json\n",
    "\n",
    "from sklearn.metrics import f1_score, accuracy_score\n",
    "\n",
    "pd.set_option('display.max_columns', None)\n",
    "pd.set_option('display.max_rows', None)\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Initiate H2O cluster"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Checking whether there is an H2O instance running at http://localhost:54321 ..... not found.\n",
      "Attempting to start a local H2O server...\n",
      "  Java Version: openjdk version \"11.0.13\" 2021-10-19; OpenJDK Runtime Environment (build 11.0.13+8); OpenJDK 64-Bit Server VM (build 11.0.13+8, mixed mode)\n",
      "  Starting server from /home/amin/.local/share/virtualenvs/e2e-insurance-cross-sell-prediction-automl-eEZ4gcUa/lib/python3.9/site-packages/h2o/backend/bin/h2o.jar\n",
      "  Ice root: /tmp/tmp1mc4xd5k\n",
      "  JVM stdout: /tmp/tmp1mc4xd5k/h2o_amin_started_from_python.out\n",
      "  JVM stderr: /tmp/tmp1mc4xd5k/h2o_amin_started_from_python.err\n",
      "  Server is running at http://127.0.0.1:54321\n",
      "Connecting to H2O server at http://127.0.0.1:54321 ... successful.\n",
      "Warning: Your H2O cluster version is too old (3 months and 14 days)!Please download and install the latest version from http://h2o.ai/download/\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div style=\"overflow:auto\"><table style=\"width:50%\"><tr><td>H2O_cluster_uptime:</td>\n",
       "<td>01 secs</td></tr>\n",
       "<tr><td>H2O_cluster_timezone:</td>\n",
       "<td>Africa/Tunis</td></tr>\n",
       "<tr><td>H2O_data_parsing_timezone:</td>\n",
       "<td>UTC</td></tr>\n",
       "<tr><td>H2O_cluster_version:</td>\n",
       "<td>3.34.0.3</td></tr>\n",
       "<tr><td>H2O_cluster_version_age:</td>\n",
       "<td>3 months and 14 days !!!</td></tr>\n",
       "<tr><td>H2O_cluster_name:</td>\n",
       "<td>H2O_from_python_amin_tvm89v</td></tr>\n",
       "<tr><td>H2O_cluster_total_nodes:</td>\n",
       "<td>1</td></tr>\n",
       "<tr><td>H2O_cluster_free_memory:</td>\n",
       "<td>1.910 Gb</td></tr>\n",
       "<tr><td>H2O_cluster_total_cores:</td>\n",
       "<td>8</td></tr>\n",
       "<tr><td>H2O_cluster_allowed_cores:</td>\n",
       "<td>8</td></tr>\n",
       "<tr><td>H2O_cluster_status:</td>\n",
       "<td>locked, healthy</td></tr>\n",
       "<tr><td>H2O_connection_url:</td>\n",
       "<td>http://127.0.0.1:54321</td></tr>\n",
       "<tr><td>H2O_connection_proxy:</td>\n",
       "<td>{\"http\": null, \"https\": null}</td></tr>\n",
       "<tr><td>H2O_internal_security:</td>\n",
       "<td>False</td></tr>\n",
       "<tr><td>H2O_API_Extensions:</td>\n",
       "<td>Amazon S3, XGBoost, Algos, AutoML, Core V3, TargetEncoder, Core V4</td></tr>\n",
       "<tr><td>Python_version:</td>\n",
       "<td>3.9.10 final</td></tr></table></div>"
      ],
      "text/plain": [
       "--------------------------  ------------------------------------------------------------------\n",
       "H2O_cluster_uptime:         01 secs\n",
       "H2O_cluster_timezone:       Africa/Tunis\n",
       "H2O_data_parsing_timezone:  UTC\n",
       "H2O_cluster_version:        3.34.0.3\n",
       "H2O_cluster_version_age:    3 months and 14 days !!!\n",
       "H2O_cluster_name:           H2O_from_python_amin_tvm89v\n",
       "H2O_cluster_total_nodes:    1\n",
       "H2O_cluster_free_memory:    1.910 Gb\n",
       "H2O_cluster_total_cores:    8\n",
       "H2O_cluster_allowed_cores:  8\n",
       "H2O_cluster_status:         locked, healthy\n",
       "H2O_connection_url:         http://127.0.0.1:54321\n",
       "H2O_connection_proxy:       {\"http\": null, \"https\": null}\n",
       "H2O_internal_security:      False\n",
       "H2O_API_Extensions:         Amazon S3, XGBoost, Algos, AutoML, Core V3, TargetEncoder, Core V4\n",
       "Python_version:             3.9.10 final\n",
       "--------------------------  ------------------------------------------------------------------"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Start the H2O cluster (locally)\n",
    "h2o.init()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setup MLflow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- First open Powershell terminal and change path to the directory hosting this notebook\n",
    "- Enter `mlflow ui` to initiate MLFlow server\n",
    "- Once done, access the MLFlow UI served on http://127.0.0.1:5000"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Name: automl-insurance\n",
      "Experiment_id: 1\n",
      "Artifact Location: file:///home/amin/Documents/e2e-insurance-cross-sell-prediction-automl/mlruns/1\n",
      "Tags: {}\n",
      "Lifecycle_stage: active\n",
      "Tracking uri: file:///home/amin/Documents/e2e-insurance-cross-sell-prediction-automl/mlruns\n"
     ]
    }
   ],
   "source": [
    "# Initialize MLFlow client\n",
    "client = MlflowClient()\n",
    "\n",
    "# Set up MlFlow experiment\n",
    "experiment_name = 'automl-insurance'\n",
    "\n",
    "try:\n",
    "    experiment_id = mlflow.create_experiment(experiment_name)\n",
    "    experiment = client.get_experiment_by_name(experiment_name)\n",
    "except:\n",
    "    experiment = client.get_experiment_by_name(experiment_name)\n",
    "    \n",
    "mlflow.set_experiment(experiment_name)\n",
    "\n",
    "# Print experiment details\n",
    "print(f\"Name: {experiment_name}\")\n",
    "print(f\"Experiment_id: {experiment.experiment_id}\")\n",
    "print(f\"Artifact Location: {experiment.artifact_location}\")\n",
    "print(f\"Tags: {experiment.tags}\")\n",
    "print(f\"Lifecycle_stage: {experiment.lifecycle_stage}\")\n",
    "print(f\"Tracking uri: {mlflow.get_tracking_uri()}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___\n",
    "<a name=\"automl\"></a>\n",
    "## (2) H2O AutoML Training with MLFlow Tracking"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import training data\n",
    "- Not splitting further into train/val set because 5-fold cross-val is applied by default in the AutoML training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Parse progress: |████████████████████████████████████████████████████████████████| (done) 100%\n"
     ]
    }
   ],
   "source": [
    "# Import data directly as H2O frame\n",
    "main_frame = h2o.import_file(path='data/processed/train.csv')\n",
    "\n",
    "# Save data types of columns in H2O frame (for matching with test set during prediction)\n",
    "with open('data/processed/train_col_types.json', 'w') as fp:\n",
    "    json.dump(main_frame.types, fp)\n",
    "\n",
    "# Alternatively, can first import as pandas csv, then convert to H2O frame\n",
    "# main_df = pd.read_csv('data/processed/train.csv')\n",
    "# main_frame = h2o.H2OFrame(main_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead>\n",
       "<tr><th style=\"text-align: right;\">  Gender</th><th style=\"text-align: right;\">      Age</th><th style=\"text-align: right;\">  Driving_License</th><th style=\"text-align: right;\">  Previously_Insured</th><th style=\"text-align: right;\">  Annual_Premium</th><th style=\"text-align: right;\">  Vintage</th><th style=\"text-align: right;\">  Response</th><th style=\"text-align: right;\">  Region_Code_1</th><th style=\"text-align: right;\">  Region_Code_10</th><th style=\"text-align: right;\">  Region_Code_11</th><th style=\"text-align: right;\">  Region_Code_12</th><th style=\"text-align: right;\">  Region_Code_13</th><th style=\"text-align: right;\">  Region_Code_14</th><th style=\"text-align: right;\">  Region_Code_15</th><th style=\"text-align: right;\">  Region_Code_16</th><th style=\"text-align: right;\">  Region_Code_17</th><th style=\"text-align: right;\">  Region_Code_18</th><th style=\"text-align: right;\">  Region_Code_19</th><th style=\"text-align: right;\">  Region_Code_2</th><th style=\"text-align: right;\">  Region_Code_20</th><th style=\"text-align: right;\">  Region_Code_21</th><th style=\"text-align: right;\">  Region_Code_22</th><th style=\"text-align: right;\">  Region_Code_23</th><th style=\"text-align: right;\">  Region_Code_24</th><th style=\"text-align: right;\">  Region_Code_25</th><th style=\"text-align: right;\">  Region_Code_26</th><th style=\"text-align: right;\">  Region_Code_27</th><th style=\"text-align: right;\">  Region_Code_28</th><th style=\"text-align: right;\">  Region_Code_29</th><th style=\"text-align: right;\">  Region_Code_3</th><th style=\"text-align: right;\">  Region_Code_30</th><th style=\"text-align: right;\">  Region_Code_31</th><th style=\"text-align: right;\">  Region_Code_32</th><th style=\"text-align: right;\">  Region_Code_33</th><th style=\"text-align: right;\">  Region_Code_34</th><th style=\"text-align: right;\">  Region_Code_35</th><th style=\"text-align: right;\">  Region_Code_36</th><th style=\"text-align: right;\">  Region_Code_37</th><th style=\"text-align: right;\">  Region_Code_38</th><th style=\"text-align: right;\">  Region_Code_39</th><th style=\"text-align: right;\">  Region_Code_4</th><th style=\"text-align: right;\">  Region_Code_40</th><th style=\"text-align: right;\">  Region_Code_41</th><th style=\"text-align: right;\">  Region_Code_42</th><th style=\"text-align: right;\">  Region_Code_43</th><th style=\"text-align: right;\">  Region_Code_44</th><th style=\"text-align: right;\">  Region_Code_45</th><th style=\"text-align: right;\">  Region_Code_46</th><th style=\"text-align: right;\">  Region_Code_47</th><th style=\"text-align: right;\">  Region_Code_48</th><th style=\"text-align: right;\">  Region_Code_49</th><th style=\"text-align: right;\">  Region_Code_5</th><th style=\"text-align: right;\">  Region_Code_50</th><th style=\"text-align: right;\">  Region_Code_51</th><th style=\"text-align: right;\">  Region_Code_52</th><th style=\"text-align: right;\">  Region_Code_6</th><th style=\"text-align: right;\">  Region_Code_7</th><th style=\"text-align: right;\">  Region_Code_8</th><th style=\"text-align: right;\">  Region_Code_9</th><th style=\"text-align: right;\">  Vehicle_Age_lt_1Y</th><th style=\"text-align: right;\">  Vehicle_Age_gt_2Y</th><th style=\"text-align: right;\">  Vehicle_Damage_Yes</th><th style=\"text-align: right;\">  Policy_Sales_Channel_10</th><th style=\"text-align: right;\">  Policy_Sales_Channel_100</th><th style=\"text-align: right;\">  Policy_Sales_Channel_101</th><th style=\"text-align: right;\">  Policy_Sales_Channel_102</th><th style=\"text-align: right;\">  Policy_Sales_Channel_103</th><th style=\"text-align: right;\">  Policy_Sales_Channel_104</th><th style=\"text-align: right;\">  Policy_Sales_Channel_105</th><th style=\"text-align: right;\">  Policy_Sales_Channel_106</th><th style=\"text-align: right;\">  Policy_Sales_Channel_107</th><th style=\"text-align: right;\">  Policy_Sales_Channel_108</th><th style=\"text-align: right;\">  Policy_Sales_Channel_109</th><th style=\"text-align: right;\">  Policy_Sales_Channel_11</th><th style=\"text-align: right;\">  Policy_Sales_Channel_110</th><th style=\"text-align: right;\">  Policy_Sales_Channel_111</th><th style=\"text-align: right;\">  Policy_Sales_Channel_113</th><th style=\"text-align: right;\">  Policy_Sales_Channel_114</th><th style=\"text-align: right;\">  Policy_Sales_Channel_115</th><th style=\"text-align: right;\">  Policy_Sales_Channel_116</th><th style=\"text-align: right;\">  Policy_Sales_Channel_117</th><th style=\"text-align: right;\">  Policy_Sales_Channel_118</th><th style=\"text-align: right;\">  Policy_Sales_Channel_119</th><th style=\"text-align: right;\">  Policy_Sales_Channel_12</th><th style=\"text-align: right;\">  Policy_Sales_Channel_120</th><th style=\"text-align: right;\">  Policy_Sales_Channel_121</th><th style=\"text-align: right;\">  Policy_Sales_Channel_122</th><th style=\"text-align: right;\">  Policy_Sales_Channel_124</th><th style=\"text-align: right;\">  Policy_Sales_Channel_125</th><th style=\"text-align: right;\">  Policy_Sales_Channel_127</th><th style=\"text-align: right;\">  Policy_Sales_Channel_128</th><th style=\"text-align: right;\">  Policy_Sales_Channel_129</th><th style=\"text-align: right;\">  Policy_Sales_Channel_13</th><th style=\"text-align: right;\">  Policy_Sales_Channel_130</th><th style=\"text-align: right;\">  Policy_Sales_Channel_131</th><th style=\"text-align: right;\">  Policy_Sales_Channel_132</th><th style=\"text-align: right;\">  Policy_Sales_Channel_133</th><th style=\"text-align: right;\">  Policy_Sales_Channel_134</th><th style=\"text-align: right;\">  Policy_Sales_Channel_135</th><th style=\"text-align: right;\">  Policy_Sales_Channel_136</th><th style=\"text-align: right;\">  Policy_Sales_Channel_137</th><th style=\"text-align: right;\">  Policy_Sales_Channel_138</th><th style=\"text-align: right;\">  Policy_Sales_Channel_139</th><th style=\"text-align: right;\">  Policy_Sales_Channel_14</th><th style=\"text-align: right;\">  Policy_Sales_Channel_140</th><th style=\"text-align: right;\">  Policy_Sales_Channel_143</th><th style=\"text-align: right;\">  Policy_Sales_Channel_145</th><th style=\"text-align: right;\">  Policy_Sales_Channel_146</th><th style=\"text-align: right;\">  Policy_Sales_Channel_147</th><th style=\"text-align: right;\">  Policy_Sales_Channel_148</th><th style=\"text-align: right;\">  Policy_Sales_Channel_149</th><th style=\"text-align: right;\">  Policy_Sales_Channel_15</th><th style=\"text-align: right;\">  Policy_Sales_Channel_150</th><th style=\"text-align: right;\">  Policy_Sales_Channel_151</th><th style=\"text-align: right;\">  Policy_Sales_Channel_152</th><th style=\"text-align: right;\">  Policy_Sales_Channel_153</th><th style=\"text-align: right;\">  Policy_Sales_Channel_154</th><th style=\"text-align: right;\">  Policy_Sales_Channel_155</th><th style=\"text-align: right;\">  Policy_Sales_Channel_156</th><th style=\"text-align: right;\">  Policy_Sales_Channel_157</th><th style=\"text-align: right;\">  Policy_Sales_Channel_158</th><th style=\"text-align: right;\">  Policy_Sales_Channel_159</th><th style=\"text-align: right;\">  Policy_Sales_Channel_16</th><th style=\"text-align: right;\">  Policy_Sales_Channel_160</th><th style=\"text-align: right;\">  Policy_Sales_Channel_163</th><th style=\"text-align: right;\">  Policy_Sales_Channel_17</th><th style=\"text-align: right;\">  Policy_Sales_Channel_18</th><th style=\"text-align: right;\">  Policy_Sales_Channel_19</th><th style=\"text-align: right;\">  Policy_Sales_Channel_2</th><th style=\"text-align: right;\">  Policy_Sales_Channel_20</th><th style=\"text-align: right;\">  Policy_Sales_Channel_21</th><th style=\"text-align: right;\">  Policy_Sales_Channel_22</th><th style=\"text-align: right;\">  Policy_Sales_Channel_23</th><th style=\"text-align: right;\">  Policy_Sales_Channel_24</th><th style=\"text-align: right;\">  Policy_Sales_Channel_25</th><th style=\"text-align: right;\">  Policy_Sales_Channel_26</th><th style=\"text-align: right;\">  Policy_Sales_Channel_27</th><th style=\"text-align: right;\">  Policy_Sales_Channel_28</th><th style=\"text-align: right;\">  Policy_Sales_Channel_29</th><th style=\"text-align: right;\">  Policy_Sales_Channel_3</th><th style=\"text-align: right;\">  Policy_Sales_Channel_30</th><th style=\"text-align: right;\">  Policy_Sales_Channel_31</th><th style=\"text-align: right;\">  Policy_Sales_Channel_32</th><th style=\"text-align: right;\">  Policy_Sales_Channel_34</th><th style=\"text-align: right;\">  Policy_Sales_Channel_35</th><th style=\"text-align: right;\">  Policy_Sales_Channel_36</th><th style=\"text-align: right;\">  Policy_Sales_Channel_37</th><th style=\"text-align: right;\">  Policy_Sales_Channel_38</th><th style=\"text-align: right;\">  Policy_Sales_Channel_39</th><th style=\"text-align: right;\">  Policy_Sales_Channel_4</th><th style=\"text-align: right;\">  Policy_Sales_Channel_40</th><th style=\"text-align: right;\">  Policy_Sales_Channel_41</th><th style=\"text-align: right;\">  Policy_Sales_Channel_42</th><th style=\"text-align: right;\">  Policy_Sales_Channel_44</th><th style=\"text-align: right;\">  Policy_Sales_Channel_45</th><th style=\"text-align: right;\">  Policy_Sales_Channel_46</th><th style=\"text-align: right;\">  Policy_Sales_Channel_47</th><th style=\"text-align: right;\">  Policy_Sales_Channel_48</th><th style=\"text-align: right;\">  Policy_Sales_Channel_49</th><th style=\"text-align: right;\">  Policy_Sales_Channel_51</th><th style=\"text-align: right;\">  Policy_Sales_Channel_52</th><th style=\"text-align: right;\">  Policy_Sales_Channel_53</th><th style=\"text-align: right;\">  Policy_Sales_Channel_54</th><th style=\"text-align: right;\">  Policy_Sales_Channel_55</th><th style=\"text-align: right;\">  Policy_Sales_Channel_56</th><th style=\"text-align: right;\">  Policy_Sales_Channel_57</th><th style=\"text-align: right;\">  Policy_Sales_Channel_58</th><th style=\"text-align: right;\">  Policy_Sales_Channel_59</th><th style=\"text-align: right;\">  Policy_Sales_Channel_6</th><th style=\"text-align: right;\">  Policy_Sales_Channel_60</th><th style=\"text-align: right;\">  Policy_Sales_Channel_61</th><th style=\"text-align: right;\">  Policy_Sales_Channel_62</th><th style=\"text-align: right;\">  Policy_Sales_Channel_63</th><th style=\"text-align: right;\">  Policy_Sales_Channel_64</th><th style=\"text-align: right;\">  Policy_Sales_Channel_65</th><th style=\"text-align: right;\">  Policy_Sales_Channel_66</th><th style=\"text-align: right;\">  Policy_Sales_Channel_67</th><th style=\"text-align: right;\">  Policy_Sales_Channel_68</th><th style=\"text-align: right;\">  Policy_Sales_Channel_69</th><th style=\"text-align: right;\">  Policy_Sales_Channel_7</th><th style=\"text-align: right;\">  Policy_Sales_Channel_70</th><th style=\"text-align: right;\">  Policy_Sales_Channel_71</th><th style=\"text-align: right;\">  Policy_Sales_Channel_73</th><th style=\"text-align: right;\">  Policy_Sales_Channel_74</th><th style=\"text-align: right;\">  Policy_Sales_Channel_75</th><th style=\"text-align: right;\">  Policy_Sales_Channel_76</th><th style=\"text-align: right;\">  Policy_Sales_Channel_78</th><th style=\"text-align: right;\">  Policy_Sales_Channel_79</th><th style=\"text-align: right;\">  Policy_Sales_Channel_8</th><th style=\"text-align: right;\">  Policy_Sales_Channel_80</th><th style=\"text-align: right;\">  Policy_Sales_Channel_81</th><th style=\"text-align: right;\">  Policy_Sales_Channel_82</th><th style=\"text-align: right;\">  Policy_Sales_Channel_84</th><th style=\"text-align: right;\">  Policy_Sales_Channel_86</th><th style=\"text-align: right;\">  Policy_Sales_Channel_87</th><th style=\"text-align: right;\">  Policy_Sales_Channel_88</th><th style=\"text-align: right;\">  Policy_Sales_Channel_89</th><th style=\"text-align: right;\">  Policy_Sales_Channel_9</th><th style=\"text-align: right;\">  Policy_Sales_Channel_90</th><th style=\"text-align: right;\">  Policy_Sales_Channel_91</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "<tr><td style=\"text-align: right;\">       0</td><td style=\"text-align: right;\">0        </td><td style=\"text-align: right;\">                1</td><td style=\"text-align: right;\">                   0</td><td style=\"text-align: right;\">       0.0439283</td><td style=\"text-align: right;\">0.958478 </td><td style=\"text-align: right;\">         0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">              1</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">                  1</td><td style=\"text-align: right;\">                  0</td><td style=\"text-align: right;\">                   1</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                         1</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                       0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                       0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                       0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                       0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                       0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                       0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                       0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td></tr>\n",
       "<tr><td style=\"text-align: right;\">       0</td><td style=\"text-align: right;\">0.415385 </td><td style=\"text-align: right;\">                1</td><td style=\"text-align: right;\">                   1</td><td style=\"text-align: right;\">       0.0681388</td><td style=\"text-align: right;\">0.820069 </td><td style=\"text-align: right;\">         0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               1</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">                  0</td><td style=\"text-align: right;\">                  0</td><td style=\"text-align: right;\">                   0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                       0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        1</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                       0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                       0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                       0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                       0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                       0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                       0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td></tr>\n",
       "<tr><td style=\"text-align: right;\">       0</td><td style=\"text-align: right;\">0.415385 </td><td style=\"text-align: right;\">                1</td><td style=\"text-align: right;\">                   0</td><td style=\"text-align: right;\">       0.0625448</td><td style=\"text-align: right;\">0.854671 </td><td style=\"text-align: right;\">         0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               1</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">                  0</td><td style=\"text-align: right;\">                  0</td><td style=\"text-align: right;\">                   1</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                       0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        1</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                       0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                       0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                       0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                       0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                       0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                       0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td></tr>\n",
       "<tr><td style=\"text-align: right;\">       0</td><td style=\"text-align: right;\">0.0307692</td><td style=\"text-align: right;\">                1</td><td style=\"text-align: right;\">                   0</td><td style=\"text-align: right;\">       0        </td><td style=\"text-align: right;\">0.979239 </td><td style=\"text-align: right;\">         0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               1</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">                  1</td><td style=\"text-align: right;\">                  0</td><td style=\"text-align: right;\">                   0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         1</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                       0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                       0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                       0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                       0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                       0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                       0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                       0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td></tr>\n",
       "<tr><td style=\"text-align: right;\">       0</td><td style=\"text-align: right;\">0.430769 </td><td style=\"text-align: right;\">                1</td><td style=\"text-align: right;\">                   0</td><td style=\"text-align: right;\">       0.102691 </td><td style=\"text-align: right;\">0.532872 </td><td style=\"text-align: right;\">         1</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">              1</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">                  0</td><td style=\"text-align: right;\">                  0</td><td style=\"text-align: right;\">                   1</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         1</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                       0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                       0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                       0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                       0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                       0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                       0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                       0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td></tr>\n",
       "<tr><td style=\"text-align: right;\">       0</td><td style=\"text-align: right;\">0.323077 </td><td style=\"text-align: right;\">                1</td><td style=\"text-align: right;\">                   0</td><td style=\"text-align: right;\">       0        </td><td style=\"text-align: right;\">0.0726644</td><td style=\"text-align: right;\">         0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">               1</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">                  0</td><td style=\"text-align: right;\">                  0</td><td style=\"text-align: right;\">                   1</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         1</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                       0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                       0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                       0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                       0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                       0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                       0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                       0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td></tr>\n",
       "<tr><td style=\"text-align: right;\">       0</td><td style=\"text-align: right;\">0.507692 </td><td style=\"text-align: right;\">                1</td><td style=\"text-align: right;\">                   0</td><td style=\"text-align: right;\">       0.0769141</td><td style=\"text-align: right;\">0.0138408</td><td style=\"text-align: right;\">         0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               1</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">                  0</td><td style=\"text-align: right;\">                  0</td><td style=\"text-align: right;\">                   1</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                       0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        1</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                       0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                       0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                       0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                       0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                       0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                       0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td></tr>\n",
       "<tr><td style=\"text-align: right;\">       1</td><td style=\"text-align: right;\">0.0461538</td><td style=\"text-align: right;\">                1</td><td style=\"text-align: right;\">                   1</td><td style=\"text-align: right;\">       0.0594733</td><td style=\"text-align: right;\">0.882353 </td><td style=\"text-align: right;\">         0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               1</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">                  1</td><td style=\"text-align: right;\">                  0</td><td style=\"text-align: right;\">                   0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         1</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                       0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                       0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                       0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                       0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                       0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                       0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                       0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td></tr>\n",
       "<tr><td style=\"text-align: right;\">       0</td><td style=\"text-align: right;\">0.476923 </td><td style=\"text-align: right;\">                1</td><td style=\"text-align: right;\">                   0</td><td style=\"text-align: right;\">       0.0397202</td><td style=\"text-align: right;\">0.207612 </td><td style=\"text-align: right;\">         0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">               1</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">                  0</td><td style=\"text-align: right;\">                  0</td><td style=\"text-align: right;\">                   1</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                       0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        1</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                       0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                       0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                       0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                       0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                       0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                       0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td></tr>\n",
       "<tr><td style=\"text-align: right;\">       1</td><td style=\"text-align: right;\">0.276923 </td><td style=\"text-align: right;\">                1</td><td style=\"text-align: right;\">                   1</td><td style=\"text-align: right;\">       0.10188  </td><td style=\"text-align: right;\">0.574394 </td><td style=\"text-align: right;\">         0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               1</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">               0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">              0</td><td style=\"text-align: right;\">                  0</td><td style=\"text-align: right;\">                  0</td><td style=\"text-align: right;\">                   0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         1</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                         0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                       0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                       0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                       0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                       0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                       0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                       0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                       0</td><td style=\"text-align: right;\">                        0</td><td style=\"text-align: right;\">                        0</td></tr>\n",
       "</tbody>\n",
       "</table>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": []
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Set predictor and target columns\n",
    "target = 'Response'\n",
    "predictors = [n for n in main_frame.col_names if n != target]\n",
    "\n",
    "# Factorize target variable so that autoML tackles classification problem (instead of regression)\n",
    "main_frame[target] = main_frame[target].asfactor()\n",
    "\n",
    "# Visualize H2O frame structure\n",
    "main_frame.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Start H2O AutoML training with MLflow tracking"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "AutoML progress: |\n",
      "16:35:21.159: Project: AutoML_1_20220121_163521\n",
      "16:35:21.165: Setting stopping tolerance adaptively based on the training frame: 0.0033065124063248246\n",
      "16:35:21.166: Build control seed: 42\n",
      "16:35:21.166: training frame: Frame key: AutoML_1_20220121_163521_training_py_1_sid_8f18    cols: 208    rows: 91466  chunks: 32    size: 2498353  checksum: -277864416819399133\n",
      "16:35:21.166: validation frame: NULL\n",
      "16:35:21.167: leaderboard frame: NULL\n",
      "16:35:21.167: blending frame: NULL\n",
      "16:35:21.167: response column: Response\n",
      "16:35:21.167: fold column: null\n",
      "16:35:21.167: weights column: null\n",
      "16:35:21.175: Loading execution steps: [{XGBoost : [def_2 (1g, 10w), def_1 (2g, 10w), def_3 (3g, 10w), grid_1 (4g, 90w), lr_search (6g, 30w)]}, {GLM : [def_1 (1g, 10w)]}, {DRF : [def_1 (2g, 10w), XRT (3g, 10w)]}, {GBM : [def_5 (1g, 10w), def_2 (2g, 10w), def_3 (2g, 10w), def_4 (2g, 10w), def_1 (3g, 10w), grid_1 (4g, 60w), lr_annealing (6g, 10w)]}, {DeepLearning : [def_1 (3g, 10w), grid_1 (4g, 30w), grid_2 (5g, 30w), grid_3 (5g, 30w)]}, {completion : [resume_best_grids (10g, 60w)]}, {StackedEnsemble : [best_of_family_1 (1g, 5w), best_of_family_2 (2g, 5w), best_of_family_3 (3g, 5w), best_of_family_4 (4g, 5w), best_of_family_5 (5g, 5w), all_2 (2g, 10w), all_3 (3g, 10w), all_4 (4g, 10w), all_5 (5g, 10w), monotonic (6g, 10w), best_of_family_xgboost (6g, 10w), best_of_family_gbm (6g, 10w), all_xgboost (7g, 10w), all_gbm (7g, 10w), best_of_family_xglm (8g, 10w), all_xglm (8g, 10w), best_of_family (10g, 10w), best_N (10g, 10w)]}]\n",
      "16:35:21.189: Disabling Algo: GLM as requested by the user.\n",
      "16:35:21.189: Disabling Algo: DRF as requested by the user.\n",
      "16:35:21.190: AutoML job created: 2022.01.21 16:35:21.137\n",
      "16:35:21.191: AutoML build started: 2022.01.21 16:35:21.190\n",
      "16:35:21.198: AutoML: starting XGBoost_1_AutoML_1_20220121_163521 model training\n",
      "\n",
      "███\n",
      "16:35:48.303: New leader: XGBoost_1_AutoML_1_20220121_163521, logloss: 0.2791987442093622\n",
      "16:35:48.309: AutoML: starting GBM_1_AutoML_1_20220121_163521 model training\n",
      "\n",
      "████████\n",
      "16:37:22.336: New leader: GBM_1_AutoML_1_20220121_163521, logloss: 0.2701999690529526\n",
      "16:37:22.347: AutoML: starting StackedEnsemble_BestOfFamily_1_AutoML_1_20220121_163521 model training\n",
      "\n",
      "█\n",
      "16:37:25.354: New leader: StackedEnsemble_BestOfFamily_1_AutoML_1_20220121_163521, logloss: 0.2696886785240582\n",
      "16:37:25.354: AutoML: starting XGBoost_2_AutoML_1_20220121_163521 model training\n",
      "\n",
      "█\n",
      "16:37:45.364: AutoML: starting GBM_2_AutoML_1_20220121_163521 model training\n",
      "\n",
      "███\n",
      "16:38:41.383: New leader: GBM_2_AutoML_1_20220121_163521, logloss: 0.26861513904645684\n",
      "16:38:41.383: AutoML: starting GBM_3_AutoML_1_20220121_163521 model training\n",
      "\n",
      "██\n",
      "16:39:40.406: AutoML: starting GBM_4_AutoML_1_20220121_163521 model training\n",
      "\n",
      "████\n",
      "16:40:46.426: AutoML: starting StackedEnsemble_BestOfFamily_2_AutoML_1_20220121_163521 model training\n",
      "16:40:48.438: New leader: StackedEnsemble_BestOfFamily_2_AutoML_1_20220121_163521, logloss: 0.2682276526415742\n",
      "16:40:48.440: AutoML: starting StackedEnsemble_AllModels_1_AutoML_1_20220121_163521 model training\n",
      "\n",
      "\n",
      "16:40:51.452: New leader: StackedEnsemble_AllModels_1_AutoML_1_20220121_163521, logloss: 0.2681357023038465\n",
      "16:40:51.453: AutoML: starting XGBoost_3_AutoML_1_20220121_163521 model training\n",
      "\n",
      "█\n",
      "16:41:06.460: AutoML: starting GBM_5_AutoML_1_20220121_163521 model training\n",
      "\n",
      "██\n",
      "16:41:56.488: AutoML: starting DeepLearning_1_AutoML_1_20220121_163521 model training\n",
      "\n",
      "███\n",
      "16:42:45.506: AutoML: starting StackedEnsemble_BestOfFamily_3_AutoML_1_20220121_163521 model training\n",
      "16:42:47.516: New leader: StackedEnsemble_BestOfFamily_3_AutoML_1_20220121_163521, logloss: 0.2677798866572095\n",
      "16:42:47.517: AutoML: starting StackedEnsemble_AllModels_2_AutoML_1_20220121_163521 model training\n",
      "\n",
      "█████\n",
      "16:42:50.532: New leader: StackedEnsemble_AllModels_2_AutoML_1_20220121_163521, logloss: 0.26769269449672595\n",
      "16:42:50.535: AutoML: starting XGBoost_grid_1_AutoML_1_20220121_163521 hyperparameter search\n",
      "16:43:41.592: AutoML: starting GBM_grid_1_AutoML_1_20220121_163521 hyperparameter search\n",
      "\n",
      "██████████\n",
      "16:44:26.611: AutoML: starting DeepLearning_grid_1_AutoML_1_20220121_163521 hyperparameter search\n",
      "\n",
      "██████████████\n",
      "16:49:41.688: AutoML: starting StackedEnsemble_AllModels_3_AutoML_1_20220121_163521 model training\n",
      "\n",
      "\n",
      "16:49:44.789: No base models, due to timeouts or the exclude_algos option. Skipping StackedEnsemble 'monotonic'.\n",
      "16:49:44.790: AutoML: starting StackedEnsemble_BestOfFamily_4_AutoML_1_20220121_163521 model training\n",
      "\n",
      "██\n",
      "16:50:08.809: AutoML: starting StackedEnsemble_BestOfFamily_5_AutoML_1_20220121_163521 model training\n",
      "\n",
      "\n",
      "16:50:23.833: AutoML: starting StackedEnsemble_AllModels_4_AutoML_1_20220121_163521 model training\n",
      "\n",
      "\n",
      "16:50:48.850: AutoML: starting StackedEnsemble_AllModels_5_AutoML_1_20220121_163521 model training\n",
      "\n",
      "\n",
      "16:51:06.877: AutoML: starting StackedEnsemble_BestOfFamily_6_AutoML_1_20220121_163521 model training\n",
      "\n",
      "█\n",
      "16:51:25.900: AutoML: starting StackedEnsemble_AllModels_6_AutoML_1_20220121_163521 model training\n",
      "\n",
      "███| (done) 100%\n",
      "\n",
      "16:51:48.927: Actual modeling steps: [{XGBoost : [def_2 (1g, 10w)]}, {GBM : [def_5 (1g, 10w)]}, {StackedEnsemble : [best_of_family_1 (1g, 5w)]}, {XGBoost : [def_1 (2g, 10w)]}, {GBM : [def_2 (2g, 10w), def_3 (2g, 10w), def_4 (2g, 10w)]}, {StackedEnsemble : [best_of_family_2 (2g, 5w), all_2 (2g, 10w)]}, {XGBoost : [def_3 (3g, 10w)]}, {GBM : [def_1 (3g, 10w)]}, {DeepLearning : [def_1 (3g, 10w)]}, {StackedEnsemble : [best_of_family_3 (3g, 5w), all_3 (3g, 10w)]}, {XGBoost : [grid_1 (4g, 90w)]}, {GBM : [grid_1 (4g, 60w)]}, {DeepLearning : [grid_1 (4g, 30w)]}, {StackedEnsemble : [all_4 (4g, 10w), best_of_family_xgboost (6g, 10w), best_of_family_gbm (6g, 10w), all_xgboost (7g, 10w), all_gbm (7g, 10w), best_of_family_xglm (8g, 10w), all_xglm (8g, 10w)]}]\n",
      "16:51:48.928: AutoML build stopped: 2022.01.21 16:51:48.927\n",
      "16:51:48.928: AutoML build done: built 13 models\n",
      "16:51:48.928: AutoML duration: 16 min 27.737 sec\n",
      "\n",
      "file:///home/amin/Documents/e2e-insurance-cross-sell-prediction-automl/mlruns/1/6f2d4b47e37645b88a01aea5ba465e28/artifacts/model\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead>\n",
       "<tr><th>model_id                                               </th><th style=\"text-align: right;\">  logloss</th><th style=\"text-align: right;\">     auc</th><th style=\"text-align: right;\">   aucpr</th><th style=\"text-align: right;\">  mean_per_class_error</th><th style=\"text-align: right;\">    rmse</th><th style=\"text-align: right;\">      mse</th><th style=\"text-align: right;\">  training_time_ms</th><th style=\"text-align: right;\">  predict_time_per_row_ms</th><th>algo           </th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "<tr><td>StackedEnsemble_AllModels_2_AutoML_1_20220121_163521   </td><td style=\"text-align: right;\"> 0.267693</td><td style=\"text-align: right;\">0.857561</td><td style=\"text-align: right;\">0.370057</td><td style=\"text-align: right;\">              0.237628</td><td style=\"text-align: right;\">0.296148</td><td style=\"text-align: right;\">0.0877036</td><td style=\"text-align: right;\">              2279</td><td style=\"text-align: right;\">                 0.020652</td><td>StackedEnsemble</td></tr>\n",
       "<tr><td>StackedEnsemble_AllModels_3_AutoML_1_20220121_163521   </td><td style=\"text-align: right;\"> 0.267708</td><td style=\"text-align: right;\">0.857512</td><td style=\"text-align: right;\">0.370124</td><td style=\"text-align: right;\">              0.23828 </td><td style=\"text-align: right;\">0.296157</td><td style=\"text-align: right;\">0.0877089</td><td style=\"text-align: right;\">              2556</td><td style=\"text-align: right;\">                 0.025513</td><td>StackedEnsemble</td></tr>\n",
       "<tr><td>StackedEnsemble_BestOfFamily_3_AutoML_1_20220121_163521</td><td style=\"text-align: right;\"> 0.26778 </td><td style=\"text-align: right;\">0.857369</td><td style=\"text-align: right;\">0.369595</td><td style=\"text-align: right;\">              0.239951</td><td style=\"text-align: right;\">0.29621 </td><td style=\"text-align: right;\">0.0877403</td><td style=\"text-align: right;\">              1838</td><td style=\"text-align: right;\">                 0.005853</td><td>StackedEnsemble</td></tr>\n",
       "<tr><td>StackedEnsemble_AllModels_6_AutoML_1_20220121_163521   </td><td style=\"text-align: right;\"> 0.267796</td><td style=\"text-align: right;\">0.857505</td><td style=\"text-align: right;\">0.369818</td><td style=\"text-align: right;\">              0.229687</td><td style=\"text-align: right;\">0.296192</td><td style=\"text-align: right;\">0.0877297</td><td style=\"text-align: right;\">             22324</td><td style=\"text-align: right;\">                 0.022587</td><td>StackedEnsemble</td></tr>\n",
       "<tr><td>StackedEnsemble_BestOfFamily_6_AutoML_1_20220121_163521</td><td style=\"text-align: right;\"> 0.26787 </td><td style=\"text-align: right;\">0.857296</td><td style=\"text-align: right;\">0.369078</td><td style=\"text-align: right;\">              0.23661 </td><td style=\"text-align: right;\">0.296256</td><td style=\"text-align: right;\">0.0877674</td><td style=\"text-align: right;\">             18751</td><td style=\"text-align: right;\">                 0.004722</td><td>StackedEnsemble</td></tr>\n",
       "<tr><td>StackedEnsemble_AllModels_1_AutoML_1_20220121_163521   </td><td style=\"text-align: right;\"> 0.268136</td><td style=\"text-align: right;\">0.856936</td><td style=\"text-align: right;\">0.370027</td><td style=\"text-align: right;\">              0.235645</td><td style=\"text-align: right;\">0.296295</td><td style=\"text-align: right;\">0.0877905</td><td style=\"text-align: right;\">              2218</td><td style=\"text-align: right;\">                 0.00939 </td><td>StackedEnsemble</td></tr>\n",
       "<tr><td>StackedEnsemble_BestOfFamily_2_AutoML_1_20220121_163521</td><td style=\"text-align: right;\"> 0.268228</td><td style=\"text-align: right;\">0.856598</td><td style=\"text-align: right;\">0.369316</td><td style=\"text-align: right;\">              0.234249</td><td style=\"text-align: right;\">0.296381</td><td style=\"text-align: right;\">0.0878418</td><td style=\"text-align: right;\">              1773</td><td style=\"text-align: right;\">                 0.003572</td><td>StackedEnsemble</td></tr>\n",
       "<tr><td>GBM_2_AutoML_1_20220121_163521                         </td><td style=\"text-align: right;\"> 0.268615</td><td style=\"text-align: right;\">0.856495</td><td style=\"text-align: right;\">0.369483</td><td style=\"text-align: right;\">              0.226896</td><td style=\"text-align: right;\">0.296556</td><td style=\"text-align: right;\">0.0879456</td><td style=\"text-align: right;\">              8298</td><td style=\"text-align: right;\">                 0.002441</td><td>GBM            </td></tr>\n",
       "<tr><td>XGBoost_3_AutoML_1_20220121_163521                     </td><td style=\"text-align: right;\"> 0.268711</td><td style=\"text-align: right;\">0.855771</td><td style=\"text-align: right;\">0.364893</td><td style=\"text-align: right;\">              0.229421</td><td style=\"text-align: right;\">0.296751</td><td style=\"text-align: right;\">0.0880609</td><td style=\"text-align: right;\">              1514</td><td style=\"text-align: right;\">                 0.001294</td><td>XGBoost        </td></tr>\n",
       "<tr><td>GBM_5_AutoML_1_20220121_163521                         </td><td style=\"text-align: right;\"> 0.268782</td><td style=\"text-align: right;\">0.856372</td><td style=\"text-align: right;\">0.370302</td><td style=\"text-align: right;\">              0.231764</td><td style=\"text-align: right;\">0.296605</td><td style=\"text-align: right;\">0.0879743</td><td style=\"text-align: right;\">              6948</td><td style=\"text-align: right;\">                 0.002437</td><td>GBM            </td></tr>\n",
       "<tr><td>GBM_grid_1_AutoML_1_20220121_163521_model_1            </td><td style=\"text-align: right;\"> 0.268925</td><td style=\"text-align: right;\">0.856051</td><td style=\"text-align: right;\">0.368464</td><td style=\"text-align: right;\">              0.227623</td><td style=\"text-align: right;\">0.296688</td><td style=\"text-align: right;\">0.088024 </td><td style=\"text-align: right;\">              6328</td><td style=\"text-align: right;\">                 0.002371</td><td>GBM            </td></tr>\n",
       "<tr><td>StackedEnsemble_BestOfFamily_5_AutoML_1_20220121_163521</td><td style=\"text-align: right;\"> 0.269056</td><td style=\"text-align: right;\">0.855554</td><td style=\"text-align: right;\">0.365585</td><td style=\"text-align: right;\">              0.233361</td><td style=\"text-align: right;\">0.296672</td><td style=\"text-align: right;\">0.0880143</td><td style=\"text-align: right;\">             14098</td><td style=\"text-align: right;\">                 0.0058  </td><td>StackedEnsemble</td></tr>\n",
       "<tr><td>GBM_3_AutoML_1_20220121_163521                         </td><td style=\"text-align: right;\"> 0.269112</td><td style=\"text-align: right;\">0.855932</td><td style=\"text-align: right;\">0.367536</td><td style=\"text-align: right;\">              0.22818 </td><td style=\"text-align: right;\">0.29673 </td><td style=\"text-align: right;\">0.088049 </td><td style=\"text-align: right;\">              9526</td><td style=\"text-align: right;\">                 0.002241</td><td>GBM            </td></tr>\n",
       "<tr><td>StackedEnsemble_AllModels_5_AutoML_1_20220121_163521   </td><td style=\"text-align: right;\"> 0.269266</td><td style=\"text-align: right;\">0.855713</td><td style=\"text-align: right;\">0.362235</td><td style=\"text-align: right;\">              0.236468</td><td style=\"text-align: right;\">0.296738</td><td style=\"text-align: right;\">0.0880535</td><td style=\"text-align: right;\">             17350</td><td style=\"text-align: right;\">                 0.022831</td><td>StackedEnsemble</td></tr>\n",
       "<tr><td>StackedEnsemble_BestOfFamily_1_AutoML_1_20220121_163521</td><td style=\"text-align: right;\"> 0.269689</td><td style=\"text-align: right;\">0.854826</td><td style=\"text-align: right;\">0.364755</td><td style=\"text-align: right;\">              0.235164</td><td style=\"text-align: right;\">0.296939</td><td style=\"text-align: right;\">0.0881725</td><td style=\"text-align: right;\">              2240</td><td style=\"text-align: right;\">                 0.004272</td><td>StackedEnsemble</td></tr>\n",
       "<tr><td>GBM_1_AutoML_1_20220121_163521                         </td><td style=\"text-align: right;\"> 0.2702  </td><td style=\"text-align: right;\">0.854675</td><td style=\"text-align: right;\">0.36523 </td><td style=\"text-align: right;\">              0.232099</td><td style=\"text-align: right;\">0.297146</td><td style=\"text-align: right;\">0.0882957</td><td style=\"text-align: right;\">             11845</td><td style=\"text-align: right;\">                 0.002832</td><td>GBM            </td></tr>\n",
       "<tr><td>StackedEnsemble_BestOfFamily_4_AutoML_1_20220121_163521</td><td style=\"text-align: right;\"> 0.270251</td><td style=\"text-align: right;\">0.853664</td><td style=\"text-align: right;\">0.360066</td><td style=\"text-align: right;\">              0.23097 </td><td style=\"text-align: right;\">0.297339</td><td style=\"text-align: right;\">0.0884107</td><td style=\"text-align: right;\">             23548</td><td style=\"text-align: right;\">                 0.005015</td><td>StackedEnsemble</td></tr>\n",
       "<tr><td>GBM_4_AutoML_1_20220121_163521                         </td><td style=\"text-align: right;\"> 0.270573</td><td style=\"text-align: right;\">0.854889</td><td style=\"text-align: right;\">0.366482</td><td style=\"text-align: right;\">              0.234305</td><td style=\"text-align: right;\">0.297141</td><td style=\"text-align: right;\">0.0882925</td><td style=\"text-align: right;\">              8411</td><td style=\"text-align: right;\">                 0.002426</td><td>GBM            </td></tr>\n",
       "<tr><td>StackedEnsemble_AllModels_4_AutoML_1_20220121_163521   </td><td style=\"text-align: right;\"> 0.271489</td><td style=\"text-align: right;\">0.852343</td><td style=\"text-align: right;\">0.35501 </td><td style=\"text-align: right;\">              0.227192</td><td style=\"text-align: right;\">0.298188</td><td style=\"text-align: right;\">0.0889158</td><td style=\"text-align: right;\">             24735</td><td style=\"text-align: right;\">                 0.02179 </td><td>StackedEnsemble</td></tr>\n",
       "<tr><td>XGBoost_grid_1_AutoML_1_20220121_163521_model_2        </td><td style=\"text-align: right;\"> 0.272306</td><td style=\"text-align: right;\">0.851759</td><td style=\"text-align: right;\">0.35419 </td><td style=\"text-align: right;\">              0.241911</td><td style=\"text-align: right;\">0.298513</td><td style=\"text-align: right;\">0.0891102</td><td style=\"text-align: right;\">              2617</td><td style=\"text-align: right;\">                 0.001484</td><td>XGBoost        </td></tr>\n",
       "<tr><td>XGBoost_2_AutoML_1_20220121_163521                     </td><td style=\"text-align: right;\"> 0.273702</td><td style=\"text-align: right;\">0.849899</td><td style=\"text-align: right;\">0.352636</td><td style=\"text-align: right;\">              0.230878</td><td style=\"text-align: right;\">0.299754</td><td style=\"text-align: right;\">0.0898527</td><td style=\"text-align: right;\">              1976</td><td style=\"text-align: right;\">                 0.001462</td><td>XGBoost        </td></tr>\n",
       "<tr><td>XGBoost_grid_1_AutoML_1_20220121_163521_model_1        </td><td style=\"text-align: right;\"> 0.277766</td><td style=\"text-align: right;\">0.847015</td><td style=\"text-align: right;\">0.342767</td><td style=\"text-align: right;\">              0.229324</td><td style=\"text-align: right;\">0.3015  </td><td style=\"text-align: right;\">0.0909022</td><td style=\"text-align: right;\">              4016</td><td style=\"text-align: right;\">                 0.001673</td><td>XGBoost        </td></tr>\n",
       "<tr><td>DeepLearning_1_AutoML_1_20220121_163521                </td><td style=\"text-align: right;\"> 0.277793</td><td style=\"text-align: right;\">0.847569</td><td style=\"text-align: right;\">0.338316</td><td style=\"text-align: right;\">              0.23506 </td><td style=\"text-align: right;\">0.299883</td><td style=\"text-align: right;\">0.0899296</td><td style=\"text-align: right;\">              9306</td><td style=\"text-align: right;\">                 0.001814</td><td>DeepLearning   </td></tr>\n",
       "<tr><td>XGBoost_1_AutoML_1_20220121_163521                     </td><td style=\"text-align: right;\"> 0.279199</td><td style=\"text-align: right;\">0.845413</td><td style=\"text-align: right;\">0.345154</td><td style=\"text-align: right;\">              0.234903</td><td style=\"text-align: right;\">0.302745</td><td style=\"text-align: right;\">0.0916543</td><td style=\"text-align: right;\">              3189</td><td style=\"text-align: right;\">                 0.001637</td><td>XGBoost        </td></tr>\n",
       "<tr><td>DeepLearning_grid_1_AutoML_1_20220121_163521_model_1   </td><td style=\"text-align: right;\"> 0.316993</td><td style=\"text-align: right;\">0.841382</td><td style=\"text-align: right;\">0.326395</td><td style=\"text-align: right;\">              0.225108</td><td style=\"text-align: right;\">0.302785</td><td style=\"text-align: right;\">0.0916791</td><td style=\"text-align: right;\">             62659</td><td style=\"text-align: right;\">                 0.002145</td><td>DeepLearning   </td></tr>\n",
       "</tbody>\n",
       "</table>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Leaderboard saved in mlruns/1/6f2d4b47e37645b88a01aea5ba465e28/artifacts/model/leaderboard.csv\n"
     ]
    }
   ],
   "source": [
    "# Wrap autoML training with MLflow\n",
    "with mlflow.start_run():\n",
    "    aml = H2OAutoML(\n",
    "                    max_models=13, # Run AutoML for n base models\n",
    "                    seed=42, \n",
    "                    balance_classes=True, # Our target classes are imbalanced, so we set this to True\n",
    "                    sort_metric='logloss', # Sort models by logloss (main metric for multi-classification)\n",
    "                    verbosity='info', # Turn on verbose info\n",
    "                    exclude_algos = ['GLM', 'DRF'], # Specify which algorithms to exclude\n",
    "                   )\n",
    "    \n",
    "    aml.train(x=predictors, y=target, training_frame=main_frame)\n",
    "    \n",
    "    # Set metrics to log\n",
    "    mlflow.log_metric(\"log_loss\", aml.leader.logloss())\n",
    "    mlflow.log_metric(\"AUC\", aml.leader.auc())\n",
    "    \n",
    "    # Log best model (mlflow.h2o module provides API for logging & loading H2O models)\n",
    "    mlflow.h2o.log_model(aml.leader, \n",
    "                         artifact_path=\"model\"\n",
    "                        )\n",
    "    \n",
    "    model_uri = mlflow.get_artifact_uri(\"model\")\n",
    "    print(model_uri)\n",
    "    \n",
    "    # Print and view AutoML Leaderboard\n",
    "    lb = get_leaderboard(aml, extra_columns='ALL')\n",
    "    print(lb.head(rows=lb.nrows))\n",
    "    \n",
    "    # Get IDs of current experiment run\n",
    "    exp_id = experiment.experiment_id\n",
    "    run_id = mlflow.active_run().info.run_id\n",
    "    \n",
    "    # Save leaderboard as CSV\n",
    "    lb_path = f'mlruns/{exp_id}/{run_id}/artifacts/model/leaderboard.csv'\n",
    "    lb.as_data_frame().to_csv(lb_path, index=False) \n",
    "    print(f'Leaderboard saved in {lb_path}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### View AutoML logs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead>\n",
       "<tr><th>timestamp   </th><th>level  </th><th>stage     </th><th>message                                                                                                                                                           </th><th>name  </th><th>value  </th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "<tr><td>16:35:21.159</td><td>Info   </td><td>Workflow  </td><td>Project: AutoML_1_20220121_163521                                                                                                                                 </td><td>      </td><td>       </td></tr>\n",
       "<tr><td>16:35:21.165</td><td>Info   </td><td>Validation</td><td>Setting stopping tolerance adaptively based on the training frame: 0.0033065124063248246                                                                          </td><td>      </td><td>       </td></tr>\n",
       "<tr><td>16:35:21.166</td><td>Info   </td><td>Validation</td><td>Build control seed: 42                                                                                                                                            </td><td>      </td><td>       </td></tr>\n",
       "<tr><td>16:35:21.166</td><td>Info   </td><td>DataImport</td><td>training frame: Frame key: AutoML_1_20220121_163521_training_py_1_sid_8f18    cols: 208    rows: 91466  chunks: 32    size: 2498353  checksum: -277864416819399133</td><td>      </td><td>       </td></tr>\n",
       "<tr><td>16:35:21.166</td><td>Info   </td><td>DataImport</td><td>validation frame: NULL                                                                                                                                            </td><td>      </td><td>       </td></tr>\n",
       "<tr><td>16:35:21.167</td><td>Info   </td><td>DataImport</td><td>leaderboard frame: NULL                                                                                                                                           </td><td>      </td><td>       </td></tr>\n",
       "<tr><td>16:35:21.167</td><td>Info   </td><td>DataImport</td><td>blending frame: NULL                                                                                                                                              </td><td>      </td><td>       </td></tr>\n",
       "<tr><td>16:35:21.167</td><td>Info   </td><td>DataImport</td><td>response column: Response                                                                                                                                         </td><td>      </td><td>       </td></tr>\n",
       "<tr><td>16:35:21.167</td><td>Info   </td><td>DataImport</td><td>fold column: null                                                                                                                                                 </td><td>      </td><td>       </td></tr>\n",
       "<tr><td>16:35:21.167</td><td>Info   </td><td>DataImport</td><td>weights column: null                                                                                                                                              </td><td>      </td><td>       </td></tr>\n",
       "</tbody>\n",
       "</table>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": []
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get AutoML event log\n",
    "log = aml.event_log\n",
    "log"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### View best model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model Details\n",
      "=============\n",
      "H2OStackedEnsembleEstimator :  Stacked Ensemble\n",
      "Model Key:  StackedEnsemble_AllModels_2_AutoML_1_20220121_163521\n",
      "\n",
      "No model summary for this model\n",
      "\n",
      "ModelMetricsBinomialGLM: stackedensemble\n",
      "** Reported on train data. **\n",
      "\n",
      "MSE: 0.08537758056294917\n",
      "RMSE: 0.2921944225390847\n",
      "LogLoss: 0.2587869355251726\n",
      "Null degrees of freedom: 9974\n",
      "Residual degrees of freedom: 9966\n",
      "Null deviance: 7583.321730258964\n",
      "Residual deviance: 5162.799363727194\n",
      "AIC: 5180.799363727194\n",
      "AUC: 0.876164866479019\n",
      "AUCPR: 0.4475357523278229\n",
      "Gini: 0.7523297329580381\n",
      "\n",
      "Confusion Matrix (Act/Pred) for max f1 @ threshold = 0.24313533374730487: \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>Error</th>\n",
       "      <th>Rate</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>6792.0</td>\n",
       "      <td>1919.0</td>\n",
       "      <td>0.2203</td>\n",
       "      <td>(1919.0/8711.0)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>222.0</td>\n",
       "      <td>1042.0</td>\n",
       "      <td>0.1756</td>\n",
       "      <td>(222.0/1264.0)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Total</td>\n",
       "      <td>7014.0</td>\n",
       "      <td>2961.0</td>\n",
       "      <td>0.2146</td>\n",
       "      <td>(2141.0/9975.0)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               0       1   Error              Rate\n",
       "0      0  6792.0  1919.0  0.2203   (1919.0/8711.0)\n",
       "1      1   222.0  1042.0  0.1756    (222.0/1264.0)\n",
       "2  Total  7014.0  2961.0  0.2146   (2141.0/9975.0)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Maximum Metrics: Maximum metrics at their respective thresholds\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>metric</th>\n",
       "      <th>threshold</th>\n",
       "      <th>value</th>\n",
       "      <th>idx</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>max f1</td>\n",
       "      <td>0.243135</td>\n",
       "      <td>0.493254</td>\n",
       "      <td>195.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>max f2</td>\n",
       "      <td>0.138887</td>\n",
       "      <td>0.665671</td>\n",
       "      <td>280.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>max f0point5</td>\n",
       "      <td>0.351315</td>\n",
       "      <td>0.440540</td>\n",
       "      <td>96.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>max accuracy</td>\n",
       "      <td>0.421353</td>\n",
       "      <td>0.878797</td>\n",
       "      <td>38.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>max precision</td>\n",
       "      <td>0.584605</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>max recall</td>\n",
       "      <td>0.015326</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>377.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>max specificity</td>\n",
       "      <td>0.584605</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>max absolute_mcc</td>\n",
       "      <td>0.240001</td>\n",
       "      <td>0.440294</td>\n",
       "      <td>198.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>max min_per_class_accuracy</td>\n",
       "      <td>0.256273</td>\n",
       "      <td>0.791930</td>\n",
       "      <td>184.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>max mean_per_class_accuracy</td>\n",
       "      <td>0.138887</td>\n",
       "      <td>0.816474</td>\n",
       "      <td>280.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>max tns</td>\n",
       "      <td>0.584605</td>\n",
       "      <td>8711.000000</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>max fns</td>\n",
       "      <td>0.584605</td>\n",
       "      <td>1263.000000</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>max fps</td>\n",
       "      <td>0.000575</td>\n",
       "      <td>8711.000000</td>\n",
       "      <td>399.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>max tps</td>\n",
       "      <td>0.015326</td>\n",
       "      <td>1264.000000</td>\n",
       "      <td>377.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>max tnr</td>\n",
       "      <td>0.584605</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>max fnr</td>\n",
       "      <td>0.584605</td>\n",
       "      <td>0.999209</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>max fpr</td>\n",
       "      <td>0.000575</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>399.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>max tpr</td>\n",
       "      <td>0.015326</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>377.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                         metric  threshold        value    idx\n",
       "0                        max f1   0.243135     0.493254  195.0\n",
       "1                        max f2   0.138887     0.665671  280.0\n",
       "2                  max f0point5   0.351315     0.440540   96.0\n",
       "3                  max accuracy   0.421353     0.878797   38.0\n",
       "4                 max precision   0.584605     1.000000    0.0\n",
       "5                    max recall   0.015326     1.000000  377.0\n",
       "6               max specificity   0.584605     1.000000    0.0\n",
       "7              max absolute_mcc   0.240001     0.440294  198.0\n",
       "8    max min_per_class_accuracy   0.256273     0.791930  184.0\n",
       "9   max mean_per_class_accuracy   0.138887     0.816474  280.0\n",
       "10                      max tns   0.584605  8711.000000    0.0\n",
       "11                      max fns   0.584605  1263.000000    0.0\n",
       "12                      max fps   0.000575  8711.000000  399.0\n",
       "13                      max tps   0.015326  1264.000000  377.0\n",
       "14                      max tnr   0.584605     1.000000    0.0\n",
       "15                      max fnr   0.584605     0.999209    0.0\n",
       "16                      max fpr   0.000575     1.000000  399.0\n",
       "17                      max tpr   0.015326     1.000000  377.0"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Gains/Lift Table: Avg response rate: 12.67 %, avg score: 13.03 %\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>group</th>\n",
       "      <th>cumulative_data_fraction</th>\n",
       "      <th>lower_threshold</th>\n",
       "      <th>lift</th>\n",
       "      <th>cumulative_lift</th>\n",
       "      <th>response_rate</th>\n",
       "      <th>score</th>\n",
       "      <th>cumulative_response_rate</th>\n",
       "      <th>cumulative_score</th>\n",
       "      <th>capture_rate</th>\n",
       "      <th>cumulative_capture_rate</th>\n",
       "      <th>gain</th>\n",
       "      <th>cumulative_gain</th>\n",
       "      <th>kolmogorov_smirnov</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0.010025</td>\n",
       "      <td>0.442034</td>\n",
       "      <td>5.603046</td>\n",
       "      <td>5.603046</td>\n",
       "      <td>0.710000</td>\n",
       "      <td>0.468182</td>\n",
       "      <td>0.710000</td>\n",
       "      <td>0.468182</td>\n",
       "      <td>0.056171</td>\n",
       "      <td>0.056171</td>\n",
       "      <td>460.304589</td>\n",
       "      <td>460.304589</td>\n",
       "      <td>0.052842</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>0.020050</td>\n",
       "      <td>0.421141</td>\n",
       "      <td>4.419304</td>\n",
       "      <td>5.011175</td>\n",
       "      <td>0.560000</td>\n",
       "      <td>0.429921</td>\n",
       "      <td>0.635000</td>\n",
       "      <td>0.449052</td>\n",
       "      <td>0.044304</td>\n",
       "      <td>0.100475</td>\n",
       "      <td>341.930380</td>\n",
       "      <td>401.117484</td>\n",
       "      <td>0.092094</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>0.030075</td>\n",
       "      <td>0.408114</td>\n",
       "      <td>3.314478</td>\n",
       "      <td>4.445609</td>\n",
       "      <td>0.420000</td>\n",
       "      <td>0.414990</td>\n",
       "      <td>0.563333</td>\n",
       "      <td>0.437698</td>\n",
       "      <td>0.033228</td>\n",
       "      <td>0.133703</td>\n",
       "      <td>231.447785</td>\n",
       "      <td>344.560918</td>\n",
       "      <td>0.118664</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>0.040000</td>\n",
       "      <td>0.397592</td>\n",
       "      <td>3.666811</td>\n",
       "      <td>4.252373</td>\n",
       "      <td>0.464646</td>\n",
       "      <td>0.402418</td>\n",
       "      <td>0.538847</td>\n",
       "      <td>0.428944</td>\n",
       "      <td>0.036392</td>\n",
       "      <td>0.170095</td>\n",
       "      <td>266.681051</td>\n",
       "      <td>325.237342</td>\n",
       "      <td>0.148972</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>0.050025</td>\n",
       "      <td>0.391139</td>\n",
       "      <td>3.077729</td>\n",
       "      <td>4.016974</td>\n",
       "      <td>0.390000</td>\n",
       "      <td>0.394060</td>\n",
       "      <td>0.509018</td>\n",
       "      <td>0.421953</td>\n",
       "      <td>0.030854</td>\n",
       "      <td>0.200949</td>\n",
       "      <td>207.772943</td>\n",
       "      <td>301.697382</td>\n",
       "      <td>0.172824</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>6</td>\n",
       "      <td>0.100050</td>\n",
       "      <td>0.361629</td>\n",
       "      <td>3.083897</td>\n",
       "      <td>3.550436</td>\n",
       "      <td>0.390782</td>\n",
       "      <td>0.376552</td>\n",
       "      <td>0.449900</td>\n",
       "      <td>0.399253</td>\n",
       "      <td>0.154272</td>\n",
       "      <td>0.355222</td>\n",
       "      <td>208.389722</td>\n",
       "      <td>255.043552</td>\n",
       "      <td>0.292198</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>7</td>\n",
       "      <td>0.150075</td>\n",
       "      <td>0.333060</td>\n",
       "      <td>2.815045</td>\n",
       "      <td>3.305305</td>\n",
       "      <td>0.356713</td>\n",
       "      <td>0.347117</td>\n",
       "      <td>0.418838</td>\n",
       "      <td>0.381874</td>\n",
       "      <td>0.140823</td>\n",
       "      <td>0.496044</td>\n",
       "      <td>181.504465</td>\n",
       "      <td>230.530523</td>\n",
       "      <td>0.396171</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>8</td>\n",
       "      <td>0.200000</td>\n",
       "      <td>0.309387</td>\n",
       "      <td>2.408685</td>\n",
       "      <td>3.081487</td>\n",
       "      <td>0.305221</td>\n",
       "      <td>0.321366</td>\n",
       "      <td>0.390476</td>\n",
       "      <td>0.366770</td>\n",
       "      <td>0.120253</td>\n",
       "      <td>0.616297</td>\n",
       "      <td>140.868537</td>\n",
       "      <td>208.148734</td>\n",
       "      <td>0.476704</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>9</td>\n",
       "      <td>0.300050</td>\n",
       "      <td>0.239757</td>\n",
       "      <td>2.135006</td>\n",
       "      <td>2.765888</td>\n",
       "      <td>0.270541</td>\n",
       "      <td>0.278391</td>\n",
       "      <td>0.350484</td>\n",
       "      <td>0.337300</td>\n",
       "      <td>0.213608</td>\n",
       "      <td>0.829905</td>\n",
       "      <td>113.500577</td>\n",
       "      <td>176.588807</td>\n",
       "      <td>0.606739</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>10</td>\n",
       "      <td>0.400000</td>\n",
       "      <td>0.137013</td>\n",
       "      <td>1.218965</td>\n",
       "      <td>2.379351</td>\n",
       "      <td>0.154463</td>\n",
       "      <td>0.188820</td>\n",
       "      <td>0.301504</td>\n",
       "      <td>0.300199</td>\n",
       "      <td>0.121835</td>\n",
       "      <td>0.951741</td>\n",
       "      <td>21.896544</td>\n",
       "      <td>137.935127</td>\n",
       "      <td>0.631800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>11</td>\n",
       "      <td>0.500050</td>\n",
       "      <td>0.036950</td>\n",
       "      <td>0.450723</td>\n",
       "      <td>1.993471</td>\n",
       "      <td>0.057114</td>\n",
       "      <td>0.086476</td>\n",
       "      <td>0.252606</td>\n",
       "      <td>0.257437</td>\n",
       "      <td>0.045095</td>\n",
       "      <td>0.996835</td>\n",
       "      <td>-54.927656</td>\n",
       "      <td>99.347104</td>\n",
       "      <td>0.568871</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>12</td>\n",
       "      <td>0.600000</td>\n",
       "      <td>0.001310</td>\n",
       "      <td>0.031661</td>\n",
       "      <td>1.666667</td>\n",
       "      <td>0.004012</td>\n",
       "      <td>0.012880</td>\n",
       "      <td>0.211195</td>\n",
       "      <td>0.216698</td>\n",
       "      <td>0.003165</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>-96.833856</td>\n",
       "      <td>66.666667</td>\n",
       "      <td>0.458042</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>13</td>\n",
       "      <td>0.699950</td>\n",
       "      <td>0.000890</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.428674</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.001029</td>\n",
       "      <td>0.181037</td>\n",
       "      <td>0.185901</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>-100.000000</td>\n",
       "      <td>42.867373</td>\n",
       "      <td>0.343589</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>14</td>\n",
       "      <td>0.800000</td>\n",
       "      <td>0.000764</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.250000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000822</td>\n",
       "      <td>0.158396</td>\n",
       "      <td>0.162755</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>-100.000000</td>\n",
       "      <td>25.000000</td>\n",
       "      <td>0.229021</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>15</td>\n",
       "      <td>0.899950</td>\n",
       "      <td>0.000670</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.111173</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000717</td>\n",
       "      <td>0.140804</td>\n",
       "      <td>0.144759</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>-100.000000</td>\n",
       "      <td>11.117300</td>\n",
       "      <td>0.114568</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>16</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000343</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000599</td>\n",
       "      <td>0.126717</td>\n",
       "      <td>0.130335</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>-100.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    group  cumulative_data_fraction  lower_threshold      lift  \\\n",
       "0       1                  0.010025         0.442034  5.603046   \n",
       "1       2                  0.020050         0.421141  4.419304   \n",
       "2       3                  0.030075         0.408114  3.314478   \n",
       "3       4                  0.040000         0.397592  3.666811   \n",
       "4       5                  0.050025         0.391139  3.077729   \n",
       "5       6                  0.100050         0.361629  3.083897   \n",
       "6       7                  0.150075         0.333060  2.815045   \n",
       "7       8                  0.200000         0.309387  2.408685   \n",
       "8       9                  0.300050         0.239757  2.135006   \n",
       "9      10                  0.400000         0.137013  1.218965   \n",
       "10     11                  0.500050         0.036950  0.450723   \n",
       "11     12                  0.600000         0.001310  0.031661   \n",
       "12     13                  0.699950         0.000890  0.000000   \n",
       "13     14                  0.800000         0.000764  0.000000   \n",
       "14     15                  0.899950         0.000670  0.000000   \n",
       "15     16                  1.000000         0.000343  0.000000   \n",
       "\n",
       "    cumulative_lift  response_rate     score  cumulative_response_rate  \\\n",
       "0          5.603046       0.710000  0.468182                  0.710000   \n",
       "1          5.011175       0.560000  0.429921                  0.635000   \n",
       "2          4.445609       0.420000  0.414990                  0.563333   \n",
       "3          4.252373       0.464646  0.402418                  0.538847   \n",
       "4          4.016974       0.390000  0.394060                  0.509018   \n",
       "5          3.550436       0.390782  0.376552                  0.449900   \n",
       "6          3.305305       0.356713  0.347117                  0.418838   \n",
       "7          3.081487       0.305221  0.321366                  0.390476   \n",
       "8          2.765888       0.270541  0.278391                  0.350484   \n",
       "9          2.379351       0.154463  0.188820                  0.301504   \n",
       "10         1.993471       0.057114  0.086476                  0.252606   \n",
       "11         1.666667       0.004012  0.012880                  0.211195   \n",
       "12         1.428674       0.000000  0.001029                  0.181037   \n",
       "13         1.250000       0.000000  0.000822                  0.158396   \n",
       "14         1.111173       0.000000  0.000717                  0.140804   \n",
       "15         1.000000       0.000000  0.000599                  0.126717   \n",
       "\n",
       "    cumulative_score  capture_rate  cumulative_capture_rate        gain  \\\n",
       "0           0.468182      0.056171                 0.056171  460.304589   \n",
       "1           0.449052      0.044304                 0.100475  341.930380   \n",
       "2           0.437698      0.033228                 0.133703  231.447785   \n",
       "3           0.428944      0.036392                 0.170095  266.681051   \n",
       "4           0.421953      0.030854                 0.200949  207.772943   \n",
       "5           0.399253      0.154272                 0.355222  208.389722   \n",
       "6           0.381874      0.140823                 0.496044  181.504465   \n",
       "7           0.366770      0.120253                 0.616297  140.868537   \n",
       "8           0.337300      0.213608                 0.829905  113.500577   \n",
       "9           0.300199      0.121835                 0.951741   21.896544   \n",
       "10          0.257437      0.045095                 0.996835  -54.927656   \n",
       "11          0.216698      0.003165                 1.000000  -96.833856   \n",
       "12          0.185901      0.000000                 1.000000 -100.000000   \n",
       "13          0.162755      0.000000                 1.000000 -100.000000   \n",
       "14          0.144759      0.000000                 1.000000 -100.000000   \n",
       "15          0.130335      0.000000                 1.000000 -100.000000   \n",
       "\n",
       "    cumulative_gain  kolmogorov_smirnov  \n",
       "0        460.304589            0.052842  \n",
       "1        401.117484            0.092094  \n",
       "2        344.560918            0.118664  \n",
       "3        325.237342            0.148972  \n",
       "4        301.697382            0.172824  \n",
       "5        255.043552            0.292198  \n",
       "6        230.530523            0.396171  \n",
       "7        208.148734            0.476704  \n",
       "8        176.588807            0.606739  \n",
       "9        137.935127            0.631800  \n",
       "10        99.347104            0.568871  \n",
       "11        66.666667            0.458042  \n",
       "12        42.867373            0.343589  \n",
       "13        25.000000            0.229021  \n",
       "14        11.117300            0.114568  \n",
       "15         0.000000            0.000000  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "ModelMetricsBinomialGLM: stackedensemble\n",
      "** Reported on cross-validation data. **\n",
      "\n",
      "MSE: 0.08770362334242507\n",
      "RMSE: 0.2961479754150365\n",
      "LogLoss: 0.26769269449672595\n",
      "Null degrees of freedom: 91465\n",
      "Residual degrees of freedom: 91457\n",
      "Null deviance: 68811.90506765014\n",
      "Residual deviance: 48969.55998967507\n",
      "AIC: 48987.55998967507\n",
      "AUC: 0.8575608446815696\n",
      "AUCPR: 0.3700568332452403\n",
      "Gini: 0.7151216893631391\n",
      "\n",
      "Confusion Matrix (Act/Pred) for max f1 @ threshold = 0.25114266443266603: \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>Error</th>\n",
       "      <th>Rate</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>63881.0</td>\n",
       "      <td>16181.0</td>\n",
       "      <td>0.2021</td>\n",
       "      <td>(16181.0/80062.0)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>3115.0</td>\n",
       "      <td>8289.0</td>\n",
       "      <td>0.2731</td>\n",
       "      <td>(3115.0/11404.0)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Total</td>\n",
       "      <td>66996.0</td>\n",
       "      <td>24470.0</td>\n",
       "      <td>0.211</td>\n",
       "      <td>(19296.0/91466.0)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                0        1   Error                Rate\n",
       "0      0  63881.0  16181.0  0.2021   (16181.0/80062.0)\n",
       "1      1   3115.0   8289.0  0.2731    (3115.0/11404.0)\n",
       "2  Total  66996.0  24470.0   0.211   (19296.0/91466.0)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Maximum Metrics: Maximum metrics at their respective thresholds\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>metric</th>\n",
       "      <th>threshold</th>\n",
       "      <th>value</th>\n",
       "      <th>idx</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>max f1</td>\n",
       "      <td>0.251143</td>\n",
       "      <td>0.462117</td>\n",
       "      <td>187.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>max f2</td>\n",
       "      <td>0.124363</td>\n",
       "      <td>0.642087</td>\n",
       "      <td>288.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>max f0point5</td>\n",
       "      <td>0.312132</td>\n",
       "      <td>0.395582</td>\n",
       "      <td>131.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>max accuracy</td>\n",
       "      <td>0.518872</td>\n",
       "      <td>0.875396</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>max precision</td>\n",
       "      <td>0.565370</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>max recall</td>\n",
       "      <td>0.000439</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>399.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>max specificity</td>\n",
       "      <td>0.565370</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>max absolute_mcc</td>\n",
       "      <td>0.144915</td>\n",
       "      <td>0.406708</td>\n",
       "      <td>272.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>max min_per_class_accuracy</td>\n",
       "      <td>0.226797</td>\n",
       "      <td>0.773325</td>\n",
       "      <td>207.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>max mean_per_class_accuracy</td>\n",
       "      <td>0.124363</td>\n",
       "      <td>0.800319</td>\n",
       "      <td>288.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>max tns</td>\n",
       "      <td>0.565370</td>\n",
       "      <td>80062.000000</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>max fns</td>\n",
       "      <td>0.565370</td>\n",
       "      <td>11401.000000</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>max fps</td>\n",
       "      <td>0.000439</td>\n",
       "      <td>80062.000000</td>\n",
       "      <td>399.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>max tps</td>\n",
       "      <td>0.000439</td>\n",
       "      <td>11404.000000</td>\n",
       "      <td>399.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>max tnr</td>\n",
       "      <td>0.565370</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>max fnr</td>\n",
       "      <td>0.565370</td>\n",
       "      <td>0.999737</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>max fpr</td>\n",
       "      <td>0.000439</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>399.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>max tpr</td>\n",
       "      <td>0.000439</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>399.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                         metric  threshold         value    idx\n",
       "0                        max f1   0.251143      0.462117  187.0\n",
       "1                        max f2   0.124363      0.642087  288.0\n",
       "2                  max f0point5   0.312132      0.395582  131.0\n",
       "3                  max accuracy   0.518872      0.875396    5.0\n",
       "4                 max precision   0.565370      1.000000    0.0\n",
       "5                    max recall   0.000439      1.000000  399.0\n",
       "6               max specificity   0.565370      1.000000    0.0\n",
       "7              max absolute_mcc   0.144915      0.406708  272.0\n",
       "8    max min_per_class_accuracy   0.226797      0.773325  207.0\n",
       "9   max mean_per_class_accuracy   0.124363      0.800319  288.0\n",
       "10                      max tns   0.565370  80062.000000    0.0\n",
       "11                      max fns   0.565370  11401.000000    0.0\n",
       "12                      max fps   0.000439  80062.000000  399.0\n",
       "13                      max tps   0.000439  11404.000000  399.0\n",
       "14                      max tnr   0.565370      1.000000    0.0\n",
       "15                      max fnr   0.565370      0.999737    0.0\n",
       "16                      max fpr   0.000439      1.000000  399.0\n",
       "17                      max tpr   0.000439      1.000000  399.0"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Gains/Lift Table: Avg response rate: 12.47 %, avg score: 12.47 %\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>group</th>\n",
       "      <th>cumulative_data_fraction</th>\n",
       "      <th>lower_threshold</th>\n",
       "      <th>lift</th>\n",
       "      <th>cumulative_lift</th>\n",
       "      <th>response_rate</th>\n",
       "      <th>score</th>\n",
       "      <th>cumulative_response_rate</th>\n",
       "      <th>cumulative_score</th>\n",
       "      <th>capture_rate</th>\n",
       "      <th>cumulative_capture_rate</th>\n",
       "      <th>gain</th>\n",
       "      <th>cumulative_gain</th>\n",
       "      <th>kolmogorov_smirnov</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0.010004</td>\n",
       "      <td>0.434211</td>\n",
       "      <td>3.453644</td>\n",
       "      <td>3.453644</td>\n",
       "      <td>0.430601</td>\n",
       "      <td>0.458590</td>\n",
       "      <td>0.430601</td>\n",
       "      <td>0.458590</td>\n",
       "      <td>0.034549</td>\n",
       "      <td>0.034549</td>\n",
       "      <td>245.364430</td>\n",
       "      <td>245.364430</td>\n",
       "      <td>0.028042</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>0.020007</td>\n",
       "      <td>0.414185</td>\n",
       "      <td>3.646487</td>\n",
       "      <td>3.550066</td>\n",
       "      <td>0.454645</td>\n",
       "      <td>0.423167</td>\n",
       "      <td>0.442623</td>\n",
       "      <td>0.440878</td>\n",
       "      <td>0.036478</td>\n",
       "      <td>0.071028</td>\n",
       "      <td>264.648738</td>\n",
       "      <td>255.006584</td>\n",
       "      <td>0.058288</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>0.030000</td>\n",
       "      <td>0.401793</td>\n",
       "      <td>3.325795</td>\n",
       "      <td>3.475363</td>\n",
       "      <td>0.414661</td>\n",
       "      <td>0.407662</td>\n",
       "      <td>0.433309</td>\n",
       "      <td>0.429814</td>\n",
       "      <td>0.033234</td>\n",
       "      <td>0.104262</td>\n",
       "      <td>232.579513</td>\n",
       "      <td>247.536342</td>\n",
       "      <td>0.084839</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>0.040004</td>\n",
       "      <td>0.391726</td>\n",
       "      <td>3.339692</td>\n",
       "      <td>3.441436</td>\n",
       "      <td>0.416393</td>\n",
       "      <td>0.396498</td>\n",
       "      <td>0.429079</td>\n",
       "      <td>0.421483</td>\n",
       "      <td>0.033409</td>\n",
       "      <td>0.137671</td>\n",
       "      <td>233.969157</td>\n",
       "      <td>244.143619</td>\n",
       "      <td>0.111579</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>0.050008</td>\n",
       "      <td>0.384244</td>\n",
       "      <td>3.111786</td>\n",
       "      <td>3.375492</td>\n",
       "      <td>0.387978</td>\n",
       "      <td>0.387868</td>\n",
       "      <td>0.420857</td>\n",
       "      <td>0.414759</td>\n",
       "      <td>0.031129</td>\n",
       "      <td>0.168800</td>\n",
       "      <td>211.178611</td>\n",
       "      <td>237.549176</td>\n",
       "      <td>0.135714</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>6</td>\n",
       "      <td>0.100004</td>\n",
       "      <td>0.353987</td>\n",
       "      <td>3.121916</td>\n",
       "      <td>3.248718</td>\n",
       "      <td>0.389241</td>\n",
       "      <td>0.369155</td>\n",
       "      <td>0.405051</td>\n",
       "      <td>0.391960</td>\n",
       "      <td>0.156086</td>\n",
       "      <td>0.324886</td>\n",
       "      <td>212.191647</td>\n",
       "      <td>224.871798</td>\n",
       "      <td>0.256914</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>7</td>\n",
       "      <td>0.150001</td>\n",
       "      <td>0.325117</td>\n",
       "      <td>2.657137</td>\n",
       "      <td>3.051539</td>\n",
       "      <td>0.331292</td>\n",
       "      <td>0.339150</td>\n",
       "      <td>0.380466</td>\n",
       "      <td>0.374358</td>\n",
       "      <td>0.132848</td>\n",
       "      <td>0.457734</td>\n",
       "      <td>165.713677</td>\n",
       "      <td>205.153861</td>\n",
       "      <td>0.351566</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>8</td>\n",
       "      <td>0.200009</td>\n",
       "      <td>0.299131</td>\n",
       "      <td>2.474192</td>\n",
       "      <td>2.907186</td>\n",
       "      <td>0.308483</td>\n",
       "      <td>0.312427</td>\n",
       "      <td>0.362469</td>\n",
       "      <td>0.358873</td>\n",
       "      <td>0.123729</td>\n",
       "      <td>0.581463</td>\n",
       "      <td>147.419162</td>\n",
       "      <td>190.718609</td>\n",
       "      <td>0.435788</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>9</td>\n",
       "      <td>0.300002</td>\n",
       "      <td>0.221014</td>\n",
       "      <td>2.031000</td>\n",
       "      <td>2.615145</td>\n",
       "      <td>0.253225</td>\n",
       "      <td>0.263340</td>\n",
       "      <td>0.326057</td>\n",
       "      <td>0.327031</td>\n",
       "      <td>0.203087</td>\n",
       "      <td>0.784549</td>\n",
       "      <td>103.099959</td>\n",
       "      <td>161.514521</td>\n",
       "      <td>0.553566</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>10</td>\n",
       "      <td>0.400007</td>\n",
       "      <td>0.126414</td>\n",
       "      <td>1.403832</td>\n",
       "      <td>2.312309</td>\n",
       "      <td>0.175030</td>\n",
       "      <td>0.173419</td>\n",
       "      <td>0.288299</td>\n",
       "      <td>0.288627</td>\n",
       "      <td>0.140389</td>\n",
       "      <td>0.924939</td>\n",
       "      <td>40.383198</td>\n",
       "      <td>131.230862</td>\n",
       "      <td>0.599703</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>11</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>0.032019</td>\n",
       "      <td>0.633153</td>\n",
       "      <td>1.976499</td>\n",
       "      <td>0.078942</td>\n",
       "      <td>0.077502</td>\n",
       "      <td>0.246430</td>\n",
       "      <td>0.246405</td>\n",
       "      <td>0.063311</td>\n",
       "      <td>0.988250</td>\n",
       "      <td>-36.684728</td>\n",
       "      <td>97.649947</td>\n",
       "      <td>0.557796</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>12</td>\n",
       "      <td>0.600004</td>\n",
       "      <td>0.001388</td>\n",
       "      <td>0.106098</td>\n",
       "      <td>1.664755</td>\n",
       "      <td>0.013228</td>\n",
       "      <td>0.011599</td>\n",
       "      <td>0.207562</td>\n",
       "      <td>0.207269</td>\n",
       "      <td>0.010610</td>\n",
       "      <td>0.998860</td>\n",
       "      <td>-89.390152</td>\n",
       "      <td>66.475461</td>\n",
       "      <td>0.455669</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>13</td>\n",
       "      <td>0.699998</td>\n",
       "      <td>0.000938</td>\n",
       "      <td>0.003508</td>\n",
       "      <td>1.427448</td>\n",
       "      <td>0.000437</td>\n",
       "      <td>0.001096</td>\n",
       "      <td>0.177975</td>\n",
       "      <td>0.177818</td>\n",
       "      <td>0.000351</td>\n",
       "      <td>0.999211</td>\n",
       "      <td>-99.649223</td>\n",
       "      <td>42.744846</td>\n",
       "      <td>0.341833</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>14</td>\n",
       "      <td>0.800002</td>\n",
       "      <td>0.000783</td>\n",
       "      <td>0.004384</td>\n",
       "      <td>1.249558</td>\n",
       "      <td>0.000547</td>\n",
       "      <td>0.000856</td>\n",
       "      <td>0.155795</td>\n",
       "      <td>0.155696</td>\n",
       "      <td>0.000438</td>\n",
       "      <td>0.999649</td>\n",
       "      <td>-99.561577</td>\n",
       "      <td>24.955814</td>\n",
       "      <td>0.228085</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>15</td>\n",
       "      <td>0.899996</td>\n",
       "      <td>0.000647</td>\n",
       "      <td>0.001754</td>\n",
       "      <td>1.110922</td>\n",
       "      <td>0.000219</td>\n",
       "      <td>0.000717</td>\n",
       "      <td>0.138510</td>\n",
       "      <td>0.138478</td>\n",
       "      <td>0.000175</td>\n",
       "      <td>0.999825</td>\n",
       "      <td>-99.824611</td>\n",
       "      <td>11.092165</td>\n",
       "      <td>0.114049</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>16</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000149</td>\n",
       "      <td>0.001754</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000219</td>\n",
       "      <td>0.000531</td>\n",
       "      <td>0.124680</td>\n",
       "      <td>0.124682</td>\n",
       "      <td>0.000175</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>-99.824631</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    group  cumulative_data_fraction  lower_threshold      lift  \\\n",
       "0       1                  0.010004         0.434211  3.453644   \n",
       "1       2                  0.020007         0.414185  3.646487   \n",
       "2       3                  0.030000         0.401793  3.325795   \n",
       "3       4                  0.040004         0.391726  3.339692   \n",
       "4       5                  0.050008         0.384244  3.111786   \n",
       "5       6                  0.100004         0.353987  3.121916   \n",
       "6       7                  0.150001         0.325117  2.657137   \n",
       "7       8                  0.200009         0.299131  2.474192   \n",
       "8       9                  0.300002         0.221014  2.031000   \n",
       "9      10                  0.400007         0.126414  1.403832   \n",
       "10     11                  0.500000         0.032019  0.633153   \n",
       "11     12                  0.600004         0.001388  0.106098   \n",
       "12     13                  0.699998         0.000938  0.003508   \n",
       "13     14                  0.800002         0.000783  0.004384   \n",
       "14     15                  0.899996         0.000647  0.001754   \n",
       "15     16                  1.000000         0.000149  0.001754   \n",
       "\n",
       "    cumulative_lift  response_rate     score  cumulative_response_rate  \\\n",
       "0          3.453644       0.430601  0.458590                  0.430601   \n",
       "1          3.550066       0.454645  0.423167                  0.442623   \n",
       "2          3.475363       0.414661  0.407662                  0.433309   \n",
       "3          3.441436       0.416393  0.396498                  0.429079   \n",
       "4          3.375492       0.387978  0.387868                  0.420857   \n",
       "5          3.248718       0.389241  0.369155                  0.405051   \n",
       "6          3.051539       0.331292  0.339150                  0.380466   \n",
       "7          2.907186       0.308483  0.312427                  0.362469   \n",
       "8          2.615145       0.253225  0.263340                  0.326057   \n",
       "9          2.312309       0.175030  0.173419                  0.288299   \n",
       "10         1.976499       0.078942  0.077502                  0.246430   \n",
       "11         1.664755       0.013228  0.011599                  0.207562   \n",
       "12         1.427448       0.000437  0.001096                  0.177975   \n",
       "13         1.249558       0.000547  0.000856                  0.155795   \n",
       "14         1.110922       0.000219  0.000717                  0.138510   \n",
       "15         1.000000       0.000219  0.000531                  0.124680   \n",
       "\n",
       "    cumulative_score  capture_rate  cumulative_capture_rate        gain  \\\n",
       "0           0.458590      0.034549                 0.034549  245.364430   \n",
       "1           0.440878      0.036478                 0.071028  264.648738   \n",
       "2           0.429814      0.033234                 0.104262  232.579513   \n",
       "3           0.421483      0.033409                 0.137671  233.969157   \n",
       "4           0.414759      0.031129                 0.168800  211.178611   \n",
       "5           0.391960      0.156086                 0.324886  212.191647   \n",
       "6           0.374358      0.132848                 0.457734  165.713677   \n",
       "7           0.358873      0.123729                 0.581463  147.419162   \n",
       "8           0.327031      0.203087                 0.784549  103.099959   \n",
       "9           0.288627      0.140389                 0.924939   40.383198   \n",
       "10          0.246405      0.063311                 0.988250  -36.684728   \n",
       "11          0.207269      0.010610                 0.998860  -89.390152   \n",
       "12          0.177818      0.000351                 0.999211  -99.649223   \n",
       "13          0.155696      0.000438                 0.999649  -99.561577   \n",
       "14          0.138478      0.000175                 0.999825  -99.824611   \n",
       "15          0.124682      0.000175                 1.000000  -99.824631   \n",
       "\n",
       "    cumulative_gain  kolmogorov_smirnov  \n",
       "0        245.364430            0.028042  \n",
       "1        255.006584            0.058288  \n",
       "2        247.536342            0.084839  \n",
       "3        244.143619            0.111579  \n",
       "4        237.549176            0.135714  \n",
       "5        224.871798            0.256914  \n",
       "6        205.153861            0.351566  \n",
       "7        190.718609            0.435788  \n",
       "8        161.514521            0.553566  \n",
       "9        131.230862            0.599703  \n",
       "10        97.649947            0.557796  \n",
       "11        66.475461            0.455669  \n",
       "12        42.744846            0.341833  \n",
       "13        24.955814            0.228085  \n",
       "14        11.092165            0.114049  \n",
       "15         0.000000            0.000000  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": []
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Leader (best) model stored here\n",
    "aml.leader"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Learning Curve Plot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA7wAAAI1CAYAAAD1i/aTAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAABWM0lEQVR4nO3deZxWZf3/8deHYRUQDZUU3MoF1BBkFMFt3C1zS0vNSlNzyaWsLGlVv+nPLCtNy0zLNlfMXVMzcUNTUFSUVUNF0RRjGZVtuH5/3IfpZpgZbmDuuWfOvJ6Pxzy4z7nPuc7nnPu6h3nf1znnjpQSkiRJkiTlTadKFyBJkiRJUjkYeCVJkiRJuWTglSRJkiTlkoFXkiRJkpRLBl5JkiRJUi4ZeCVJkiRJuWTglSSpjYuI3SJiSqXrkCSpvTHwSpLUjIiYERH7VLKGlNKjKaWty9V+ROwfEY9ExPyIeCciHo6Ig8u1PUmSWouBV5KkCouIqgpu+wjgZuBPwACgH/BD4KDVaCsiwr8tJElthv8pSZK0GiKiU0ScExEvR8TsiLgpIj5S9PzNEfFWRMzNRk+3LXru2oj4TUTcExHvA3tmI8nfiojns3VujIju2fI1ETGzaP0ml82e/3ZEzIqINyPixIhIEbFFI/sQwM+B/0spXZ1SmptSWppSejil9JVsmXMj4i9F62yWtdc5mx4TERdExOPAB8DZETGuwXbOiog7ssfdIuJnEfFaRLwdEVdGRI81fDkkSWqUgVeSpNVzBnAosAewEfBf4Iqi5+8FtgQ2AJ4B/tpg/c8DFwC9gceyeZ8DDgA2BwYDxzWz/UaXjYgDgG8A+wBbADXNtLE1sDEwupllSvFF4CQK+3IlsHVEbFn0/OeB67LHFwFbAUOy+vpTGFGWJKnFGXglSVo9pwDfSynNTCktBM4Fjlg28plS+n1KaX7Rc9tHRJ+i9W9PKT2ejaguyOZdllJ6M6X0HnAnhVDYlKaW/Rzwh5TSiymlD7JtN6Vv9u+s0na5Sddm21uSUpoL3A4cDZAF34HAHdmI8knAWSml91JK84ELgaPWcPuSJDXKwCtJ0urZFLg1IuZExBxgElAH9IuIqoi4KDvdeR4wI1tnvaL1X2+kzbeKHn8A9Gpm+00tu1GDthvbzjKzs383bGaZUjTcxnVkgZfC6O5tWfheH1gLGF903P6ezZckqcUZeCVJWj2vA59MKa1T9NM9pfQGhZB3CIXTivsAm2XrRNH6qUx1zaJw86llNm5m2SkU9uPwZpZ5n0JIXeajjSzTcF8eANaPiCEUgu+y05nfBT4Eti06Zn1SSs0Fe0mSVpuBV5KklesSEd2LfjpTuFb1gojYFCAi1o+IQ7LlewMLKYygrkXhtN3WchPw5YgYFBFrAT9oasGUUqJwve8PIuLLEbF2djOuXSPiqmyxCcDuEbFJdkr2qJUVkFJaTOHOzz8FPkIhAJNSWgr8DvhFRGwAEBH9I2L/1d1ZSZKaY+CVJGnl7qEwMrns51zgUuAO4P6ImA88CQzPlv8T8CrwBvBS9lyrSCndC1wGPARML9r2wiaWHw0cCRwPvAm8DfyYwnW4pJQeAG4EngfGA3eVWMp1FEa4b04pLSma/51ldWWne/+Dws2zJElqcVH4cFeSJOVRRAwCJgLdGgRPSZJyzxFeSZJyJiIOy77vdl3gJ8Cdhl1JUkdk4JUkKX9OBv4DvEzhztGnVrYcSZIqw1OaJUmSJEm55AivJEmSJCmXDLySJEmSpFwy8EpSC4uIrSNiQkTMj4gzK13PmoiIzSIiZd87q3YsImoiYmYLtZUiYouWaKs1NFdvRBwXEY+VefvLbaM1jl9EzIiIfUpYzve4pFwz8EpSy/s28FBKqXdK6bJKF1OsNf64b6sioldE1EbEvY08t0IAiYhzI+IvEXFMtl5tRHwYEUuLpmuLlj8uIl6IiA8i4q2I+E1ErFNCXetExO+zdeZHxNSIOKe52tqSiLg2IhYVH5OIeK7SdVVC1mdSRAxf+dL161ybrXNIg/m/yOYf1+KFtqCIODAiHouIOVkfvjoiele6LklaxsArSS1vU+DF1Vmxo4yytPR+ltje4cBCYN+I+GipbaeU/ppS6pVS6gV8Enhz2XQ2j4j4JoWv/zkb6APsTKEfPBARXVeyiV8AvYBB2boHA9NLra+NuLj4mKSUtq90Qa0tIgL4EvBe9u+qmFq8TtafP0fhLtttXR/gx8BGFPpwf+CnFa1IkooYeCWpBUXEP4E9gcuzka6tIqJPRPwpIt6JiFcj4vsR0Slb/riIeDwbzZkNnNtIm+dGxM3ZaOP8bBRxq4gYFRH/iYjXI2K/ouX7RMQ1ETErIt6IiB9HRFVEDAKuBEZktc3Jlj8wIp6NiHlZWyvUsLK2s+c+HhH/jIjZEfFuRPy1eIQzO8XyOxHxPPB+RGyRjWAdGxGvZet8r2j5ThFxTkS8nLV5U0R8JHtu2WmYJ0TEa8A/S3h5js32/3ngCyUsX5KIWBs4DzgjpfT3lNLilNIMCoFlsxK2tSNwXUrpvymlpSmlySml0Vnbj2TLPJe9ZkdGxLoRcVfWn/6bPR5QVM9HIuIPEfFm9vxtTdR9ZkS8FBEDovCdvT/LXoe3I+LKiOhRtOzZ2Wv+ZkQcvwrHZtnr1NRrvFNEjMv63tsR8fOi53aOiLHZyOFzEVFT9NyYrO+NzY7LnRHRN+tz8yLi6YjYrEE5n4qIV7IafrrsPdhIzQMj4oGIeC8ipkTE50rc3d2ADYEzgaNi5R90FLsT2DUK35sMcACFfvpWUV2dovC749Xsff+niOhT9PwXs+dmFx/jonUbfS81FIXfSa9E4XfNvyPimOYKTyldl/X7D1JK/wV+B+yyCvsuSWVl4JWkFpRS2gt4FDg9G+maCvyKwijIx4A9KIzkfLloteHAK0A/4IImmj4I+DOwLvAscB+F3+H9gfOB3xYtey2wBNgCGArsB5yYUpoEnAI8kdW2Trb8+1lN6wAHAqdGxKFN1NFo29lzAfw//jfSszErBvijs22sk7UDsCuwNbA38MMoBHOAM4BDKRyzjYD/Alc0aG+PbFv7N1FvobCITYEa4K/Zz6qOwDVnJNAd+FvxzJRSLXAPsO9K1n8SuCAivhwRWzZoY/fs4fbZa3Yjhdf9DxRGkDcBPgQuL1rtz8BawLbABhRGkJcTET8EjgP2SCnNBC4CtgKGUHht+wM/zJY9APhWth9bAiu9LrQRTb3GlwKXppTWBj4O3JRtsz9wN4WRw49k278lItYvavMo4ItZrR8HnsiOy0eAScCPGtRwGFAN7AAcAqwQ3COiJ/AAcB2FY3cU8OuI2KaEfTyWQnC9KZs+qIR1llkA3J5tDwr9808Nljku+9mTwu+SXmSve1bfbygcj42AvsCAonVLeS8t2//LgE+mlHpT6NsTVmE/AHZnNc9wkaSySCn5448//vjTgj/AGAoBE6AKWARsU/T8ycCY7PFxwGsrae9c4IGi6YOAWqAqm+4NJAohsh+F03Z7FC1/NIVripdt77GVbO+XwC+yx5tlbXdeWduNtHMo8GzR9Azg+KLpZW0PKJr3FHBU9ngSsHfRcxsCi7Nalq37sRJfk+8DE7LH/YE6YGjR8wnYopHj/pcG82qAmQ3mfQF4q4ntXlT82jWxTA/gu8D4bP+mUwgcTdbWYP0hwH+LjtFSYN1GlqsB3gB+DjwG9MnmB4UPPT5etOwI4N/Z498DFxU9t1VxTRQ+BFkAzCn6+WOJr/EjFEbH12tQ63eAPzeYdx9wbNF77HtFz10C3NvgPTKhwTE8oGj6q8CDDd8TwJHAow22+1vgRyt5DdcC5gGHFq1ze9Hz9dto+Jpmx+/HFD4UeILC+/jtrF88BhyXLfcg8NWiNrbmf++HHwI3FD3Xk8LvnX1W4b3UOVtvDoXT/3s0t89NHId9KYTprVZ1XX/88cefcv10iGvFJKmC1gO6AK8WzXuVQuha5vUS2nm76PGHwLsppbqiaSiM+GyUbW9WRCxbvlNz24jCDXYuArYDugLdgJsbWXTT5tqOiH4URux2oxDCO1H447dYY3W8VfT4g2w/lm3v1ohYWvR8HYXg3Vx7jfkShVMtSSm9EREPUxiRe7ao3S4N1ulCIRSszLvAehHROaW0pMFzG2bPNyml9CFwIXBhdnr0OcDNEbFJSum9hstHxFoURm0PoDDiD9A7CqeWbwy8lwqnljZmHeAk4MiU0txs3voUAtv4otc1KHxYA4U+Nb6ojeK+vMzPUkrfb2Y3m3qNT6BwhsLkiPg3cF5K6S4Kr/1nI6J4lLQL8FDRdMP3RMPpXiyvuK+8SmG/GtoUGB7Z6f6ZzhRGzZtzGIUzFu7Jpv8K/CMi1k8pvbOSdQFIKT2WjWB/D7grpfRh0etBVm/D3yPLPojaiKL9Sym9H4VLJIr3a2XvpWXrHUlhRP2aiHgc+GZKafLK6o+InSmMjB+RCme2SFKb4CnNklRe71IITZsWzduEwkjbMqkFt/c6hVHY9VJK62Q/a6eUtm1mW9cBdwAbp5T6ULjONRpZbmVtX5i1/4lUOEX1C420syr7+jqFkc51in66p5RW6dhFxEgKp+KOisJdZN+icBr55+N/N7t6jcJIV7HNaTzcNfQEhePymQbbXXaTqwdLaAOAlNI8CsexZ7b9xnyTwuje8Ow4LzvtOSgcs49E03eH/i/waeAPEbHsOst3KQTEbYuOc5+U3ZALmEUhSC+zSan7szIppWkppaMpnD78E2B0dlrt6xRGeItf+54ppYvWYHMN9+HNRpZ5HXi4wXZ7pZROXUnbx1II2K9l/etmCgH986tY418ovL4NT2cmq7fh75ElFIL+cq9R9qFI36JlS3kvAZBSui+ltC+FD2smk31Q1JyIGErhd8jxKaWS+7sktQYDrySVUTYKexOFazR7Z9eSfoPCH7bl2N4s4H7gkohYO7tZzccjYo9skbeBAQ1uqNObwqjggojYiSb+SC+h7d4UTrWem12DefYa7s6VFI7bpgARsX40+OqWEh1L4brMbSic/juEwmh2DwqBFOBG4PtRuIFTpyh8f+lBwOiVNZ6NlJ4H/CoiDoiILtkNk24CZrKS0cGI+EFE7BgRXSOiO/A1CqeVTskWeZvCNZvL9KYQUOdkNx6qv1Y1e43upXDd6bpZLbsXrUtKaQxwDPC3iNgppbSUQqj5RURskNXUPyKWXRd9E3BcRGyTBamG18autoj4QjYKujTbZyickv0X4KCI2D8KN1zrHoXvER7QZGMrd3Z2TDamcIxvbGSZu4CtshtAdcl+diy65rixfehP4drkT/O//rU9hQC/qteKX0bhtOBHGnnueuCsiNg8+zDlQuDG7KyC0cCnI2LX7L19Psv/jVfSeyki+kXEIdmHDgspvJ+XNlyuwTrbAX+ncNO2O1dxfyWp7Ay8klR+Z1C4RvIVCtfkXUfhushy+RKFU5NfojCiN5rCaA0U7mb8IvBWRCw71farwPkRMZ/CtYA30bTm2j6Pwg2B5lK44dDfGmtgFVxKYdTo/qy2JymMzJYsC5CfA36VUnqr6OffFILosdmi5wNjKbw+/wUuBo5JKU0sZTsppYspXIf7MwrXcv6Lwqja3imlhStbncLNlt6lMIq3L3BgKtz0CgrXEv8xCncr/hyFa6x7ZMs/SSFsFPsihbMKJgP/Ab7eSL0PULhp050RsQOFa2anA09GxDzgHxRGkUkp3Ztt85/ZMo3dEfvbsfz38DZ7GneRA4AXo/B9xpdSuLb3w5TS6xRuLPVd4B0Kx/Js1uzvltspnJo9gUL/vKbhAiml+RRuxHYUhdfiLQrBtVsz7X6RwvXC9xf3MQrhdXAWCEuSUnovpfRgSqmxMxd+T6HPPgL8m8J102dk670InEbhd8ssCn14ZtG6pb6XOlH4QO5NCl+vtAewstHtb1I4Lf6aotffm1ZJajOi8d+pkiRJkiS1b47wSpIkSZJyycArSWr3IuKYBqfUtqlTKyPi3ibq+26la1NpIuLFJl7DYypdW7lFxJVN7PuVla5NklbGU5olSZIkSbnkCK8kSZIkKZc6r3yR9m+99dZLm222WaXLaNL7779Pz549K12GZF9Um2A/VFtgP1RbYV9Upc344N8sravjY723qHQpzRo/fvy7KaX1G87vEIF3s802Y9y4cZUuo0ljxoyhpqam0mVI9kW1CfZDtQX2Q7UV9kVV2gnjjmHOnDncss/dlS6lWRHxamPzPaVZkiRJkpRLBl5JkiRJUi4ZeCVJkiRJudQhruFtzOLFi5k5cyYLFiyodCn06dOHSZMmVboMNdC9e3cGDBhAly5dKl2KJEmSpNXQYQPvzJkz6d27N5ttthkRUdFa5s+fT+/evStag5aXUmL27NnMnDmTzTffvNLlSJIkSVoNHfaU5gULFtC3b9+Kh121TRFB375928QZAJIkSZJWT4cNvIBhV82yf0iSJEntW4cOvJU0e/ZshgwZwpAhQ9hiiy3o379//fSiRYuaXXfcuHGceeaZK93GyJEjW6pcSZIkSWp3Ouw1vKvq2bGLuP+WBcyZnVinb7Df4d0ZOrLrarfXt29fJkyYAMCoUaPo27cv3/rWt+qfX7JkCZ07N/7yVFdXU11dvdJtjB07drXrkyRJkqT2zhHeEjw7dhG3Xvshc2YnAObMTtx67Yc8O7b5kdhVddxxx3HKKacwfPhwvv3tb/PUU08xYsQIhg4dysiRI5kyZQoAY8aM4dOf/jQA5557Lscffzw1NTV87GMf47LLLqtvr1evXvXL19TUcMQRRzBw4ECOOeYYUirsyz333MPAgQMZNmwYZ555Zn27kiRJktTeOcIL3PnXD5n1Wl2Tz7/2ch11S5aft3gR3PL7D3n64cZD74abVHHQMT1WuZaZM2cyduxYqqqqmDdvHo8++iidO3fmH//4B9/97ne55ZZbVlhn8uTJPPTQQ8yfP5+tt96aU089dYWv0nn22Wd58cUX2Wijjdhll114/PHHqa6u5uSTT+aRRx5h88035+ijj17leiVJkiSprTLwlqBh2F3Z/DXx2c9+lqqqKgDmzp3Lsccey7Rp04gIFi9e3Og6Bx54IN26daNbt25ssMEGvP322wwYMGC5ZXbaaaf6eUOGDGHGjBn06tWLj33sY/Vfu3P00Udz1VVXtfxOSZIkSVIFGHhhpSOxP/nmvPrTmYut0zc4aVSvFq2lZ8+e9Y9/8IMfsOeee3LrrbcyY8YMampqGl2nW7du9Y+rqqpYsmTFJF7KMpIkSZKUJ17DW4L9Du9Olwb3p+rStTC/nObOnUv//v0BuPbaa1u8/a233ppXXnmFGTNmAHDjjTe2+DYkSZIkqVIMvCUYOrIrhx3Xg3X6Fr6XdZ2+wWHH9VijuzSX4tvf/jajRo1i6NChZRmR7dGjB7/+9a854IADGDZsGL1796ZPnz4tvh1JkiRJqgRPaS7R0JFdyxZwv/vd79K7d+8V5o8YMYKpU6fWT//4xz8GoKampv705nPPPXe5dSZOnFj/uLa2doXlAS6//PL6x3vuuSeTJ08mpcRpp51W0tcdSZIkSVJ7UNYR3og4ICKmRMT0iDinkeePi4h3ImJC9nNig+fXjoiZEXF50bxhEfFC1uZlERHl3Ie8+93vfseQIUPYdtttmTt3LieffHKlS5IkSZKkFlG2Ed6IqAKuAPYFZgJPR8QdKaWXGix6Y0rp9Caa+T/gkQbzfgN8BfgXcA9wAHBvixXewZx11lmcddZZlS5DkiRJklpcOUd4dwKmp5ReSSktAm4ADil15YgYBvQD7i+atyGwdkrpyZRSAv4EHNqiVUuSJEmScqGc1/D2B14vmp4JDG9kucMjYndgKnBWSun1iOgEXAJ8AdinQZszG7TZv7GNR8RJwEkA/fr1Y8yYMcs936dPH+bPn78q+1M2dXV1zJ83b/VW9oTuslqwYAFjxjxU6TLKoPGOU1tbu8J7RWpt9kO1BfZDtRX2RVXanCVzqFta1277YaVvWnUncH1KaWFEnAz8EdgL+CpwT0pp5upeoptSugq4CqC6ujo1/A7bSZMmNXqjqEqYP28uvbsvxvRaaQ2/azno3rmOoVu8vcbttFQ9a95G8+2MmdSNmi3eaqSZBu2s9qXzbejYtNg+NaYl2ooWqilaoJygxfaphHbGvAg1W81pvp0Webla4hi37rEpqZk20//a4D6tQjtjnkvUbNvU79A1rCda6v0Nbar/eWyaaWb12xkz/kNqhvShRf9faDPvzRbsey32f2ZbOcYt2PfWcJ/+PLE7c+Z9SMM81V6UM/C+AWxcND0gm1cvpTS7aPJq4OLs8Qhgt4j4KtAL6BoRtcClWTtNttk+JaAKoqrShaihqILuH610Fa0naqH7upWuon1pGJxXv6E20kYLWt1jEx9C5x7FDbVEMS3QRgtqkX5Tzr6XKnfI2syxWQp177dAOy2lhV6QNvU7q6Xk/Ngs7QQf/Hs1V875sVkjbWyfWuwYt4QGtSyeC6n95pRyBt6ngS0jYnMKofQo4PPFC0TEhimlWdnkwcAkgJTSMUXLHAdUp5TOyabnRcTOFG5a9SXgV2Xch7KZPXs2e++9NwCzZs2ic+cq1l9vPQCeGvsQXbs2/RVI48Y/w5/+cj2X/eKnzW5j5O77MPaRf7RIvc8++xyX/+YqrrnqCgDu/fv9/ODcC/jggw/o1q0be+25Owd/+lOc870f8cSjD9avt2TJEvpvtjXPPvUYG220Yf38c8+/kPN+fBHTXnqWLbb4OAC/vOwKzvrWKJ5+YgzVw3Yoqa5r//RXxo1/hssvvWS1l3nhhRe55Je/4tprrixpm9Jy2tRIRBuz2rsU0KnSJyBJtdC5V6WLkAofRne1L6qCOnUF6ipdxWor218UKaUlEXE6cB9QBfw+pfRiRJwPjEsp3QGcGREHA0uA94DjSmj6q8C1QA8Kd2dul3do7tu3LxMmTABg1Khz6NunJ9/65tfrn1+yZAmdOzf+8lQP26GkQNhSYRfgwp9cwvdHnQ3AxIkvcfrXv8Xdt41m4MCtqKur46qr/8Buu45k5htv8uqrr7HpppsA8I8HH2LbbQYtF3aX+cR223LDTbfw/e9+G4Cbb7mNbbcZ1GI1l+oTn9iWmW+8yWuvvc4mm2y88hUkSZIktQtl/R7elNI9KaWtUkofTyldkM37YRZ2SSmNSiltm1LaPqW0Z0ppciNtXFv8tUUppXEppe2yNk/P7tacC8edcAqnnPZ1hu+yJ98e9QOeenocI3bbm6E77srI3fdhypRpAIx5+FE+fehngcJI6fFf+So1+3yKj209mMsu/019e73W3bB++Zp9PsURR36RgdsN45gvncCyw3bPvfcxcLthDBu+O2eedXZ9u8Xmz5/P8y9MZPvtPwHAxZf8ku+dczYDB24FQFVVFaeefCKdOnXic0ccxg033VK/7g033cLRRx7R6P4eevCB3H7n3QC8/PIr9OmzNuut17f++etvuJlPDN2Z7YYM5zujflg//w9//AtbbTOUnUbW8PjYJ+vnv/POuxz+uS+w44g92HHEHss9t8zNo29luyHD2X7YSHbf64D6+QcdeMBydUuSJElq/zxnDLhhDLz+Tsu2ufH6cFTNqq838403GPvIP6iqqmLevHk8+tB9dO7cmX88+BDf/cF53HLTX1ZYZ/KUqTz0wN3Mn1/L1tvtwKknn0iXLl2WW+bZCc/z4oR/sdFGG7LLHvvy+NgnqR42lJNP+zqPPHgvm2++GUd/4cuN1jRu/LNst+029dMTX5zEN79+RqPLHn3kEXzllDP5ztlnsXDhQu75+/38/KcXNrrs2muvzcYDBjBx4kvcfufdHHnEZ/jDn/4KwJtvzuI73/sR4598hHXXXYf9PnUot91+F8N3quZH51/I+CcfoU+ftdlz3wMZOmQwAF/7xrc562unsesuI3jttdfZ/8DDmPTCuOW2ef4FP+G+u2+lf/+NmDNnTv386mE7cNHFP+fb3/p6o7VKkiRJan8MvG3MZz9zKFVVhYvC586dx7HHn8K06S8TESxevLjRdQ785P5069aNbt26scH66/P22/9hwIDlv61ppx2H1c8bsv1gZsx4lV49e/KxzTdj8803A+DoIz/LVdf8YYX2Z816i/XXX6+k+quH7UDt+7VMmTKNSZOnMHzHaj7ykY80ufxRnzucG24azX0PPMiD991ZH3ifHvcMNbvvWr/dY47+HI889jjAcvOP/OxnmDptOgD/+OcYXpo0pb7tefPnU1tbu9z2dhm5M8edeCqfO+IwPnPoQfXzN1h/Pd6c1cjdiSVJkiS1WwZeVm8ktlx69uxZ//gH5/6YPWt249bR1zFjxqvU7Htgo+t069at/nFVVRVLlixpZJmuRct0Ykld6Ree9+jRgwULFtRPb7vNQMY/O6H+FOeGjv7cEdxw02gmTZ7S5OnMy3z6wAM4e9QPqN5hKGuvvXbJNTVm6dKlPPnYg3Tv3r3JZa684pf866mnufue+xi28x6Mf/Jh+vbty4IFC+nRo+n1JEmSJLU/Zb2GV2tm7tx59N9oI6Bwp+GWtvXWW/LKv2cwY8arANx4c+PXsA4auDXTX36lfvrsb3yNC3/yM6ZOLVxTvHTpUq686pr6548+8rP85fob+eeYRzjk4MZD+jJrrbUWP7ngPL436lvLzd9px2E8/OjjvPvubOrq6rj+xtHssduuDN+pmocffZzZs2ezePFibr7ltvp19ttnL351xW/rpydMeH6F7b388isM32lHzj/3+6y/Xl9ef73wrVZTp01f7rRtSZIkSe2fI7xt2Le/9TWOPf4Ufvz/fsqBn9yvxdvv0aMHv77s5xzw6c/Qs2dPdqxu/M7PAwduxdy585g/fz69e/dm8ODt+OXPLuLoL57ABx98QETw6U/97wZQgwZtTc+1ejJs6JDlRqybclQjo8AbbvhRLvrxuey574GklDjwk/vXh+dzfzCKEbvtwzrr9GHI9oPr17nsFz/ltDO/yeAdRrBkyRJ2320Xrrzil8u1e/Y5P2Da9JdJKbH3XnvUj1I/9PAjHPjJ/VdaqyRJkqT2I3J0k+MmVVdXp3Hjlr950aRJkxg0qPW/Aqcx8+fNoXd3IFr/C51ra2vp1asXKSVOO/MbbLnFxznra6evsNwvLr2c3r17c+Lxx7Z6jeW2cOFC9tj7kzw25v4Vvgpq0pTpDPro3ApV1vrGTKilZojf9afKsh+qLbAfqq2wL6rSTph2MXNq67hl3/sqXUqzImJ8Sqm64XxPae7gfnfNtQyp3oVtt9+JuXPncfJXjm90uVNPPpFuXbs2+lx799prr3PRBec1+b3HkiRJkton/8Lv4M762umNjug21L17d774haNboaLWt+WWW7DllltUugxJkiRJLcwRXkmSJElSLhl4JUmSJEm5ZOCVJEmSJOWSgVeSJEmSlEsG3gp66623OOqooxi8/VCG7VzDpw4+nKlTp1W6rOVc+6e/cvrXvgnAlVddw5/+fN0Ky8yY8SrbDRnebDszZrzKddffVD89bvwznHnW2S1brCRJkiQV8S7NFZJS4rDDDuPYY4/ld1ddSe/u8NzzL/H2f95hq622BGDJkiVt6qtyTjnphNVed8arr3HdjTfz+aM/B0D1sB2oHrZDS5UmSZIkSStwhLdCHnroIbp06cIpp5xSP2/77T9BXV0du+25PwcfdiTbDN6RBQsW8OUTT+UTQ3dm6I678tCYRwB48cVJ7DSyhiHVuzB4hxFMmzad999/nwMPOYLth41kuyHDufGmW5bb5tKlS9lsy+2YM2dO/bwttxnC22//hzvvupfhu+zJ0B13ZZ8DDubtt/+zQs3nnn8hP/v5ZQCMf+ZZth82ku2HjeSKK39Xv8yMGa+y2577s8NOu7HDTrsx9ol/AXDO937Eo489wZDqXfjFpZcz5uFH+fShnwXgvffe49DDj2bwDiPYede9eP75ifXbO/4rX6Vmn0/xsa0Hc9nlv2mBIy9JkiSpo2g7w4eV9MLXYe6Elm2zzxD4xC+bfHrixIkMGzas0eeeefY5Jj77JJtvvhmX/OJXRAQvPPskkydPZb8DD2Xqi89w5e+u4Wunn8oxnz+SRYsWUVdXxz333s9GG27I3bePBmDu3LnLtdupUycOOehT3Hr7XXz52C/wr6eeZtNNNqFfvw3YdZedefKxfxIRXP37P3LxJb/kkosvbLL+L5/4VS6/9GfsvtsunH3O9+vnb7DB+jxw7+10796dadOmc/QXT2Dckw9z0QXn8bNfXMZdt90MwJiHH61f50fnX8jQIYO57Zbr+edDD/Ol409mwrjHAZg8ZSoPPXA38+fXsvV2O3DqySfSpUuXZg+9JEmSJIEjvG3STjsOY/PNNwPgscef4AufPxKAgQO3YtNNNmbq1OmMGL4TF/7kEn7y01/w6quv0aNHDz6x3TY88OBDfGfUD3n0sbH06dNnhbaP/Ozh3HhzYeT3hptu4cjPfgaAmTPfZP8DD+UTQ3fmp5dcyosvTWqyvjlz5jBn7lx2320XAL74+aPqn1u8eDFfOeUMPjF0Zz579LG8NGnySvf3scef5IvHFNrYa889mP3ee8ybNw+AAz+5P926dWO99fqywfrrNzryLEmSJEmNcYQXmh2JLZdtt92W0aNHN/pcz55rrXT9zx/9OYbvVM3d997Hpw45gt9ecSl77bkHz/zrEe65936+/6P/Y+8992D//fbm5K9+HYDzf/Q9Dvr0J5n+8iu888673HbH3Xx/VOHGUWecdTbf+NrpHHzQpxjz8KOc+3//b7X26xeXXkG/fhvw3PixLF26lO6911+tdpbp1q1b/eOqqiqWLFmyRu1JkiRJ6jgc4a2Qvfbai4ULF3LVVVfVz3v++Yk8+tjY5ZbbbdeR/DW7u/HUqdN47fWZbL31lrzyyr/52Mc258zTT+WQgw7k+Rcm8uabs1hrrbX4wjFHcfY3zuSZZ59j+E47MmHc40wY9zgHH/QpIoLDDjmIb5w9ikEDt6Jv374AzJ07j/79NwTgj43cibnYOuuswzp9+vDY408A1NcHMHfePDb86Efp1KkTf/7rDdTV1QHQu3cv5s+vbbS93XYdUd/GmIcfZb2+fVl77bVLPpaSJEmS1BgDb4VEBLfeeiv/+Mc/GLz9ULYdMoJRPziXj36033LLffWUE1m6dCmfGLozRx7zZa69+jd069aNm0bfynZDhjOkehcmvvgSX/rC0bww8UV2GrknQ6p34bwfX8T3v9v41/4c+dnP8JfrbuTIIz5TP+/cH4zis0cfy7Dhu7Peen1XWv8frv41p535TYZU70JK6X/1nnwif/zzdWw/bCSTJ0+lZ8+eAAz+xHZUVVWx/bCR/OLSy5dr69wfjGL8MxMYvMMIzvnej/jjNVeWfBwlSZIkqSlRHFbyqrq6Oo0bN265eZMmTWLQoEEVqmh58+fNoXd3IKoqXYoamDRlOoM+OnflC+bEmAm11AzpVeky1MHZD9UW2A/VVtgXVWknTLuYObV13LLvfZUupVkRMT6lVN1wviO8kiRJkqRcMvBKkiRJknLJwCtJkiRJyiUDryRJkiQplwy8kiRJkqRcMvBKkiRJknLJwFtBF1xwAdtuuy0jRu7CkB13419PPd3kssedcAqjb7ltlbcxY8arXHf9Tau83upuT5IkSZLais6VLqCjeuKJJ7jrrrt45plnWLTwQxbWzmbR4roW386MV1/juhtv5vNHf67F25YkSZKktswR3gqZNWsW6623Ht26dQNgvfX6stFGGzL+mWfZY+9PMmz47ux/4KHMmvXWCus2tcz06S+zzwEHs/2wkeyw0268/PIrnPO9H/HoY08wpHoXfnHp5dTV1XH2Od9nxxF7MHiHEfz2d78HIKXE6V/7JltvuwP7HHAw/3nnndY7GJIkSZJUBgbeCtlvv/14/fXX2WqrrTjrG9/k4UceZ/HixZzx9bMZfcOfGf+vRzj+2C/yvR+ev9x6zS1zzLEnctopX+G58WMZ+8gDbLjhR7nogvPYbdcRTBj3OGd97XSu+cOf6LP22jz9xMM8/cQYfnfNH/n3v2dw6213MmXqdF56/mn+9PvfMvaJpypxWCRJkiSpxXhKc4X06tWL8ePH8+ijj/L3v9/LkV84nu+POpuJL05i308eAkBdXR0bbthvufWmTJnW6DLz58/njTdncdihBwHQvXv3Rrd7/wP/5PkXJjL6b7cDMHfePKZNf5lHHnuco488gqqqKjbaaEP2qtm9XLsuSZIkSa3CwFtBVVVV1NTUMGyHIVQP2YYrrryGbbcZyBOPPtjkOimlRpeZP39+SdtMKfGrX/6U/ffbZ7n59/z9/lXfAUmSJElqwzyluUKmTJnCtGnT6qcnPPcCgwZuxTvvvssTT/4LKJy+/OKLk5Zbb+utt2x0md69ezOg/0bcdvtdACxcuJAPPviA3r17MX9+bf36+++3N7/57TUsXrwYgKlTp/H++++z+667cOPNt1BXV8esWW/x0MOPlnX/JUmSJKncHOGtkNraWs444wzmzJlDp07BVlt8jKt+8ytOOuHLnPmNbzN37jyWLFnC18/8KttuO6h+va5duzL6+j83usyf/3AVJ5/2dX543gV06dKFm6//I4M/sR1VVVVsP2wkx33p83ztjK8yY8Zr7LDTbqSUWH/99bht9HUcduhB/HPMw2wzeEc22WRjRuy8YwWPjiRJkiStuUgpVbqGsquurk7jxo1bbt6kSZMYNGhQE2u0rvnz5tC7OxBVlS5FDUyaMp1BH51b6TJazZgJtdQM6VXpMtTB2Q/VFtgP1VbYF1VpJ0y7mDm1ddyy732VLqVZETE+pVTdcL6nNEuSJEmScslTmoGLp/yYKfMnrXzBVbB170F8e+vvt2ibkiRJkqTSOcIrSZIkScolR3ihYiOxI0eOZOzYsbz66ms8/+y/+PzRR7VY2xde9DO+e863/ret3fdh7CP/aLH2JUmSJKmtc4S3gsaOHQvAa6+9xnU3jF6ldZcsWdLs8xf+5JLlt2XYlSRJktTBGHgrqFevwh33fnTueTz6+BMMqd6FX1x6OXV1dZx9zvfZccQeDN5hBL/93e8BGPPwo+y25/4cfNiRbDO48LVBhx5+NMOG78622+/EVVf/AYBzvvsjPvzwQ4ZU78IxXzqhsK11NwQgpcTZ53yf7YYM5xNDd+bGm26pb7tmn09xxJFfZOB2wzjmSyew7A7e53z3R2wzeEcG7zCCb33ne613gCRJkiRpDXhKcxtw3rk/4tdX/Iq7bi+M8l519R/os/baPP3EwyxcuJBd9tiP/fbZC4Bnnn2Oic8+yeabbwbA7393BR/5yEf48MMP2XFEDYcfdjAXXXgel//mKiaMe3yFbf3t1juY8NwLPDd+LO++O5sdR9aw+267APDshOd5ccK/2GijDdllj315fOyTDBq4FbfefieTJ44nIpgzZ06rHBNJkiRJWlOO8LZB9z/wT/70l+sZUr0Lw3fZi9nvvce06S8DsNOOw+rDLsBll1/J9sNGsvOue/P6zDfql2vKY2Of4Ogjj6Cqqop+/TZgj9124elxz9S3PWBAfzp16sSQ7QczY8ar9OnTh+7du3PCSafxt1vvYK211irbfkuSJElSS3KEtw1KKfGrX/6U/ffbZ7n5Yx5+lJ4911pu+h//HMMTj/6DtdZai5p9PsWCBQtXe7vdunWtf1xV1YkldXV07tyZp8Y+xIP/HMPov93O5b+5in/ef9dqb0OSJEmSWosjvG1Ar169mF9bWz+9/35785vfXsPixYsBmDp1Gu+///4K682dO49111mHtdZai8mTp/Lkv56uf65Lly716xfbbZeR3HjzLdTV1fHOO+/yyGNj2WnHYU3WVltby9y58/jUJ/fnFz/7fzz3/AtrsquSJEmS1Goc4W0DtttuW6qqqth+2EiO+9Ln+doZX2XGjNfYYafdSCmx/vrrcdvo61ZY74D99+HK313DoE9Us/VWW7Lz8B3rnzvphOMYvMMIdhi6PX/90zX18w879CCe+NdTbD9sJBHBxReez0c/2o/JU6Y2Wtv8+bUccvhRLFiwkJQSP7/4wpY/AJIkSZJUBrHsTrx5Vl1dncaNG7fcvEmTJjFo0KAKVbS8+fPm0Ls7EFWVLkUNTJoynUEfnVvpMlrNmAm11AzpVeky1MHZD9UW2A/VVtgXVWknTLuYObV13LLvfZUupVkRMT6lVN1wvqc0S5IkSZJyycArSZIkScqlDh14O8Lp3Fp99g9JkiSpfeuwgbd79+7Mnj3bUKNGpZSY/d+5dO9SV+lSJEmSJK2mDnuX5gEDBjBz5kzeeeedSpfCggUf0r1LogN//tAmde9Sx4B1Vvw6KEmSJEntQ4cNvF26dGHzzTevdBkAjHnwboZuUwdd1q50KZIkSZKUGw4pSpIkSZJyycArSZIkScolA68kSZIkKZcMvJIkSZKkXDLwSpIkSZJyycArSZIkScolA68kSZIkKZcMvJIkSZKkXDLwSpIkSZJyycArSZIkScolA68kSZIkKZcMvJIkSZKkXDLwSpIkSZJyycArSZIkScolA68kSZIkKZcMvJIkSZKkXDLwSpIkSZJyycArSZIkScolA68kSZIkKZcMvJIkSZKkXDLwSpIkSZJyycArSZIkScolA68kSZIkKZcMvJIkSZKkXDLwSpIkSZJyycArSZIkScolA68kSZIkKZcMvJIkSZKkXDLwSpIkSZJyycArSZIkScolA68kSZIkKZcMvJIkSZKkXDLwSpIkSZJyycArSZIkScqlsgbeiDggIqZExPSIOKeR54+LiHciYkL2c2I2f9OIeCab92JEnFK0ztER8UJEPB8Rf4+I9cq5D5IkSZKk9qlzuRqOiCrgCmBfYCbwdETckVJ6qcGiN6aUTm8wbxYwIqW0MCJ6ARMj4g7gP8ClwDYppXcj4mLgdODccu2HJEmSJKl9KucI707A9JTSKymlRcANwCGlrJhSWpRSWphNduN/dUb20zMiAlgbeLNly5YkSZIk5UHZRniB/sDrRdMzgeGNLHd4ROwOTAXOSim9DhARGwN3A1sAZ6eU3szmnwq8ALwPTANOa2zjEXEScBJAv379GDNmTAvsUnnUfrCEMROBqK10Kergaj+sY8wE+6Eqy36otsB+qLbCvqhKm8NS6pbSpvNUc8oZeEtxJ3B9duryycAfgb0AsuA7OCI2Am6LiNHAe8CpwFDgFeBXwCjgxw0bTildBVwFUF1dnWpqasq/N6tpzIN3U7NNHXTpVelS1MGNmVBLzRD7oSrLfqi2wH6otsK+qEr787ROzKmtoy3nqeaU85TmN4CNi6YHZPPqpZRmF526fDUwrGEj2cjuRGA3YEg27+WUUgJuAka2eOWSJEmSpHavnIH3aWDLiNg8IroCRwF3FC8QERsWTR4MTMrmD4iIHtnjdYFdgSkUAvM2EbF+ts6+y9aRJEmSJKlY2U5pTiktiYjTgfuAKuD3KaUXI+J8YFxK6Q7gzIg4GFhC4XTl47LVBwGXRESicJOqn6WUXgCIiPOARyJiMfBq0TqSJEmSJNUr6zW8KaV7gHsazPth0eNRFK7BbbjeA8DgJtq8EriyZSuVJEmSJOVNOU9pliRJkiSpYgy8kiRJkqRcMvBKkiRJknLJwCtJkiRJyiUDryRJkiQplwy8kiRJkqRcMvBKkiRJknLJwCtJkiRJyiUDryRJkiQplwy8kiRJkqRcMvBKkiRJknLJwCtJkiRJyiUDryRJkiQplwy8kiRJkqRcMvBKkiRJknLJwCtJkiRJyiUDryRJkiQplwy8kiRJkqRcMvBKkiRJknLJwCtJkiRJyiUDryRJkiQplwy8kiRJkqRcMvBKkiRJknLJwCtJkiRJyiUDryRJkiQplwy8kiRJkqRcMvBKkiRJknLJwCtJkiRJyiUDryRJkiQplwy8kiRJkqRcMvBKkiRJknLJwCtJkiRJyiUDryRJkiQplwy8kiRJkqRcMvBKkiRJknLJwCtJkiRJyiUDryRJkiQplwy8kiRJkqRcMvBKkiRJknLJwCtJkiRJyiUDryRJkiQplwy8kiRJkqRcMvBKkiRJknLJwCtJkiRJyiUDryRJkiQplwy8kiRJkqRcMvBKkiRJknLJwCtJkiRJyiUDryRJkiQplwy8kiRJkqRcMvBKkiRJknLJwCtJkiRJyiUDryRJkiQplwy8kiRJkqRcMvBKkiRJknLJwCtJkiRJyiUDryRJkiQplwy8kiRJkqRcMvBKkiRJknLJwCtJkiRJyiUDryRJkiQplwy8kiRJkqRcMvBKkiRJknLJwCtJkiRJyiUDryRJkiQplwy8kiRJkqRcMvBKkiRJknLJwCtJkiRJyiUDryRJkiQplwy8kiRJkqRcMvBKkiRJknLJwCtJkiRJyiUDryRJkiQplzpXugABCRbVAVHpQtTRJWDRkkpXoY7Ofqi2wH6otsK+qEpLqdIVrBkDbxuwuA5emQWpqtKVqKNbuAhefrPSVaijsx+qLbAfqq2wL6rSPlzUvkOvgbeN6FoFnbtXugp1dJ06QU/7oSrMfqi2wH6otsK+qErrNL/SFawZr+GVJEmSJOWSgVeSJEmSlEsGXkmSJElSLhl4JUmSJEm5ZOCVJEmSJOWSgVeSJEmSlEsGXkmSJElSLhl4JUmSJEm5ZOCVJEmSJOWSgVeSJEmSlEsGXkmSJElSLpU18EbEARExJSKmR8Q5jTx/XES8ExETsp8Ts/mbRsQz2bwXI+KUonW6RsRVETE1IiZHxOHl3AdJkiRJUvvUuVwNR0QVcAWwLzATeDoi7kgpvdRg0RtTSqc3mDcLGJFSWhgRvYCJ2bpvAt8D/pNS2ioiOgEfKdc+SJIkSZLar7IFXmAnYHpK6RWAiLgBOARoGHhXkFJaVDTZjeVHoo8HBmbLLQXebamCJUmSJEn5Uc5TmvsDrxdNz8zmNXR4RDwfEaMjYuNlMyNi44h4PmvjJymlNyNinezp/8tOeb45IvqVawckSZIkSe1XOUd4S3EncH126vLJwB+BvQBSSq8DgyNiI+C2iBgN1AEDgLEppW9ExDeAnwFfbNhwRJwEnATQr18/xowZ0xr7s1oWLFzC9LeATrWVLkUd3IJFdUx+zX6oyrIfqi2wH6qtsC+q0j7stJSliTadp5pTzsD7BrBx0fSAbF69lNLsosmrgYsbNpKN7E4EdgNuAT4A/pY9fTNwQmMbTyldBVwFUF1dnWpqalZrJ1rDA/fdzcfXr6Nz916VLkUd3OTXahm4if1QlWU/VFtgP1RbYV9UpfV4pxMfLqyjLeep5pTzlOangS0jYvOI6AocBdxRvEBEbFg0eTAwKZs/ICJ6ZI/XBXYFpqSUEoVR4Zpsnb0p4ZpgSZIkSVLHU7YR3pTSkog4HbgPqAJ+n1J6MSLOB8allO4AzoyIg4ElwHvAcdnqg4BLIiIBAfwspfRC9tx3gD9HxC+Bd4Avl2sfJEmSJEntV1mv4U0p3QPc02DeD4sejwJGNbLeA8DgJtp8Fdi9ZSuVJEmSJOVNOU9pliRJkiSpYgy8kiRJkqRcMvBKkiRJknLJwCtJkiRJyiUDryRJkiQpl0oKvBHRLyKuiYh7s+ltIuKE8pYmSZIkSdLqK3WE91oK36e7UTY9Ffh6GeqRJEmSJKlFlBp410sp3QQsBUgpLQHqylaVJEmSJElrqNTA+35E9AUSQETsDMwtW1WSJEmSJK2hziUu9w3gDuDjEfE4sD5wRNmqkiRJkiRpDZUUeFNKz0TEHsDWQABTUkqLy1qZJEmSJElroNS7NJ8G9EopvZhSmgj0ioivlrc0SZIkSZJWX6nX8H4lpTRn2URK6b/AV8pSkSRJkiRJLaDUwFsVEbFsIiKqgK7lKUmSJEmSpDVX6k2r/g7cGBG/zaZPzuZJkiRJktQmlRp4v0Mh5J6aTT8AXF2WiiRJkiRJagGl3qV5KfCb7EeSJEmSpDavpMAbEbsA5wKbZusEkFJKHytfaZIkSZIkrb5ST2m+BjgLGA/Ula8cSZIkSZJaRqmBd25K6d6yViJJkiRJUgsqNfA+FBE/Bf4GLFw2M6X0TFmqkiRJkiRpDZUaeIdn/1YXzUvAXi1bjiRJkiRJLaPUuzTvWe5COqwFb9N76ct0XbSUKnpCRDMLRwmPl5eKn4vS1lmt7TTZ9hpuZ4XV/zcjlbSdhvObOh5Nr7Na2ynpWDddW2pqnRVqLtcxkCRJktq/Ukd4iYgDgW2B7svmpZTOL0dRHcprf2DnRaPg7UoXIsEmANNKDfkNnystSDfddonbWY0PbkrenzX84KbJ7TTzwc2afkCUWvhDsqaOdWqy3VLbLv1DoXUWLaX7a51W8rqV8qFQaeuUvJ1oYn6ptTX3oVZJH8Ct6WvYwn2liWOdVlywxdou+QPekt6LzffjbRYvZp23uzSzjYZtr8aHjiV+uLnGH/CWeAxWfTtr2o+b65fNHdsW3M5q/M5o7cGAAXUL6TWne+OL0AL/r5b8Qfhq/L/aGv9/t/BgwPKDRaWts1rbKdvvjOb2rdR2lz8GnZZ+QKdmX7O2rdSvJboSWAvYE7gaOAJ4qox1dRwbfYYJ09/no32WUtWlR4MnU+OP04p/Tqy4PERT6zezzvJtt+B2Vqh5NWorml5um6nxZZpbv9RaoqRj3dQyzSzXZLvNHc8St1PSa9j4dt6du4j1+nRpfIVm2y6ttib3bYXVW3A7zRzr1dpOE8cgSupfzWynyX7cAtsp8XdGU7U1/LNx+cnVeY80vs6yfVu4eAldqqpWunzz2yntNSzt9+TKtrNqv4+isT+904rLlfo7o7ntlLROib8zSjnWK+7Xmv2ub7KvrPH/Kc29Xwr/rLd0KZ3f/98erXg8V2M7JbxfVms7zR6P0mprsr+06O99rY6PALxT6SrUkXXt3h9Yu9JlrLZSR3hHppQGR8TzKaXzIuISwLs2t4ReW/FO1c6s3aOOzt3bb0dSPrz8fi1d+vaqdBnq4Ca/VsvA/vZDVdbk12oZuIn9sKzWMEy3tQ+SS/qAd7nFS/vgZvob77NF/56NtlXaB8lr+gHR8tOlDQas5naa/ECy1HZX9YOb5aeb/oB35R+Srco6JW9nlT9UbuZ9sAbvt8W1d7Fgcc5HeIEPs38/iIiNgNnAhuUpSZIkSblX8imupbTV9FOVGGNuyW0uiq4s7eyHL6qcug8epo66Spex2koNvHdFxDrAT4FnKLyPry5XUZIkSZIkralS79L8f9nDWyLiLqB7Smlu+cqSJEmSJGnNNBt4I2KvlNI/I+IzjTxHSulv5Sst/54du4j7b1nAnNm78FTvhQzf4y222ua/lS5LkiRJknJhZSO8ewD/BA5q5LkEGHhX07NjF3HrtR+yeBFAUDu/Ow//fWMAQ68kSZKkipv60rr8Z0kv6uqCn3xzHvsd3p2hI7tWuqxV0mzgTSn9KHt4Ykqp/V6p3Abdf8uCLOz+z5IlVTz+YH+6dSvToW6lm6u16lcQtNo+tZIK7897/+nMzKUNvx5rTTeWr6+kyFtfaNVNldgX5vynE29GC/fDMmj26yrbqdbbp1b6vbAG+zP33eCtzqX1w1btCnn7f6+V+kJ7fr/Ofy/xTtdG+mLe+oJ/p7Y5r768NuMe35C6vQo7Mmd24tZrC/cybk+ht9SbVv07Iv4O3Aj8M6VmvxBQJZgzu/FDuODDLtxzy8dbuRrpf56rdAES8GylC5Ao3KVTagvGVboAKbN4UWHgLo+BdyDwaeA04JrsxlU3pJQeK1tlObdO32g09KYukAZWVaCijimlNvTRWxv4GGlJXaJzVesckzawu6uutQalWmE7qQ1/7LykbimdqzpVuoyyKvklbqtvlLZaVwlKLX3p0kSnTit/n7Tdd9IaKOPr2467TkEFdqBuaaKqhL5Yijb7u7/dd4wStMN97PryokZ7TFMDd21VqXdp/gC4CbgpItYFLgUeBkxmq2m/w7sXXcNb0LlzHXvs/7rX8KpiJr9Wy8BN/K4/VZb9UG2B/VBthX1RlfLnK7eldt6KI7nr9G2jH5w0oeSP0CNij4j4NTAe6A58rmxVdQBDR3blsON6ZB0m0av3AvY4wLArSZIkqfKG7/4mnTsvf2+hLl0LA3ftSUkjvBExg8IlVTcBZ6eU3i9nUR3F0JFdGTqyKw/cdzcfX7+Ozt3XrnRJkiRJklQ/EPfkkkRdXbBO38jfXZqLDE4pzStrJZIkSZKkNmOrbf7LBu/U8uHCOr5zSfscnCv1lOaPRsSDETERICIGR8T3y1iXJEmSJElrpNTA+ztgFLAYIKX0PHBUuYqSJEmSJGlNlRp410opPdVg3pKWLkaSJEmSpJZSauB9NyI+TvYNUhFxBDCrbFVJkiRJkrSGSr1p1WnAVcDAiHgD+DdwTNmqkiRJkiRpDTUbeCPiG0WT9wAPURgVfh84HPh5+UqTJEmSJGn1rWyEt3f279bAjsDtQABfBBpe0ytJkiRJUpvRbOBNKZ0HEBGPADuklOZn0+cCd5e9OkmSJEmSVlOpN63qBywqml6UzZMkSZIkqU0q9aZVfwKeiohbs+lDgWvLUZAkSZIkSS2hpMCbUrogIu4FdstmfTml9Gz5ypIkSZIkac2UOsJLSukZ4Jky1iJJkiRJUosp9RpeSZIkSZLaFQOvJEmSJCmXDLySJEmSpFwy8EqSJEmScsnAK0mSJEnKJQOvJEmSJCmXDLySJEmSpFwy8EqSJEmScsnAK0mSJEnKJQOvJEmSJCmXDLySJEmSpFwy8EqSJEmScsnAK0mSJEnKJQOvJEmSJCmXDLySJEmSpFwy8EqSJEmScsnAK0mSJEnKJQOvJEmSJCmXDLySJEmSpFwy8EqSJEmScsnAK0mSJEnKJQOvJEmSJCmXDLySJEmSpFwy8EqSJEmScsnAK0mSJEnKJQOvJEmSJCmXDLySJEmSpFwy8EqSJEmScsnAK0mSJEnKJQOvJEmSJCmXDLySJEmSpFwy8EqSJEmScsnAK0mSJEnKJQOvJEmSJCmXDLySJEmSpFwy8EqSJEmScsnAK0mSJEnKJQOvJEmSJCmXOpez8Yg4ALgUqAKuTild1OD544CfAm9ksy5PKV0dEZsCt1II5F2AX6WUrmyw7h3Ax1JK25VzH1rLoiWwcEGlq2i7UpnbjzK3314sXQrv2w/blI7Y9+2Hgsr3/Y7cD8t97LVqli6F2hbqi23xd35bYt9v3NJ2fmDKFngjogq4AtgXmAk8HRF3pJRearDojSml0xvMmwWMSCktjIhewMRs3Teztj8D1Jar9tbWuQo2Xp8yf/zQfrXz91i7Mmt21helCmoL/dDfO5WV2sALMOtd6L9epatQR9NY35/1LvTv2/q1SMt0+wAWLa50FauvnBFrJ2B6SukVgIi4ATgEaBh4V5BSWlQ02Y2iU6+zAPwN4CTgppYsuFIioGd3CmPZUgV1CujVo9JVqKOzH6ot6NQJ1l6r0lVIUNUJ+vSsdBXqyKqqKl3Bmiln4O0PvF40PRMY3shyh0fE7sBU4KyU0usAEbExcDewBXD2stFd4P+AS4APmtt4RJxEIRTTr18/xowZs/p7Uma1HyxhzEQgcjNorXaq9sM6xkywH6qy7IdqC+yHaivsi6q0OSylbiltOk81p9In0d4JXJ+dunwy8EdgL4As+A6OiI2A2yJiNLAh8PGU0lkRsVlzDaeUrgKuAqiurk41NTXl24s1NObBu6nZpg669Kp0KergxkyopWaI/VCVZT9UW2A/VFthX1Sl/XlaJ+bU1tGW81RzynmX5jeAjYumB/C/m1MBkFKanVJamE1eDQxr2Eg2sjsR2A0YAVRHxAzgMWCriBjT4pVLkiRJktq9cgbep4EtI2LziOgKHAXcUbxARGxYNHkwMCmbPyAiemSP1wV2BaaklH6TUtoopbRZNm9qSqmmjPsgSZIkSWqnynZKc0ppSUScDtxH4WuJfp9SejEizgfGpZTuAM6MiIOBJcB7wHHZ6oOASyIiUbiD+s9SSi+Uq1ZJkiRJUv6U9RrelNI9wD0N5v2w6PEoYFQj6z0ADF5J2zOAXHwHryRJkiSp5ZXzlGZJkiRJkirGwCtJkiRJyiUDryRJkiQplwy8kiRJkqRcMvBKkiRJknLJwCtJkiRJyiUDryRJkiQplwy8kiRJkqRcMvBKkiRJknLJwCtJkiRJyiUDryRJkiQplwy8kiRJkqRcMvBKkiRJknLJwCtJkiRJyiUDryRJkiQplwy8kiRJkqRcMvBKkiRJknLJwCtJkiRJyiUDryRJkiQplwy8kiRJkqRcMvBKkiRJknLJwCtJkiRJyiUDryRJkiQplwy8kiRJkqRcMvBKkiRJknLJwCtJkiRJyiUDryRJkiQplwy8kiRJkqRcMvBKkiRJknLJwCtJkiRJyiUDryRJkiQplwy8kiRJkqRcMvBKkiRJknLJwCtJkiRJyiUDryRJkiQplwy8kiRJkqRcMvBKkiRJknLJwCtJkiRJyiUDryRJkiQplwy8kiRJkqRcMvBKkiRJknLJwCtJkiRJyiUDryRJkiQplwy8kiRJkqRcMvBKkiRJknLJwCtJkiRJyiUDryRJkiQplwy8kiRJkqRcMvBKkiRJknLJwCtJkiRJyiUDryRJkiQplwy8kiRJkqRcMvBKkiRJknLJwCtJkiRJyiUDryRJkiQplwy8kiRJkqRcMvBKkiRJknLJwCtJkiRJyiUDryRJkiQplwy8kiRJkqRcMvBKkiRJknLJwCtJkiRJyiUDryRJkiQplwy8kiRJkqRcMvBKkiRJknLJwCtJkiRJyiUDryRJkiQplwy8kiRJkqRcMvBKkiRJknLJwCtJkiRJyiUDryRJkiQplwy8kiRJkqRcMvBKkiRJknLJwCtJkiRJyiUDryRJkiQplwy8kiRJkqRcMvBKkiRJknLJwCtJkiRJyiUDryRJkiQplwy8kiRJkqRcMvBKkiRJknLJwCtJkiRJyiUDryRJkiQplwy8kiRJkqRcMvBKkiRJknLJwCtJkiRJyiUDryRJkiQpl8oaeCPigIiYEhHTI+KcRp4/LiLeiYgJ2c+J2fxNI+KZbN6LEXFKNn+tiLg7IiZn8y8qZ/2SJEmSpParc7kajogq4ApgX2Am8HRE3JFSeqnBojemlE5vMG8WMCKltDAiegETI+IOYA7ws5TSQxHRFXgwIj6ZUrq3XPshSZIkSWqfyjnCuxMwPaX0SkppEXADcEgpK6aUFqWUFmaT3cjqTCl9kFJ6aNkywDPAgBavXJIkSZLU7pUz8PYHXi+anpnNa+jwiHg+IkZHxMbLZkbExhHxfNbGT1JKbxavFBHrAAcBD7Z45ZIkSZKkdq9spzSX6E7g+uzU5ZOBPwJ7AaSUXgcGR8RGwG0RMTql9DZARHQGrgcuSym90ljDEXEScBJAv379GDNmTNl3ZnXVfrCEMROBqK10Kergaj+sY8wE+6Eqy36otsB+qLbCvqhKm8NS6pbSpvNUc8oZeN8ANi6aHpDNq5dSml00eTVwccNGUkpvRsREYDdgdDb7KmBaSumXTW08pXRVthzV1dWppqZm1feglYx58G5qtqmDLr0qXYo6uDETaqkZYj9UZdkP1RbYD9VW2BdVaX+e1ok5tXW05TzVnHKe0vw0sGVEbJ7dYOoo4I7iBSJiw6LJg4FJ2fwBEdEje7wusCswJZv+MdAH+HoZa5ckSZIktXNlG+FNKS2JiNOB+4Aq4PcppRcj4nxgXErpDuDMiDgYWAK8BxyXrT4IuCQiEhAU7sz8QkQMAL4HTAaeiQiAy1NKV5drPyRJkiRJ7VNZr+FNKd0D3NNg3g+LHo8CRjWy3gPA4Ebmz6QQgCVJkiRJalY5T2mWJEmSJKliDLySJEmSpFwy8EqSJEmScsnAK0mSJEnKJQOvJEmSJCmXDLySJEmSpFwy8EqSJEmScsnAK0mSJEnKJQOvJEmSJCmXDLySJEmSpFwy8EqSJEmScsnAK0mSJEnKJQOvJEmSJCmXDLySJEmSpFwy8EqSJEmScsnAK0mSJEnKJQOvJEmSJCmXDLySJEmSpFwy8EqSJEmScsnAK0mSJEnKJQOvJEmSJCmXDLySJEmSpFwy8EqSJEmScsnAK0mSJEnKJQOvJEmSJCmXDLySJEmSpFwy8EqSJEmScsnAK0mSJEnKJQOvJEmSJCmXDLySJEmSpFwy8EqSJEmScsnAK0mSJEnKJQOvJEmSJCmXDLySJEmSpFwy8EqSJEmScsnAK0mSJEnKJQOvJEmSJCmXDLySJEmSpFwy8EqSJEmScsnAK0mSJEnKJQOvJEmSJCmXDLySJEmSpFwy8EqSJEmScsnAK0mSJEnKJQOvJEmSJCmXDLySJEmSpFwy8EqSJEmScsnAK0mSJEnKJQOvJEmSJCmXDLySJEmSpFwy8EqSJEmScsnAK0mSJEnKpc6VLkCZxXOh7oOWbTMVPY7WbHBVNtzSbZa4XCpaLlp6X1rkYC+v2RqXW3DNlkudYeE7Jbaxmttu8X0p1Sq019ZrLLm+VWizxdtbk31ZCnUfllrQ6m+7LMe7VG29xjL0sbIcR0mSmmfgbQuqekC/A0pYMK18kfpFS122hZcrebtl2HaL11iGfUmpxO2nEtsscblU4nJVM6D3pi27bVKJh6eF96V+2y19vEtZrNTtruq2K1HjKh7vFukTSyC6lN5eyX2i4XabW6eozWabbuEa00o3uGrbXZVlS932qhzvUpX8IWRxuy20XFPbTt1gwVuNtNeC227x5UrU4h+IFytxX1b2OqdSi2wPr8vq7Evx7IZ9sQzK8pl9pY93C76vUon9tlLv6WXK8oF4wNLFtOcTgw28bUIn6LZ+pYuQoNNbsPagSlehjq7zGNigptJVqLVU8oPS5rY9/THYaNdV3G4LbXt12msPH0xXap9XZdm2+LpMmwDr71aZba9OexXddhn2pZJ9olIfTDf8YLNzL4iFJW6j7THwSpKkyqnkKfora66TfyapDYjO0G29SlehjqxqLWBRpatYbe13bFqSJEmSpGYYeCVJkiRJuWTglSRJkiTlkoFXkiRJkpRLBl5JkiRJUi4ZeCVJkiRJuWTglSRJkiTlkoFXkiRJkpRLBl5JkiRJUi4ZeCVJkiRJuWTglSRJkiTlkoFXkiRJkpRLBl5JkiRJUi4ZeCVJkiRJuWTglSRJkiTlkoFXkiRJkpRLBl5JkiRJUi4ZeCVJkiRJuWTglSRJkiTlkoFXkiRJkpRLkVKqdA1lFxHvAK9Wuo5mrAe8W+kiJOyLahvsh2oL7IdqK+yLagvaQz/cNKW0fsOZHSLwtnURMS6lVF3pOiT7otoC+6HaAvuh2gr7otqC9twPPaVZkiRJkpRLBl5JkiRJUi4ZeNuGqypdgJSxL6otsB+qLbAfqq2wL6otaLf90Gt4JUmSJEm55AivJEmSJCmXDLytKCJmRMQLETEhIsY18nxExGURMT0ino+IHSpRp/KthH54TNb/XoiIsRGxfSXqVP6trC8WLbdjRCyJiCNasz51DKX0w4ioyZ5/MSIebu0alX8l/N/cJyLujIjnsn745UrUqfyLiHUiYnRETI6ISRExosHz7S6vdK50AR3Qnimlpr7D6pPAltnPcOA32b9SS2uuH/4b2COl9N+I+CSFazbshyqX5voiEVEF/AS4v/VKUgfUZD+MiHWAXwMHpJRei4gNWrUydSTN/T48DXgppXRQRKwPTImIv6aUFrVifeoYLgX+nlI6IiK6Ams1eL7d5RVHeNuWQ4A/pYIngXUiYsNKF6WOJaU0NqX032zySWBAJetRh3cGcAvwn0oXog7r88DfUkqvAaSU7IuqhAT0jogAegHvAUsqW5LyJiL6ALsD1wCklBallOY0WKzd5RUDb+tKwP0RMT4iTmrk+f7A60XTM7N5UktaWT8sdgJwbyvUpI6p2b4YEf2Bwyh8eiyVy8p+J24FrBsRY7JlvtTK9aljWFk/vBwYBLwJvAB8LaW0tDULVIewOfAO8IeIeDYiro6Ing2WaXd5xVOaW9euKaU3stOhHoiIySmlRypdlDqckvphROxJIfDu2uoVqqNYWV/8JfCdlNLSwqCGVBYr64edgWHA3kAP4ImIeDKlNLUSxSq3VtYP9wcmAHsBH8+WeTSlNK8CtSq/OgM7AGeklP4VEZcC5wA/qGxZa8YR3laUUnoj+/c/wK3ATg0WeQPYuGh6QDZPajEl9EMiYjBwNXBISml261aojqKEvlgN3BARM4AjgF9HxKGtWaPyr4R+OBO4L6X0fnZ95SOAN/NTiyqhH36Zwqn1KaU0ncL9Nga2bpXqAGYCM1NK/8qmR1MIwMXaXV4x8LaSiOgZEb2XPQb2AyY2WOwO4EvZ3c92BuamlGa1cqnKsVL6YURsAvwN+KIjGCqXUvpiSmnzlNJmKaXNKPyn+9WU0m2tXavyq8T/m28Hdo2IzhGxFoWbs0xq3UqVZyX2w9conGVARPQDtgZeac06lX8ppbeA1yNi62zW3sBLDRZrd3nFU5pbTz/g1uy0vM7AdSmlv0fEKQAppSuBe4BPAdOBDyh8mie1pFL64Q+BvhRG0wCWpJSqK1Sv8quUviiV20r7YUppUkT8HXgeWApcnVJqGEakNVHK78P/A66NiBeAoHC5R5N3uJfWwBnAX7M7NL8CfLm955VIKVW6BkmSJEmSWpynNEuSJEmScsnAK0mSJEnKJQOvJEmSJCmXDLySJEmSpFwy8EqSJEmScsnAK0lSBUTE2OzfzSLi8y3c9ncb25YkSR2NX0skSVIFRUQN8K2U0qdXYZ3OKaUlzTxfm1Lq1QLlSZLUrjnCK0lSBUREbfbwImC3iJgQEWdFRFVE/DQino6I5yPi5Gz5moh4NCLuAF7K5t0WEeMj4sWIOCmbdxHQI2vvr8XbioKfRsTEiHghIo4santMRIyOiMkR8deIiGXtRcRLWS0/a81jJEnSmupc6QIkSergzqFohDcLrnNTSjtGRDfg8Yi4P1t2B2C7lNK/s+njU0rvRUQP4OmIuCWldE5EnJ5SGtLItj4DDAG2B9bL1nkke24osC3wJvA4sEtETAIOAwamlFJErNOyuy5JUnk5witJUtuyH/CliJgA/AvoC2yZPfdUUdgFODMingOeBDYuWq4puwLXp5TqUkpvAw8DOxa1PTOltBSYAGwGzAUWANdExGeAD9Zw3yRJalUGXkmS2pYAzkgpDcl+Nk8pLRvhfb9+ocK1v/sAI1JK2wPPAt3XYLsLix7XAcuuE94JGA18Gvj7GrQvSVKrM/BKklRZ84HeRdP3AadGRBeAiNgqIno2sl4f4L8ppQ8iYiCwc9Fzi5et38CjwJHZdcLrA7sDTzVVWET0AvqklO4BzqJwKrQkSe2G1/BKklRZzwN12anJ1wKXUjid+JnsxlHvAIc2st7fgVOy62ynUDiteZmrgOcj4pmU0jFF828FRgDPAQn4dkrprSwwN6Y3cHtEdKcw8vyN1dpDSZIqxK8lkiRJkiTlkqc0S5IkSZJyycArSZIkScolA68kSZIkKZcMvJIkSZKkXDLwSpIkSZJyycArSZIkScolA68kSZIkKZcMvJIkSZKkXPr/1JvAFdbQFWYAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 1152x648 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Display learning curve\n",
    "learning_curve_plot = aml.leader.learning_curve_plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___\n",
    "<a name=\"predict\"></a>\n",
    "## (3) Predict with H2O AutoML Best Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prepare test data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Parse progress: |████████████████████████████████████████████████████████████████| (done) 100%\n"
     ]
    }
   ],
   "source": [
    "# Import test data\n",
    "test_frame = h2o.import_file(path='data/processed/test.csv')\n",
    "\n",
    "# Drop ID column for test set\n",
    "X_test_frame = test_frame.drop('Response')\n",
    "y_test_frame = test_frame[:, 'Response']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load leader model from MLflow saved in artifacts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('48e97c2acef0496a826191104ff94dc3', '1')"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get dataframe of all runs\n",
    "all_experiments = [exp.experiment_id for exp in client.list_experiments()]\n",
    "runs = mlflow.search_runs(experiment_ids=all_experiments, run_view_type=ViewType.ALL)\n",
    "\n",
    "# Identify best model (experiment id and run id) amongst all runs in the experiment\n",
    "run_id, exp_id = runs.loc[runs['metrics.log_loss'].idxmin()]['run_id'], runs.loc[runs['metrics.log_loss'].idxmin()]['experiment_id']\n",
    "run_id, exp_id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "stackedensemble prediction progress: |███████████████████████████████████████████| (done) 100%\n"
     ]
    }
   ],
   "source": [
    "# Load best model (AutoML leader)\n",
    "best_model = mlflow.h2o.load_model(f\"mlruns/{exp_id}/{run_id}/artifacts/model/\")\n",
    "\n",
    "# Generate predictions with best model (output is H2O frame)\n",
    "preds_frame = best_model.predict(X_test_frame)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get y values (ground truth and predicted)\n",
    "y_pred = preds_frame.as_data_frame()['predict']\n",
    "y_true = y_test_frame.as_data_frame()['Response']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get Performance Metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.45247813411078724"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import f1_score, accuracy_score\n",
    "f1_score(y_true, y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7946822932610311"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "accuracy_score(y_true, y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___\n",
    "<a name=\"explain\"></a>\n",
    "## (4) H2O Model Explainability\n",
    "- More info: https://docs.h2o.ai/h2o/latest-stable/h2o-docs/explain.html#output-explanations"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "bdd26ecccbd0f3e40fb5cf5d30fdca40f940aba434b243f1db962da2809839ff"
  },
  "kernelspec": {
   "display_name": "Python 3.10.1 64-bit ('e2e-insurance-cross-sell-prediction-automl-eEZ4gcUa': pipenv)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.10"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}