Python58-XGBoost学习与实战之初识XGBoost（一）.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 学习目标\n",
    "\n",
    "1. 从数学原理到项目实战\n",
    "    - XGBoost参数调优需要了解模型基本原理\n",
    "    - 参照Kaggle竞赛任务进行讲解\n",
    "\n",
    "2. 举一反三，可以更快更好学习其他机器学习模型\n",
    "    - 从机器学习模型的一般原理，到XGBoost的特别之处\n",
    "\n",
    "# XGBoost\n",
    "1. XGBoost：eXtreme Gradient Boosting\n",
    "    - 应用机器学习领域一个强有力的工具\n",
    "    - Gradient Boosting Machine（GBM）的优化实现、快速有效\n",
    "2. 很多数据柯旭竞赛中取得了很好的成绩\n",
    "    - Rank 1，KDD Cup 2016 Competition\n",
    "    - Rank 1,Dato Truly Native(Kaggle)\n",
    "    - Rank1 CERN LHCb Experiment Flavor of Physics Competion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#  环境准备\n",
    "1. Python环境：python3.6\n",
    "2. Python工具包:\n",
    "    - 科学计算（numpy,SciPy,pandas）,机器学习（scikit-learn）、图形绘制（matplotlib）\n",
    "    - Anaconda,集成了上面所有的环境\n",
    "    - Jupyter NoteBook\n",
    "3. XGBoost安装\n",
    "4. graphviz 安装（可选）\n",
    "    - 可视化工具包，显示树结构"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 第一讲：初识XGBoost"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## XGBoost\n",
    "- XGBoost：eXtreme Gradient Boosting\n",
    "- Gradient Boosting Machines(GBM) 的C++优化实现，快速有效\n",
    "\n",
    "–深盟分布式机器学习开源平台(Distributed Machine Learning Community， DMLC)的一个分支\n",
    "\n",
    "–DMLC也开包含流行的深度学习库mxnet\n",
    "\n",
    "The name xgboost, though, actually refers to the engineering goal to\n",
    "push the limit of computations resources for boosted tree algorithms.\n",
    "Which is the reason why many people use xgboost."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Gradient Boosting Machines\n",
    "- Machines：机器学习模型\n",
    "– 对数据的产生规律建模\n",
    "- Boosting Machines\n",
    "– 弱学习器组合成强学习器／模型\n",
    "- Gradient Boosting Machines\n",
    "– 根据梯度下降方式组合弱学习器"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 与XGBoost的第一次亲密接触"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://upload-images.jianshu.io/upload_images/1531909-7205a4b46048e54f.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/700)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 确定任务\n",
    "    - 数据集：UCI机器学习库的Mushroom 数据集(XGBoost安装包中的demo数据)\n",
    "    - 任务：根据蘑菇的22个特征判断蘑菇是否有毒\n",
    "    - 总样本数：8124\n",
    "        - 可食用：4208，51.8%\n",
    "        - 有毒：3916，48.2%\n",
    "        - 训练样本：6513\n",
    "        - 测试样本：1611\n",
    "    - 特征：Demo中22维特征经过处理，变成了126维特征量\n",
    "\n",
    "    – xgboost/demo/data/featmap.txt 文件中有特征映射说明\n",
    "\n",
    "    http://archive.ics.uci.edu/ml/machine-learning-databases/mushroo/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1_Mushroom_baseline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 导入必要的工具包"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import xgboost as xgb\n",
    "# 计算分类正确率\n",
    "from sklearn.metrics import accuracy_score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 数据读取\n",
    "XGBoost可以加载libsvm格式的文本数据，libsvm的文件格式（稀疏特征）如下：\n",
    "\n",
    "1 101:1.2 102:0.03\n",
    "\n",
    "0 1:2.1 10001:300 10002:400\n",
    "...\n",
    "\n",
    "每一行表示一个样本，第一行的开头的“1”是样本的标签。“101”和“102”为特征索引，'1.2'和'0.03' 为特征的值。\n",
    "在两类分类中，用“1”表示正样本，用“0” 表示负样本。也支持[0,1]表示概率用来做标签，表示为正样本的概率。\n",
    "\n",
    "下面的示例数据需要我们通过一些蘑菇的若干属性判断这个品种是否有毒。\n",
    "UCI数据描述：http://archive.ics.uci.edu/ml/machine-learning-databases/mushroom/ ，\n",
    "每个样本描述了蘑菇的22个属性，比如形状、气味等等（将22维原始特征用加工后变成了126维特征，\n",
    "并存为libsvm格式)，然后给出了这个蘑菇是否可食用。其中6513个样本做训练，1611个样本做测试。\n",
    "\n",
    "XGBoost加载的数据存储在对象DMatrix中\n",
    "XGBoost自定义了一个数据矩阵类DMatrix，优化了存储和运算速度\n",
    "DMatrix文档：http://xgboost.readthedocs.io/en/latest/python/python_api.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# read in data\n",
    "my_workpath = 'files/data/python58-data/'\n",
    "dtrain = xgb.DMatrix(my_workpath+'agaricus.txt.train')\n",
    "dtest = xgb.DMatrix(my_workpath+'agaricus.txt.test')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "查看数据情况"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "127"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dtrain.num_col()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6513"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dtrain.num_row()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1611"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dtest.num_row()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 训练参数设置"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- max_depth： 树的最大深度。缺省值为6，取值范围为：[1,∞]\n",
    "- eta：为了防止过拟合，更新过程中用到的收缩步长。在每次提升计算之后，算法会直接获得新特征的权重。 \n",
    " eta通过缩减特征的权重使提升计算过程更加保守。缺省值为0.3，取值范围为：[0,1]\n",
    "- silent：取0时表示打印出运行时信息，取1时表示以缄默方式运行，不打印运行时信息。缺省值为0\n",
    "- objective： 定义学习任务及相应的学习目标，“binary:logistic” 表示二分类的逻辑回归问题，输出为概率。\n",
    "\n",
    "- 其他参数取默认值。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'eta': 1, 'max_depth': 2, 'objective': 'binary:logistic', 'silent': 0}"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 设置参数\n",
    "param={'max_depth':2,'eta':1,'silent':0,'objective':'binary:logistic'}\n",
    "param"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 训练模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.022146545997031808\n"
     ]
    }
   ],
   "source": [
    "# 设置boosting迭代计算次数\n",
    "num_round = 2\n",
    "\n",
    "# 开始时间\n",
    "import time\n",
    "start_time = time.clock()\n",
    "\n",
    "# 训练模型\n",
    "bst = xgb.train(params=param,dtrain=dtrain,num_boost_round= num_round)\n",
    "\n",
    "# 结束时间\n",
    "end_time = time.clock()\n",
    "print(end_time-start_time)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "查看模型在训练集上的分类性能\n",
    "\n",
    "XGBoost预测的输出是概率。这里蘑菇分类是一个二类分类问题，输出值是样本为第一类的概率。 我们需要将概率值转换为0或1。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train Accuracy 97.77%\n"
     ]
    }
   ],
   "source": [
    "train_preds = bst.predict(dtrain)\n",
    "train_predictions = [round(value) for value in train_preds]\n",
    "y_train = dtrain.get_label()\n",
    "train_accuracy = accuracy_score(y_train, train_predictions)\n",
    "print(\"Train Accuracy %.2f%%\" % (train_accuracy*100.00))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 测试\n",
    "模型训练好后，可以用训练好的模型对测试数据进行预测"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.28583017, 0.9239239 , 0.28583017, ..., 0.9239239 , 0.05169873,\n",
       "       0.9239239 ], dtype=float32)"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# make predictions\n",
    "preds=bst.predict(dtest)\n",
    "preds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "检查模型在测试集上的正确率\n",
    "XGBoost预测的输出是概率，输出值是样本为第一类的概率。我们需要将概率值转换为0或1。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions=[round(value) for value in preds]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test Accuracy 97.83%\n"
     ]
    }
   ],
   "source": [
    "y_test=dtest.get_label()\n",
    "test_accuracy=accuracy_score(y_test,predictions)\n",
    "print(\"Test Accuracy %.2f%%\" % (test_accuracy*100.00))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 模型可视化\n",
    "调用XGBoost工具包中的plot_tree，在显示 要可视化模型需要安装graphviz软件包（请见预备0）\n",
    "\n",
    "plot_tree（）的三个参数：\n",
    "\n",
    "模型\n",
    "树的索引，从0开始\n",
    "显示方向，缺省为竖直，‘LR'是水平方向"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAADYCAYAAADlAyjBAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xl0U2XeB/BvurEOLVtZFIoCZRlbNkUQqiDvsLeAzIECLuOIGlRknfeFkjqDqHS05eig4LQ4Do6lRRCQRQacgtIyLWJpWiiFIksKAokWEuhC6fK8f1xvSNKkTdMkz703v885OWmTu3xbyi9Pntz7uyrGGAghhCiXH+8AhBBCPIsKPSGEKBwVekIIUTgq9IQQonBU6AkhROGo0BNCiMJRoSeEEIWjQk8IIQpHhZ4QQhQugHeAX9HpuYQQ0nQqZxaiET0hhCicVEb0hMhSUlIStmzZghMnTiAiIgJTp05FeHg4wsPD0aJFCwBATU0NSktLkZ2djcLCQuzcuRP+/v5YvXo1nnnmGfTs2ZPzT0GUTiWRpmaSCEFIY0wmE5YvX46tW7di+fLleOONN5q1vZycHMyYMQOVlZU4deoU7r//fjclJT7CqakbKvSENOLLL7/Eyy+/jK1bt2LcuHEe398DDzyAjh074ocffvD4vojsUaEnpDlmzpyJ3NxcHD16FPfdd5/X9z9s2DC0atUKR44cgZ8ffZxG7KJCT4gr/Pz8UFhYiAEDBvCOYtaqVSusX78e8+fP5x2FSItThZ4+jCXkVy+88AJOnz6Nuro63lHqqaysxPnz5zFp0iTs37+fdxwiMzSiJwTAo48+ioyMDLRt25Z3lAbV1dWhc+fOKC0t5R2FSAMdR09IY9atWwedTodjx45JvsgDwrRSaWkpbt26RfP2xGn0l0J8Vrdu3RAVFYWwsDDeUZqsXbt2qKurQ+vWrXlHITJAUzfE51y/fh2RkZEwGAy8o7hFcXEx7ty5g8jISN5RiPfR1A0h9sydO1cxRR4AwsPD8dFHHyElJYV3FCJRNKInPuOTTz7BlStX8Oc//5l3FI9ZuHAh1q9fzzsG8R4a0RNiqbKyUtFFHhBO8lqyZAnvGERiaERPFO/YsWPYvn073nvvPd5RvCY1NRXz5s3jHYN4Ho3oCTGZTIiLi/OpIg8AR48exZkzZ3jHIBJBhZ4o2okTJ5CRkcE7htdt2LABLVu2xKpVq3hHIRJALRCIYg0YMABFRUW8Y3DTq1cvtGnThncMSbl9+zbGjh3LO4bLbty4gQsXLjR5PSr0RJFCQ0MVdQilq+Li4hAUFIS7d+/yjiIJly5dknX755UrV7q0Hk3dEEU6fvw47wiScevWLaxYsYJ3DMIRFXqiOH/4wx9k2dbAU1q2bIlbt27xjkE4okJPFCU5ORkbNmzgHUNyNmzYgAULFvCOQTihQk8U5dNPP6VGX4TYoEJPFOP8+fPIzs7mHUOyNm7ciHbt2vGOQTigQk8Uo1+/frwjSF5JSQnvCJKTnp7e6DI5OTlQqZw6CVWSqNATRXjrrbfoEEInhISEYPr06bxjSMqcOXMaXWbkyJFw1C6moReA+Ph4h89ZrufpFxEq9EQR3n33XbrikpOuXr3q8rpJSUnmopSUlISSkhIYDAbExMSYH3PULlkcOYsfCov34roqlcp87kNSUpL56/T0dPM+VCqV+V1JTEyMW8+VEHPk5+djwYIFKCkpqfcBtkqlsro1Zs2aNQ6f2717t93HVSqVU+8ymoQxJoUbIcSLxo8f7/K6Wq2WMcZYYmIiY4wxCE0JzTej0Wh3PaHcOL5PS0urt6yj5dVqdb3lnFFQUNBoLtHZs2ft7tvy1lAGjUbjMIflz2q7DaPRyM6ePWt3vRUrVtSL78yNhkBE9g4fPsw7guw0p+HZoEGDkJ6ejlGjRpkfsywqrp6cNWDAAPMomTHW4Ih548aNSEpKQlpamkv7csRkMpnftYSHh9d73raANsTRaN5gMCA2Nhb5+fl2nw8ODra772Zx9hXBwzdCXBYQEMA7gs+Jjo42f63T6cwjbL1ebx6dajQaq+VgMxqGzcg4MTHR/C5Br9c7fMcAgGm1WvPX4r4tVVUxBgg3f3/hfvFixv7znyLP/EK8xNURPe8CT4WeNNvatWt5R5Cl0tJSl9fNyMhwarns7GyX9+Es22kQxhjLzb1X6G0LvpzR1A3xSRMnTqQ+Li4aMGCAy+s++eSTTi03YsQIl/dhz5EjwFtvAb/9LaBSAV27Atu2xWLHDsDyc9mhQ4GgoHvff/MNUFMDFBScdGseuaDulUTWDhw4wDuCbEn1cNTMTOG2ZQtQWCg8Nn068PjjwJw5wr1G0/h27t4Fxo8H6E+ECj2RuRkzZvCOIFvLly/ntu/Ll4HUVODoUWDvXuGxadOAqChg7lzhPi6uefto5LNSn0KFnsjaxo0beUeQrVWrVmHdunVYunSp27a5f79QvFNTgUuXgMBAIDYWGDUKmDcPaNtWWK5HD4Bm3LyHCj2RtS5duvCOIGtbtmxpcqE/fRpISxOmV777DggIuFfM584FJk0S5tGJdFChJ7JVXl5Ol8prptzcXPPX27cDWVnC3PjPPwsfeEZFCQU8NlYo6AAwcCDQwAmfRIJUTBoTWZIIQeRl3759mDJlCu8YsjZw4EBERZ3G6NFCMQ8M5J3Is+rq6hDX3Ml/zhISEiy/dapJDo3oiWwdOHCACn0zTZw4EevW8U7hPX5+fraF0ifQcfQ2ioqKMG3aNLRt2xb+/v4YMGAApk2bhv/93/9FXFwcVqxYgSVLlmDy5Mno3bs3VCoVQkND8de//hVXrlzhHd+nFBUVubyu2DgqPz+/3qn2YnOrhqSkpNg9Rd+y2ZXBYKi3jLg/k8lktzFWY42yYmJisGfPHqt9NafzYXOOpScy4uyZVR6+cfHDDz+wRx55hE2ePJkdPXrUbdvdu3cv8/PzY9OmTXPbNkl9YWFhLq0HBw2p7DWh0mg05iZe9tiefm+7DaPRaH7Mclm9Xs8Ysz6r07J9gF6vt2ofYLmsuK6ooXyN+eabb1xel0gCtUBwZM2aNQyA3U527vbHP/6R+fn5sc2bN3t8X74mNDTU5XXFgmo0Gh12U3TUhZExZu4uCJtz6sUeLJb7EXu2aLVaZjQardoC2BZv2wyWLxy2+xK32RzuHOAQLqjQW7p06RJr164du3jxojd2Z9eRI0eYn58fKysr45ZBSbp37+7yuvaKpr1RPmNCMbYdRbuyL/GFQxyp27bldfROQ5ScnMwYE5qIiZpb6DMzM5u1PuHOqRqr+Dn6y5cvIygoCJmZmTCZTOjVqxe3LFFRUaitrUVCQgId/+0GrVq1cmk9sT2seHEHcc5bo9HAYDAgOjraavnY2FiEhobW20ZSUpLVY7bbS0pKgslkgubX8/WfeeYZAPYvOCH+h7RkeZENAHjxxRdhMpmwdu1a82ODBg1y5kd26Pbt281avyksLxwiXtAjJSUFMTExMJlMXsvhixR7eOU333yDqVOnoqqqyt2bdpvTp0/jt7/9bb3/4MQ5U6ZMwb59+3jHkLX169dj4cKFXtufSqUCY8yq77zl46TJnPokXpEj+vT0dJSXl0u6yAPCMcyMMbRo0QInT/pmV73moIuBN9+5c+e47TstLQ05OTkwmUzIyMjglsMXKK7Qd+3aFd27d5fVBZCrqqpw6NAhXLp0iXcUWXnooYd4R5A9bw4wxFG8eB8bG4uTJ09i//79Trc9Jq5RzNTNnj178PbbbyMnJ8cdebjp1KkTfvnlF94xZKG6uhqBSj+V08OspkwqKu71B87MBC5eBHr2FPoCx8YK97/5Dd/AzRAZGYmCggLeMdzNd86MLSsrwy+//CL7Ig8Av/zyC/z9/VFbW8s7iuRRkW++Dh063PumdWtgwgThZqusTCj+aWnC1T90OqBXL6El5ahRQiczIlmyn7oJCQlBbW0tnn/+ed5R3Ka2thZ5eXmyGtnHx8dz2e+PP/7IZb9KYXkET4PathWK+WefCf2HGRNG/G+9ZV3ka2qEzmivvAJERAiXgercWbhyyLZtwLVrHvk5SMNkPXWze/duREdHN+sUcCkbOnQodu7cibCwMN5RJGvVqlV4++23eceQpZ07d2LChAlo3bq1d3ZYXQ3k5AjN6rOyhMtHhYYKl42KigJmzvTo7n156ob3iVIunzAVFBTkymqyU15ezvr06eP27VqenanX663OzBTP5ISdk3aio6MZAJadnc3UajXT6XRMrVabl9XpdOYTgxITE81nnoonCcHOWahpaWl29+UMV9cjEv8/dOIEY8uWMTZiBGMqFWMtWzL2u98x9s9/MnbuXOPrA4wFBzNmMpkfioiI8GBgbpR7ZmxBQQGrq6tr6mqyZTAYWHl5udu3K55pydi9Ag6AqdVqq7M2bdkr1pZfi4VerVbXayNgb13Le0cqKoS/1sBA4R5gLCSEsaAg9/9efIVsC19uLmNLlzI2fLjwh9CqFWPz5zO2eTNjVVXCMuIfCcDY//0fY0zGP2/DlFnoY2JimMFgaNqvQgEyMjLYhg0b3LpNtVptHmknJyczo9Fo1VjLUfFtqNBbFnZ727F9rKF+MvX3a30TOyAsXbrU6W2Qe+7cucM7gufY/rGINxvOtrYA0Ox2Ex6izEJ/8ODBpv0aFGTJkiVu36bYnIsx4Y9Zp9NZTd8wZt11USzqjm62LxS2X9sWest3EvYcPszYI48wNnIkYxMm3Pv/umvXvWVo+qbpFF3kGRP+SIKChPuEBMYqKlwe0dt2KLVHo9HU+zvMzs62+n+RlpbGjEaj+Z10dHS0w3e8jNVvkOeA8gp9//79nV1Usdw1r6rRaOy25bXH8sXAnaKjo80NupKTk1llpfAuOyCAsTfftL/O44/Xf+zMmTMeyadkv/nNb3hH8LqGCr3lZ1RqtZrp9Xpzgbc3WLFXgB0VZXvbsSS+qxZfGJzZpuUiztx4F3inC33Hjh2dWUzxampq2KRJk3jHcKtnnxX+ElNSXN9GdXW1+wL5gPnz5/OO4HUNFXrbd5biO1vGGi629qYlbTlqQd3QNsRRv7sKvSwOrzQYDPW6BxJ5unYNGDIEqKsDTpwA7r/fPdulk8ycd//99/vk1dAaOrwyJSUFs2bNQlVVFbp06SIUR4sGbM7UycaWs91eTk4ORowYgZKSEvTs2dPuNpzYt3KamvFsLSxVcnrhO3ECGDdOOH9GqwWuXwcMBvcVeQBITU1FWVmZ+zaoYHPmzOEdQTLESzG++OKLCAkJwZ07d6DVahEfHw+1Wl2vP09D27HH8kRC2xeNn3/+GTExMeYi70mSH9EnJibi9ddfR1BQkDfzSJ5Wq0VAQIBkG3vt2wcsXw706AG8+y4weLDn9xkWFgadTuf5HRFZ8uUTpiQ9oj916hTGjx9PRd6OwYMHY+/evbxjmC1cKJztvm6d8P2UKUBREXDwoHeKPAAUFhZ6Z0cyFhCgiPZWpIkkPaIPCgrC3bt3vZ1FVnh2cPz4Y2DBAuD554F//INLhHqSk5Px0ksv8Y4hScePH0dwcDDCw8N5R+GCRvQSRUW+cd27d/fKfs6dE/pS9eoFfPGF8JhaLRzVLpUiDwg96s+fP887hiR9//33PlvkfZ1kC/3rr7/OO4IsFBUV4cCBAx7Z9pEjwKOPAgEBwgeou3YJjQtnzfLI7tzisccew2BvzRXJyOnTp/Hqq6/yjkE4kWShZ4yha9euvGPIQqdOnbB69Wq3bCsuDvD3B1avFrrNPv44cOyY8HVUlFt24RW3b9/GN998wzuGZFy6dIkuVenjJDlHP2/ePKSmpvLKIjvl5eVo06aNS+umpgJPPy10iN2+3c3BOBoxYoQiLkTjDt26dcM16gPv03P0kiz08+fPx6ZNm3hlkaX169dj4cKFDS5z/Trwpz8B//43sHYtMH++l8JxEhsbi/T0dN4xuJo8eTK+/vpr3jEkYdeuXYp58U9ISBC/lG+hr6mpocPAmsjRmaF5eUJxz8gQjm2fPJlDOI7eeecdxMXF8Y7BhVjgJ/vaP7pvkedRN3v37qUi74L09HRUVlZi7VqgRQvhZKWKCqHdwH/+Ixwd44v/31euXIkuXbrwjuF1I0aMwJgxY6jIEwASHNE/+OCDuHDhAs8ssjV+/BYcODAXCr2yostMJhMmTpyI7Oxs3lG8gjGGU6dOISIigncU4nnyHNE/8MADvCPI1qFDz1KRtyM4OBjZ2dl46qmneEfxuPj4eOzdu5eKPLEiqUJvMpmwfPly3jFkS61W844gaWvWrMETTzzBO4bH1NbWYvjw4YiOjuYdhUiMpKZuPvjgAyxatIh3FtnKzc3FsGHDeMeQtJKSElRVVaFv3768o7jVzZs30bt3b9y4cYN3FOJd8jvqpnXr1qioqOCdhSgcYwz+/v6oq6vjHcUtNm3ahDlz5rh8LgWRNfnN0VdWVvKOQHyASqVCXV0devTowTtKsz3yyCPo3r07FXnSIEkV+ilTpvCOIHtfiB3HSKMuX76MU6dOYfPmzbyjNNl3332Hbt264fjx43QIJWmUpA5YHzhwIO8Iskc92ZvmoYcewu3bt9G2bVvZXKFqx44dqK6uprYGxGmSGtE/9thjzVrfZDLBZDKZL+sVExMDg8FgdTkvkeVjJpMJKSkpiImJqbeceKkxAMjJyUF6ejqSkpLMz+/Zs8dqOctLiuXn5zeYNz093bxvy20YDAZnf+R6/vvf/7q8rq8aOXIkysrKsHv3brRu3Zp3HIc2bNiADRs24KmnnsLs2bN5xyFy4uxVxD18Y4wxlpmZ2dgVz+1fBt3i6u3i9/buGWNMo9HUWz86OpoxxpjRaGRqtdr8uE6nq7e80Wg0P2a5rEir1Zq/Fq/+Ln4t7sdRTndo6Er3xDnBwcHs3Xff5R3DrLq6mgUGBrKSkhLeUYj0OFVjeRd4q0JfUFDglp88Oztb+A3YFFK1Wm1VfMUXiOzsbKZWq5lWq2WJiYn1tpecnGz+WqvVWr2waLVauwXflriO7c0yH2P2X4iaomfPns1an9yzadMmBoClpKR4fd93795l/v7+LCEhwev7JrLim4XeXqG0fcy24IvsFXl7ADC1Ws2MRiNj7N47grS0NKtlHBVzy2Vsn7N8R+AKKvTuZzQaGQC2aNEij++rsLCQAWCrV6/2+L6IIsiv0Dd36katVjdaXB2tbzlqF18YAFi9SACwejFITk62Ws/ei4dIp9PZHfm7czTPGGMPP/xws7dBGrZr1y4GgD366KPNemEuLi5m8fHxDACLjY11Y0LiQ5yqsZI6YSo1NRXz5s3jnUXWZsyYgZ07d/KO4TPOnj2L1NRUrFu3DuXl5YiIiEB4eDj69OmD9u3bAxDOWi0vL0dWVhbOnj2L2tpajB07FmvXrsWQIUM4/wRE5uR3Zuxrr72GDz/8kHcWWUtMTKR+QYT4DvmdGXv8+HHeEWTv4Ycf5h2BECIxkhrRq1QqSCSPbNHVuQjxKfIb0ffp04d3BNmjIk8IsSWpQk8fxDbPxYsXeUcghEiQpAr9X/7yF/ztb3/jHUO26MIjhBB7JDVHDwA9evTA5cuXeWaRLfqMgxCfI785eoCOGnFVbW2tuTkaIYRYktyIvqSkBD179uSZRZbi4+OxZs0a3jEIId4lvxOmRMXFxQgPD+eVRZbCwsKg0+l4x/B52dnZKC4uxo8//ojq6mrU1NQgMDAQHTp0wMiRIxEeHo7Q0FDeMYlyyLfQR0ZGoqCggFcWWTp+/DgeeeQR3jF8yubNm7FkyRIYjUY8/fTTGDNmDH73u985vETh1atXUVxcjH/961/49ttvcf36dcydOxcff/wx/P39vZyeKIR8C/2mTZvw7LPPIigoiFceWcnMzERUVBTvGIp28eJFzJ49G927d8f777+PXr16uX0fTz/9NFJTU5GXl4fBgwe7fftEkeT5YSwAzJ8/n64f2wQTJkwAXnoJ8NLVkdLT07Fnzx7zFbnEq3o158pYUrVjxw74+/tDp9Ph+++/x65duzxS5AHg888/Nx811adPHyQnJ3tkP8T3SHJEDwD79u2jYu+EL774AhMnTkS7du3uPfj3vwMffggcOQL82kHRncTDOB3dy92OHTswc+ZM1NTUSGJKZdq0aZg8eTJefvll3lGI9Mh36kZUWVmJVq1aeTuLrLRo0QJVVVX2n0xOBv72NyAz060FX8mFfujQodiyZQv69+/PO4oVvV6PXr164datWwgMDOQdh0iHfKduRJ56i6wkFy5ccPzkSy8Bp04B27YBKhVw86ZHMkRHRwOQ95m5P/30E8aOHYsTJ05IrsgDQJcuXVBZWYnnnnsOY8eO5R2HyIykR/R1dXXQaDR45513vJ1HFoYMGYK8vLymrXTiBDB6NHDwoHDv427duoW5c+di7969vKM0yauvvoqLFy/i66+/5h2F8OXUiF7SrQ79/Pzw448/8o4hSUajEcuWLWv6ikOHAhUVQsFv3dqnC/7rr7+O0tJS2RV5APjoo49QVVWFKVOmYN++fbzjEImT9IheFB4ejuLiYm9lkYVXX30VH330kXs2tmkT8OKLQGkp0KGDe7YpYRUVFWjXrh1qamp4R3GLrVu3YvTo0bjvvvt4RyHeJ/8PYy3V1tZK4ggIKTh//jx69+7t/g37SMEfPnw4vv/+e94x3Gr27NkYPXo0Fi5cyDsK8S75fxhracmSJbh9+zbvGNxlZWXh/Pnzntn4/PkAY0KRz8sDWrYUDtFUiFmzZmHbtm2KK/KAMKpfuHAhnR1N7JLNiB4AWrVqhcrKSk9nkTSXPoBtDq0WGDMG2L0bePxx7+3XzUwmE65duybJI2rc6ebNmxg8eDD1PfIdyhrRA8Jx9b58aFnHjh29W+QBYPBgwGgUinxZmXCY5rp13s3QTMOGDcOdO3cUX+QBoH379tDpdNQ4jViRVaEHgISEBKxdu5Z3DK8bNWpUw8fMe0PbtsLUzssvy6bgjxs3Dl999RW6dOnCO4pXXbp0ia7BTMxkNXVj6fDhwz4zun/jjTfw5ptv8o5hX34+8MQTwK5dwhSPhGzcuBELFizgHYOr0NBQRfYgImbKm7qx9Omnn3p/GoOD9evXS7s3/6BBwtROhw5ASAjw7be8EwEAioqKcOXKFd4xuDMYDPjqq694xyCcyXZEDwiHXP773/9WbPOzBvvYSFlFBdCmDZCYCLhyUpcbnDlzxifm5J1hMpnw8ssvIz09nXcU4n7KHtEDgL+/PwoKCrBq1SreUdzuiSeekGeRB4QzbhkDXnlFmMtPTPTq7nv37k1F3kJwcDAmTpzIOwbhSNYjelFWVhYeeOABxZwZ2LlzZxQWFirnyInKSmDcOOCpp4Dlyz26q23btqFfv36IjIz06H7kaOXKlT55IIPooYceQmFhIe8YbmFRt5V1ZqwzAgMDUV1d7Y5NcXH8+HFMnz4dP/30E+8ontW+PTBhAuCBqYRz586hb9++bt+uEpSVleH999+HRqPhHYULhV6iVPlTN7aqq6vh5+cnyykPnU6Ho0ePKr/IA0K75PR0oeDHxrpts4sXL6Yi34C2bdtSzygfpahCDwitjVu0aIH777+fdxSnlJWV4dFHH0VYWBgWL17MO453iQW/slLooPnuuy5vat++fVixYoUbwynTZ599hrfffpt3DOJliiv0oitXrmDkyJG4du0a7ygOjRw5EosWLcKxY8d4R+GrVSsgKwtYuNDlgv/KK6+ga9euHginPJmZmbwjEC9T1By9I++88w7ee+893PTQFZaayt/fH9u2bcNTTz3FO4q0dewofIi7datw9M6rrwI7dwJXr1otVl5ejjZt2nAKKU8BAQGKadPsLF+eo/eJQi+KjIxEp06dcOjQIW/srp6+ffti4MCBdAJLUy1eLBT3L78E6uqAgADA4kP3bt26SfqdmxTl5eVhyJAhvGN4lS8XesVO3dhTUFCAQ4cOgTGGpUuXerwT5MGDB6FSqZD463Hk586doyLvivffB0JDhSIPADU1QrE3mfD111/j5MmTfPPJ0JAhQ/DWW2/xjsGdMy0yxIFhTEyMw2VycnLsPp+SklLvRDWVSqjNe/bsMa9jMpmgUqnMz5lMpgb312SMMSncuCkqKmItW7Zkffv2ZYWFhc3eXkJCAuvYsSObN28eKy0tdUNCwhhjTDgFq95NpVLxTiZbbdu2bfI6iYmJTCgbwtc6nY7p9XoWHR1tfiw5OdnuumlpaYwxxtRqtdW9uC4AptfrzdsRv05LSzPvAwDT6XTm9cRlnBEREdGkn1Uk/rxNXS4tLY0ZjUam1WpZRkYGY4wxo9FoXi47O5sxxphWqzX/bix/Hif361SN5V3guRd6e27cuMG2b9/OpkyZwh588EEGYWrJ6ubn58f69+/Pli1bxvbt28c7ss8ymUy8I8haXFxck9fRarWMMaEYM8bq/d8wGo121xMLl6N7sdhZPuZoefFFwtkizFjDhV7cNwCm0Wjs7rcxtsup1Wp29uxZ83NiEbddzvJ7R7+DhnbrzI13gZdkoSfycOPGDd4RZM/Pz8+l9dLS0swjUnsFzp7GCr1Wq220wFveJyYmWhXGxjRU6G1/BvHdg+1+LW+269krzNHR0SwjI4NptVrzOxFHvy/bdydU6AlhjAUGBvKOIHvXrl1zaT1xuoUxxnQ6nVWxEguURqOxWs7eO2PLopmYmGh+l6DX6x2+YwBgflGwHN1bGjuWsf37rR+zV+j1er35Jm4vOjqa6XS6Jo3oLadkLInvfiyJy6nVaqt3P7YvWu4s9D511A1RlpkzZ+LLL7/kHcMnHTp0CE8++WSjy+Xk5GDEiBEezZKeno5YmzOsVTbHovzP/wBXrz6GwsL/ejQLB3R4JVGuo0ePIiwsTDZnQEvZhx9+iNdee83p5VUqFXjXjfT0dMyZMwcZGRn1XnBOnQIiIuqv06bNDygre9hLCb2GCj1RLtn26pcgKRTuxtTVATk5QFoakJkpXNisfXtg3jxg1Chg1izAz+JgcXFEHxAAGAzCsr58HH2Ap1MQ4gk0knef3//+97wjABCOlz12TCjmR44AWi0QHAxERQFz5gjXp1+/3vntJSUBS5d6Lq8n508+AAAJuElEQVSc0IieyFJWVhZGjx7NO4YilJeX49tvv/XYldoqK4VWRqmpwNGjwI8/AvfdJ4zGo6KAqVM9stt6fHlET4WeEILY2NhmXWrwzh2hiIvFvLgY6N5dKOTi9EqHDm4M7AIq9PxJIgQhvsrRPP0vvwij8S1bhPtr14D+/YWplKgoYOxYDmFd5MuFnuboiez4YkMuTwsODsbs2UIxv3oV6NfvXjGfNAmYPp13wua7e/cuLly4wDuGWzz44INNWp4KPZGdrVu3UqF3s1mzZiE5mXcKzzpz5gzvCNz4VPdKogx0OTz369+/P+8IxIOo0BPZyc3NdXldsS1tfHy8uSUsILSgFZ8T28PGx8fXW9/2sZSUFBgMhnrLWbaczcnJAQAkJSVZ7U98znJZAMjPz7f63h6TyWTOK67f2DoNGT58uMvrEhlwtleCh2+EOC04ONil9eCgIVVaWpq53W5ycrLdHicajcZu3xKRbb8VsQOivccsl7VsxWv5mLhvy69F9lraMma/r4qzCgoKXF6XcEVNzYgy2RY+V9e1V9DFFwLLJlViP3R72xGLttiAy5JtX3bb3PYKvriOZTZ7N1v2XliaQuysSGSHCj1RJlcumCGyV+iTk5OZXq9nOp3OqjjbFk97Bd+VfYsdC8XOjrY9yO2967AkZrQszs0ZzTPmnRG95YVDxBe55ORkFh0d7bCHPWkUFXqiTJ07d3Z5XXuFnjFhdC0WH6PR6LCnui2tVltvNC8W7t27dzPG7o32xRcOy9a9lsvbyykWfdsRd1MyOiMrK8tt22qI5c9l+fu2/Z0QpzlVY+mEKSI7jz/+OI4cOcI7hqJs3rwZzz33nMf3I56YpVKpkJaWhl69emHAgAHIzc11qu0xqYcuDk6UKTw8nHcExfHGIaviUUHifWxsLE6ePIn9+/dTkfcwGtET2dm5cydmzJjBO4aiDB48GNpBg+41p2nVinckj7hy5Qp69OjBO0az2NRsaoFAlGnMmDG8IyhOfn6+0BcYAPbuFZq+p6YCP/0EtGgBxMYKLwJz58r6ReDmzZu2hVJWVq5c6dJ6VOiJ7LRv3553BMUZOHDgvW+mThVuf/1rwytdvCi8GGRlAQcOCI/NnHmvgXxoqOcCkyahQk9kyWQyITg4mHcMxZg7d27TV3rgAUCjsf/c4cPAxx8LPYsPHhQu+fT73wOjRwvvCjp1al5g0iRU6IksLVq0CP/85z95x1CE9evXY9WqVe7d6NixzvUw1uvvXR9wxw7hsUmThAb28+YBvXq5nuHnn4HOnV1fX0How1giS3K4zqlc9OjRA5cvX+Ydo77vvxemhTIzgV27hMemTBFeBObOBcLCGl5fpRIuJFteDrRsCQA4efIkIuxdOVwmVq5cibVr11o+RB/GEuWKi4vjHUExJFv4hg8Xbo1d+LWiQvisIDMTSE8HqquFKaLAQOFr8cPj1asBHz1ai46jJ7L05ptv4pNPPuEdQxHef/993hGap3Vr4MUXgc8+A+7eFa4ynpkJ1NZaL/fnPyMiMrLe6s5cQlHsMipXVOiJLPn7+0OtVvOOIXsGg0G5J6DV1QFDhgA6nVD8GcNJO5cSnDNnTqObGjlypMOpQnsvALYnh+3Zs8fhcwsWLIDJZGo0Q3NQoSey9fHHH/OOIHuzZ8/mHcFzGANOnAB69nRqcfEaAfn5+ViwYAFKSkrMPf9Flr3/GxrhazQa5OTkQK/Xm7ctbt/W5MmTERISYrUPR8u6igo9ka0XXniBdwTZo6t13bNnzx6oVCoMHjwYGzduxLZt2+oNJmybhTkSEhKCAQMGYP78+eb1oqOjzc9bjuCjo6PBGDO/qNgu6w5U6Ims0ZWRXLdgwQL89NNPvGNIitFoRGJiIlQqFZYtW+bydpYvX47g4GCrKZtZs2YBEAp5cHAwjEYjgF/PSgbwxBNP1FvWXajQE1lr06YN7wiytX37dt4RJEEcmYsFeNmyZVaPWd43tL6jbYpsT/ATvx80aBAAocmbo2Wbiwo9kbXDhw+je/fuvGPI0s8//8w7AvESKvRE9l555RXeEWTHcvRIlI8KPZE9jUaDf/zjH7xjyMaJEyewevVq3jGIF1GhJ4qQkJDAO4JsREVFoV+/frxjEC+iQk8Uobi4GC1atOAdQ/JKS0tRXl7OOwbxMir0RDEMBgPvCJJ333338Y5AOKCmZkQxgoOD0aFDB9y4cYN3FEkKCgrC3bt3ecfg7sKFC7wjuMzV7NSmmCjOkCFDkJeXxzuG5FRVVdH0FoDc3FzeEZpl2LBhlt861WmNCj1RpHPnzqFv3768Y0jC7du38fzzz9MJUspEhZ74rr59++LcuXO8Y0hCu3btcOvWLd4xiGc4Vejpw1iiSOfOnYO/vz/vGNzl5uZSkSdU6Ily1dbWokePHrxjcBMVFYXevXvzjkEkgAo9UbRTp05J91J5HjR9+nR8+umnVn3Oie+iOXriEyIjI1Fg5+pCSjRhwgQcOHCAdwziHfRhLCGWunXrhmvXrvGO4VFDhgzB9u3bacrGd9CHsYRYunbtGj744ANUV1fzjuIRgYGByMvLoyJP6qFCT3zKokWLMGjQIGRmZvKO4ladO3dW7AsYaT4q9MTnnD59GlFRUejcuTPvKM3Wv39/7Nixgy4iQhpEc/TEp3Xt2hXXr1/nHcMlgwYNQl5eHvz8aLzmw2iOnpDGXL9+HZWVlWjdujXvKE4bM2YM9Ho98vPzqcgTp9CInpBfDR06FP369UNaWhrvKHbp9XoMHz4cly5dgkrl1ECOKB8dXkmIK9LT0zFnzhzU1dVJoqDOmjULjz32GBYvXsw7CpEeKvSENMeiRYuwceNG7N27F+PHj/f6/kNCQjB16lR8/vnnXt83kQ2aoyekOT744APcvXsXvXv3hkqlwp/+9CeP7/PkyZPmfRmNRiryxC2o0BPSiN69e4Mxhvfeew8AkJmZiZCQEISGhuKzzz5z+apN27Ztw4wZM9CyZUusWLECABAREWG1L0LcgaZuCGmGuro67NixA3//+9/x3Xffobq6Gp06dUJYWBjat28PALh58yYqKipQVFQEABg9ejTGjRuHuLg4BAUF8YxP5I/m6AkhROFojp4QQggQwDvAr/gfw0YIIQpFI3pCCFE4KvSEEKJwVOgJIUThqNATQojCUaEnhBCFo0JPCCEKR4WeEEIUjgo9IYQoHBV6QghROCr0hBCicFToCSFE4ajQE0KIwlGhJ4QQhaNCTwghCkeFnhBCFI4KPSGEKBwVekIIUTgq9IQQonBU6AkhROGo0BNCiMJRoSeEEIWjQk8IIQpHhZ4QQhTu/wGY+nv7IapIRwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from matplotlib import pyplot\n",
    "import graphviz\n",
    "xgb.plot_tree(bst, num_trees=0, rankdir= 'LR' )\n",
    "pyplot.show()\n",
    "#xgb.plot_tree(bst,num_trees=1, rankdir= 'LR' )\n",
    "#pyplot.show()\n",
    "#xgb.to_graphviz(bst,num_trees=0)\n",
    "#xgb.to_graphviz(bst,num_trees=1)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}