Deep learning Project on MNIST DATA.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "id": "8qVhGJc29eiR"
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn.datasets import load_iris\n",
    "from sklearn.linear_model import Perceptron"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting threadpoolctl\n",
      "  Downloading threadpoolctl-3.1.0-py3-none-any.whl (14 kB)\n",
      "Installing collected packages: threadpoolctl\n",
      "Successfully installed threadpoolctl-3.1.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
      "mlxtend 0.21.0 requires matplotlib>=3.0.0, which is not installed.\n",
      "\n",
      "[notice] A new release of pip available: 22.2.2 -> 22.3\n",
      "[notice] To update, run: python.exe -m pip install --upgrade pip\n"
     ]
    }
   ],
   "source": [
    "!pip3 install threadpoolctl\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting matplotlib==3.0.0\n",
      "  Using cached matplotlib-3.0.0.tar.gz (36.3 MB)\n",
      "  Preparing metadata (setup.py): started\n",
      "  Preparing metadata (setup.py): finished with status 'error'\n",
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  error: subprocess-exited-with-error\n",
      "  \n",
      "  python setup.py egg_info did not run successfully.\n",
      "  exit code: 1\n",
      "  \n",
      "  [47 lines of output]\n",
      "  ============================================================================\n",
      "  Edit setup.cfg to change the build options\n",
      "  \n",
      "  BUILDING MATPLOTLIB\n",
      "              matplotlib: yes [3.0.0]\n",
      "                  python: yes [3.8.10 (default, May 19 2021, 13:12:57) [MSC\n",
      "                          v.1916 64 bit (AMD64)]]\n",
      "                platform: yes [win32]\n",
      "  \n",
      "  REQUIRED DEPENDENCIES AND EXTENSIONS\n",
      "                   numpy: yes [version 1.20.2]\n",
      "        install_requires: yes [handled by setuptools]\n",
      "                  libagg: yes [pkg-config information for 'libagg' could not\n",
      "                          be found. Using local copy.]\n",
      "                freetype: no  [The C/C++ header for freetype\n",
      "                          (freetype2\\ft2build.h) could not be found.  You may\n",
      "                          need to install the development package.]\n",
      "                     png: no  [The C/C++ header for png (png.h) could not be\n",
      "                          found.  You may need to install the development\n",
      "                          package.]\n",
      "                   qhull: yes [pkg-config information for 'libqhull' could not\n",
      "                          be found. Using local copy.]\n",
      "  \n",
      "  OPTIONAL SUBPACKAGES\n",
      "             sample_data: yes [installing]\n",
      "                toolkits: yes [installing]\n",
      "                   tests: no  [skipping due to configuration]\n",
      "          toolkits_tests: no  [skipping due to configuration]\n",
      "  \n",
      "  OPTIONAL BACKEND EXTENSIONS\n",
      "                     agg: yes [installing]\n",
      "                   tkagg: yes [installing; run-time loading from Python Tcl /\n",
      "                          Tk]\n",
      "                  macosx: no  [Mac OS-X only]\n",
      "               windowing: yes [installing]\n",
      "  \n",
      "  OPTIONAL PACKAGE DATA\n",
      "                    dlls: no  [skipping due to configuration]\n",
      "  \n",
      "  ============================================================================\n",
      "                          * The following required packages can not be built:\n",
      "                          * freetype, png\n",
      "                          * Please check http://gnuwin32.sourceforge.net/packa\n",
      "                          * ges/freetype.htm for instructions to install\n",
      "                          * freetype\n",
      "                          * Please check http://gnuwin32.sourceforge.net/packa\n",
      "                          * ges/libpng.htm for instructions to install png\n",
      "  [end of output]\n",
      "  \n",
      "  note: This error originates from a subprocess, and is likely not a problem with pip.\n",
      "error: metadata-generation-failed\n",
      "\n",
      "Encountered error while generating package metadata.\n",
      "\n",
      "See above for output.\n",
      "\n",
      "note: This is an issue with the package mentioned above, not pip.\n",
      "hint: See above for details.\n",
      "\n",
      "[notice] A new release of pip available: 22.2.2 -> 22.3\n",
      "[notice] To update, run: python.exe -m pip install --upgrade pip\n"
     ]
    }
   ],
   "source": [
    "pip install matplotlib==3.0.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 36
    },
    "id": "S_kjO0xQ94nV",
    "outputId": "86656e42-9e65-42fa-fcef-dededb061f77"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'2.4.0'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "\n",
    "from tensorflow.keras.layers import BatchNormalization\n",
    "tf.__version__\n",
    "keras.__version__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "xDrqlCAu99fG",
    "outputId": "b91188c8-d999-49dc-b9fc-55ba1c1d57f2"
   },
   "outputs": [],
   "source": [
    "fashion_mnist = keras.datasets.fashion_mnist\n",
    "(X_train_full, y_train_full), (X_test, y_test) = fashion_mnist.load_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "w12AR46U-DV1",
    "outputId": "2369c087-c963-4860-f009-611044b93b78"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(60000, 28, 28)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train_full.shape\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "1asHEAuR-KOb",
    "outputId": "c1ab63d9-8724-4827-8615-90b014bcbc87"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dtype('uint8')"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "X_train_full.dtype"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "id": "76k8x8O9-UTw"
   },
   "outputs": [],
   "source": [
    "X_valid, X_train = X_train_full[:5000] / 255.0, X_train_full[5000:] / 255.0\n",
    "y_valid, y_train = y_train_full[:5000], y_train_full[5000:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "id": "044_DyN1-ZBI"
   },
   "outputs": [],
   "source": [
    "features_names = [\"T-shirt/top\", \"Trouser\", \"Pullover\", \"Dress\", \"Coat\",\n",
    "\"Sandal\", \"Shirt\", \"Sneaker\", \"Bag\", \"Ankle boot\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 36
    },
    "id": "rXdf2ahQ-iyL",
    "outputId": "bc32fb0e-a4f3-45b8-8f67-603308196816"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Coat'"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "features_names[y_train[0]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# SEQUENTIAL API METHOD OF TRAINING"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "id": "eY4e-x2V-o8s"
   },
   "outputs": [],
   "source": [
    "model = keras.models.Sequential()\n",
    "model.add(keras.layers.Flatten(input_shape=[28, 28]))\n",
    "model.add(keras.layers.Dense(300, activation=\"relu\"))\n",
    "model.add(keras.layers.Dense(100, activation=\"relu\"))\n",
    "model.add(keras.layers.Dense(10, activation=\"softmax\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "GMI6rzdj-snV",
    "outputId": "528419f9-793b-454d-f146-3e4bd38296ea"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "flatten (Flatten)            (None, 784)               0         \n",
      "_________________________________________________________________\n",
      "dense (Dense)                (None, 300)               235500    \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 100)               30100     \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 10)                1010      \n",
      "=================================================================\n",
      "Total params: 266,610\n",
      "Trainable params: 266,610\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "id": "1OJ4hKCy-ypc"
   },
   "outputs": [],
   "source": [
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=\"sgd\", metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "4nCnf6-j-_Nw",
    "outputId": "01af3340-fce8-41e1-aa65-e3205f4a238c",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/30\n",
      "1719/1719 [==============================] - 5s 3ms/step - loss: 0.7183 - accuracy: 0.7637 - val_loss: 0.5056 - val_accuracy: 0.8316\n",
      "Epoch 2/30\n",
      "1719/1719 [==============================] - 5s 3ms/step - loss: 0.4899 - accuracy: 0.8294 - val_loss: 0.4507 - val_accuracy: 0.8480\n",
      "Epoch 3/30\n",
      "1719/1719 [==============================] - 5s 3ms/step - loss: 0.4432 - accuracy: 0.8460 - val_loss: 0.4210 - val_accuracy: 0.8548\n",
      "Epoch 4/30\n",
      "1719/1719 [==============================] - 4s 3ms/step - loss: 0.4161 - accuracy: 0.8529 - val_loss: 0.4205 - val_accuracy: 0.8540\n",
      "Epoch 5/30\n",
      "1719/1719 [==============================] - 5s 3ms/step - loss: 0.3947 - accuracy: 0.8616 - val_loss: 0.4410 - val_accuracy: 0.8418\n",
      "Epoch 6/30\n",
      "1719/1719 [==============================] - 5s 3ms/step - loss: 0.3788 - accuracy: 0.8665 - val_loss: 0.3669 - val_accuracy: 0.8746\n",
      "Epoch 7/30\n",
      "1719/1719 [==============================] - 5s 3ms/step - loss: 0.3655 - accuracy: 0.8703 - val_loss: 0.3670 - val_accuracy: 0.8696\n",
      "Epoch 8/30\n",
      "1719/1719 [==============================] - 5s 3ms/step - loss: 0.3532 - accuracy: 0.8754 - val_loss: 0.3626 - val_accuracy: 0.8692\n",
      "Epoch 9/30\n",
      "1719/1719 [==============================] - 4s 2ms/step - loss: 0.3423 - accuracy: 0.8781 - val_loss: 0.3485 - val_accuracy: 0.8732\n",
      "Epoch 10/30\n",
      "1719/1719 [==============================] - 4s 3ms/step - loss: 0.3333 - accuracy: 0.8817 - val_loss: 0.3412 - val_accuracy: 0.8772\n",
      "Epoch 11/30\n",
      "1719/1719 [==============================] - 6s 3ms/step - loss: 0.3243 - accuracy: 0.8842 - val_loss: 0.3529 - val_accuracy: 0.8702\n",
      "Epoch 12/30\n",
      "1719/1719 [==============================] - 7s 4ms/step - loss: 0.3157 - accuracy: 0.8875 - val_loss: 0.3462 - val_accuracy: 0.8740\n",
      "Epoch 13/30\n",
      "1719/1719 [==============================] - 6s 3ms/step - loss: 0.3090 - accuracy: 0.8903 - val_loss: 0.3292 - val_accuracy: 0.8818\n",
      "Epoch 14/30\n",
      "1719/1719 [==============================] - 6s 3ms/step - loss: 0.3016 - accuracy: 0.8914 - val_loss: 0.3282 - val_accuracy: 0.8846\n",
      "Epoch 15/30\n",
      "1719/1719 [==============================] - 5s 3ms/step - loss: 0.2951 - accuracy: 0.8938 - val_loss: 0.3236 - val_accuracy: 0.8804\n",
      "Epoch 16/30\n",
      "1719/1719 [==============================] - 5s 3ms/step - loss: 0.2904 - accuracy: 0.8952 - val_loss: 0.3159 - val_accuracy: 0.8878\n",
      "Epoch 17/30\n",
      "1719/1719 [==============================] - 6s 3ms/step - loss: 0.2834 - accuracy: 0.8978 - val_loss: 0.3235 - val_accuracy: 0.8820\n",
      "Epoch 18/30\n",
      "1719/1719 [==============================] - 6s 3ms/step - loss: 0.2782 - accuracy: 0.9001 - val_loss: 0.3267 - val_accuracy: 0.8832 - loss: 0.2778 - accuracy: 0. - ETA: 0s - loss: 0.2781 - accu\n",
      "Epoch 19/30\n",
      "1719/1719 [==============================] - 6s 4ms/step - loss: 0.2729 - accuracy: 0.9011 - val_loss: 0.3143 - val_accuracy: 0.8840\n",
      "Epoch 20/30\n",
      "1719/1719 [==============================] - 6s 3ms/step - loss: 0.2669 - accuracy: 0.9045 - val_loss: 0.3197 - val_accuracy: 0.8836\n",
      "Epoch 21/30\n",
      "1719/1719 [==============================] - 5s 3ms/step - loss: 0.2632 - accuracy: 0.9052 - val_loss: 0.3103 - val_accuracy: 0.8870\n",
      "Epoch 22/30\n",
      "1719/1719 [==============================] - 6s 3ms/step - loss: 0.2586 - accuracy: 0.9063 - val_loss: 0.3174 - val_accuracy: 0.8812\n",
      "Epoch 23/30\n",
      "1719/1719 [==============================] - 6s 3ms/step - loss: 0.2530 - accuracy: 0.9086 - val_loss: 0.3055 - val_accuracy: 0.8902\n",
      "Epoch 24/30\n",
      "1719/1719 [==============================] - 6s 3ms/step - loss: 0.2481 - accuracy: 0.9104 - val_loss: 0.3303 - val_accuracy: 0.8778\n",
      "Epoch 25/30\n",
      "1719/1719 [==============================] - 6s 3ms/step - loss: 0.2446 - accuracy: 0.9119 - val_loss: 0.3062 - val_accuracy: 0.8876\n",
      "Epoch 26/30\n",
      "1719/1719 [==============================] - 5s 3ms/step - loss: 0.2400 - accuracy: 0.9139 - val_loss: 0.3045 - val_accuracy: 0.8882\n",
      "Epoch 27/30\n",
      "1719/1719 [==============================] - 6s 4ms/step - loss: 0.2364 - accuracy: 0.9141 - val_loss: 0.3015 - val_accuracy: 0.8872\n",
      "Epoch 28/30\n",
      "1719/1719 [==============================] - 6s 3ms/step - loss: 0.2325 - accuracy: 0.9163 - val_loss: 0.2941 - val_accuracy: 0.8906\n",
      "Epoch 29/30\n",
      "1719/1719 [==============================] - 6s 3ms/step - loss: 0.2284 - accuracy: 0.9178 - val_loss: 0.3039 - val_accuracy: 0.8900\n",
      "Epoch 30/30\n",
      "1719/1719 [==============================] - 6s 3ms/step - loss: 0.2256 - accuracy: 0.9192 - val_loss: 0.3173 - val_accuracy: 0.8876\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(X_train, y_train, epochs=30, validation_data=(X_valid, y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Here We have put batch size to 8"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n",
      "6875/6875 [==============================] - 12s 2ms/step - loss: 0.5518 - accuracy: 0.8083\n",
      "Epoch 2/20\n",
      "6875/6875 [==============================] - 12s 2ms/step - loss: 0.4061 - accuracy: 0.8534\n",
      "Epoch 3/20\n",
      "6875/6875 [==============================] - 13s 2ms/step - loss: 0.3622 - accuracy: 0.8673\n",
      "Epoch 4/20\n",
      "6875/6875 [==============================] - 12s 2ms/step - loss: 0.3351 - accuracy: 0.8770\n",
      "Epoch 5/20\n",
      "6875/6875 [==============================] - 16s 2ms/step - loss: 0.3166 - accuracy: 0.8841\n",
      "Epoch 6/20\n",
      "6875/6875 [==============================] - 18s 3ms/step - loss: 0.2997 - accuracy: 0.8893 0s - loss: 0\n",
      "Epoch 7/20\n",
      "6875/6875 [==============================] - 16s 2ms/step - loss: 0.2867 - accuracy: 0.8949\n",
      "Epoch 8/20\n",
      "6875/6875 [==============================] - 14s 2ms/step - loss: 0.2748 - accuracy: 0.9001\n",
      "Epoch 9/20\n",
      "6875/6875 [==============================] - 16s 2ms/step - loss: 0.2644 - accuracy: 0.9014\n",
      "Epoch 10/20\n",
      "6875/6875 [==============================] - 14s 2ms/step - loss: 0.2543 - accuracy: 0.9063\n",
      "Epoch 11/20\n",
      "6875/6875 [==============================] - 16s 2ms/step - loss: 0.2450 - accuracy: 0.9098\n",
      "Epoch 12/20\n",
      "6875/6875 [==============================] - 13s 2ms/step - loss: 0.2382 - accuracy: 0.9115\n",
      "Epoch 13/20\n",
      "6875/6875 [==============================] - 13s 2ms/step - loss: 0.2303 - accuracy: 0.9141\n",
      "Epoch 14/20\n",
      "6875/6875 [==============================] - 12s 2ms/step - loss: 0.2236 - accuracy: 0.9165\n",
      "Epoch 15/20\n",
      "6875/6875 [==============================] - 13s 2ms/step - loss: 0.2163 - accuracy: 0.9202\n",
      "Epoch 16/20\n",
      "6875/6875 [==============================] - 12s 2ms/step - loss: 0.2096 - accuracy: 0.9222\n",
      "Epoch 17/20\n",
      "6875/6875 [==============================] - 13s 2ms/step - loss: 0.2036 - accuracy: 0.9247\n",
      "Epoch 18/20\n",
      "6875/6875 [==============================] - 12s 2ms/step - loss: 0.1997 - accuracy: 0.9250\n",
      "Epoch 19/20\n",
      "6875/6875 [==============================] - 13s 2ms/step - loss: 0.1923 - accuracy: 0.9284\n",
      "Epoch 20/20\n",
      "6875/6875 [==============================] - 12s 2ms/step - loss: 0.1880 - accuracy: 0.9303\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(X_train, y_train, epochs=20, batch_size=8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "313/313 [==============================] - 1s 2ms/step - loss: 86.6045 - accuracy: 0.8272\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[86.6044921875, 0.8271999955177307]"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.evaluate(X_test, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Batch Normalization in sequential api"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.models.Sequential()\n",
    "model.add(keras.layers.Flatten(input_shape=[28, 28]))\n",
    "model.add(keras.layers.Dense(300, activation=\"relu\"))\n",
    "model.add(BatchNormalization())\n",
    "model.add(keras.layers.Dense(100, activation=\"relu\"))\n",
    "model.add(BatchNormalization())\n",
    "model.add(keras.layers.Dense(10, activation=\"softmax\"))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_3\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "flatten_3 (Flatten)          (None, 784)               0         \n",
      "_________________________________________________________________\n",
      "dense_9 (Dense)              (None, 300)               235500    \n",
      "_________________________________________________________________\n",
      "batch_normalization_5 (Batch (None, 300)               1200      \n",
      "_________________________________________________________________\n",
      "dense_10 (Dense)             (None, 100)               30100     \n",
      "_________________________________________________________________\n",
      "batch_normalization_6 (Batch (None, 100)               400       \n",
      "_________________________________________________________________\n",
      "dense_11 (Dense)             (None, 10)                1010      \n",
      "=================================================================\n",
      "Total params: 268,210\n",
      "Trainable params: 267,410\n",
      "Non-trainable params: 800\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=\"adam\", metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "6875/6875 [==============================] - 18s 3ms/step - loss: 0.6188 - accuracy: 0.7845\n",
      "Epoch 2/10\n",
      "6875/6875 [==============================] - 18s 3ms/step - loss: 0.5041 - accuracy: 0.8201\n",
      "Epoch 3/10\n",
      "6875/6875 [==============================] - 21s 3ms/step - loss: 0.4663 - accuracy: 0.8354\n",
      "Epoch 4/10\n",
      "6875/6875 [==============================] - 22s 3ms/step - loss: 0.4369 - accuracy: 0.8439\n",
      "Epoch 5/10\n",
      "6875/6875 [==============================] - 18s 3ms/step - loss: 0.4156 - accuracy: 0.8490\n",
      "Epoch 6/10\n",
      "6875/6875 [==============================] - ETA: 0s - loss: 0.4119 - accuracy: 0.85 - 19s 3ms/step - loss: 0.4119 - accuracy: 0.8507\n",
      "Epoch 7/10\n",
      "6875/6875 [==============================] - 19s 3ms/step - loss: 0.3990 - accuracy: 0.8561\n",
      "Epoch 8/10\n",
      "6875/6875 [==============================] - 18s 3ms/step - loss: 0.3955 - accuracy: 0.8574 0s -\n",
      "Epoch 9/10\n",
      "6875/6875 [==============================] - 18s 3ms/step - loss: 0.3810 - accuracy: 0.8609\n",
      "Epoch 10/10\n",
      "6875/6875 [==============================] - 18s 3ms/step - loss: 0.3724 - accuracy: 0.8641\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(X_train, y_train, epochs=10, batch_size=8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "313/313 [==============================] - 1s 2ms/step - loss: 66.9301 - accuracy: 0.8591\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[66.9300765991211, 0.8590999841690063]"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.evaluate(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Adding Dropout layers in Sequential API"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.models.Sequential()\n",
    "model.add(keras.layers.Flatten(input_shape=[28, 28]))\n",
    "model.add(keras.layers.Dense(300, activation=\"relu\"))\n",
    "keras.layers.Dropout(0.5)\n",
    "model.add(keras.layers.Dense(100, activation=\"relu\"))\n",
    "keras.layers.Dropout(0.5)\n",
    "model.add(keras.layers.Dense(10, activation=\"softmax\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_5\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "flatten_5 (Flatten)          (None, 784)               0         \n",
      "_________________________________________________________________\n",
      "dense_15 (Dense)             (None, 300)               235500    \n",
      "_________________________________________________________________\n",
      "dense_16 (Dense)             (None, 100)               30100     \n",
      "_________________________________________________________________\n",
      "dense_17 (Dense)             (None, 10)                1010      \n",
      "=================================================================\n",
      "Total params: 266,610\n",
      "Trainable params: 266,610\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=\"adam\", metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "6875/6875 [==============================] - 16s 2ms/step - loss: 0.4865 - accuracy: 0.8222\n",
      "Epoch 2/10\n",
      "6875/6875 [==============================] - 15s 2ms/step - loss: 0.3713 - accuracy: 0.8625\n",
      "Epoch 3/10\n",
      "6875/6875 [==============================] - 15s 2ms/step - loss: 0.3364 - accuracy: 0.8741\n",
      "Epoch 4/10\n",
      "6875/6875 [==============================] - 15s 2ms/step - loss: 0.3139 - accuracy: 0.8831\n",
      "Epoch 5/10\n",
      "6875/6875 [==============================] - 15s 2ms/step - loss: 0.2974 - accuracy: 0.8895\n",
      "Epoch 6/10\n",
      "6875/6875 [==============================] - 15s 2ms/step - loss: 0.2852 - accuracy: 0.8935\n",
      "Epoch 7/10\n",
      "6875/6875 [==============================] - 15s 2ms/step - loss: 0.2748 - accuracy: 0.8967\n",
      "Epoch 8/10\n",
      "6875/6875 [==============================] - 15s 2ms/step - loss: 0.2692 - accuracy: 0.9006\n",
      "Epoch 9/10\n",
      "6875/6875 [==============================] - 16s 2ms/step - loss: 0.2574 - accuracy: 0.9042\n",
      "Epoch 10/10\n",
      "6875/6875 [==============================] - 16s 2ms/step - loss: 0.2509 - accuracy: 0.9056\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(X_train, y_train, epochs=10, batch_size=8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "rAmeSb8WBOS0",
    "outputId": "15d2b691-c274-46e0-f2f9-3cad4c97478a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "313/313 [==============================] - 1s 3ms/step - loss: 91.6958 - accuracy: 0.8145\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[91.69583129882812, 0.8144999742507935]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.evaluate(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# functional api method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_ = keras.layers.Input(shape=[28, 28])\n",
    "flatten = keras.layers.Flatten(input_shape=[28, 28])(input_)\n",
    "hidden1 = keras.layers.Dense(2**14, activation=\"relu\")(flatten)\n",
    "hidden2 = keras.layers.Dense(512, activation='relu')(hidden1)\n",
    "hidden3 = keras.layers.Dense(28*28, activation='relu')(hidden2)\n",
    "reshap = keras.layers.Reshape((28, 28))(hidden3)\n",
    "concat_ = keras.layers.Concatenate()([input_, reshap])\n",
    "flatten2 = keras.layers.Flatten(input_shape=[28, 28])(concat_)\n",
    "output = keras.layers.Dense(10, activation='softmax')(flatten2)\n",
    "model = keras.Model(inputs=[input_], outputs=[output] )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"functional_5\"\n",
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "input_4 (InputLayer)            [(None, 28, 28)]     0                                            \n",
      "__________________________________________________________________________________________________\n",
      "flatten_2 (Flatten)             (None, 784)          0           input_4[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "dense_14 (Dense)                (None, 16384)        12861440    flatten_2[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "dense_15 (Dense)                (None, 512)          8389120     dense_14[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "dense_16 (Dense)                (None, 784)          402192      dense_15[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "reshape (Reshape)               (None, 28, 28)       0           dense_16[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "concatenate_3 (Concatenate)     (None, 28, 56)       0           input_4[0][0]                    \n",
      "                                                                 reshape[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "flatten_3 (Flatten)             (None, 1568)         0           concatenate_3[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "dense_17 (Dense)                (None, 10)           15690       flatten_3[0][0]                  \n",
      "==================================================================================================\n",
      "Total params: 21,668,442\n",
      "Trainable params: 21,668,442\n",
      "Non-trainable params: 0\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/8\n",
      "6875/6875 [==============================] - 819s 119ms/step - loss: 0.4732 - accuracy: 0.8354\n",
      "Epoch 2/8\n",
      "6875/6875 [==============================] - 800s 116ms/step - loss: 0.4386 - accuracy: 0.8459\n",
      "Epoch 3/8\n",
      "6875/6875 [==============================] - 874s 127ms/step - loss: 0.4263 - accuracy: 0.8512\n",
      "Epoch 4/8\n",
      "6875/6875 [==============================] - 810s 118ms/step - loss: 0.4169 - accuracy: 0.8557\n",
      "Epoch 5/8\n",
      "6875/6875 [==============================] - 776s 113ms/step - loss: 0.4120 - accuracy: 0.8559\n",
      "Epoch 6/8\n",
      "6875/6875 [==============================] - 775s 113ms/step - loss: 0.4071 - accuracy: 0.8590\n",
      "Epoch 7/8\n",
      "6875/6875 [==============================] - 775s 113ms/step - loss: 0.4049 - accuracy: 0.8590\n",
      "Epoch 8/8\n",
      "6875/6875 [==============================] - 763s 111ms/step - loss: 0.4005 - accuracy: 0.8608\n"
     ]
    }
   ],
   "source": [
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=\"adam\",\n",
    "              metrics=[\"accuracy\"])\n",
    "h = model.fit(X_train, y_train, epochs=8, batch_size=8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Here we have done batch normalization to functional api"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train_full=X_train_full.reshape(60000,784)/255.0\n",
    "\n",
    "testdata=X_test.reshape(10000,784)/255.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "Layer1=keras.layers.Input(shape=(784,))\n",
    "Layer2=keras.layers.Dense(80,activation=\"relu\")(Layer1)\n",
    "Layer3=keras.layers.Dense(320,activation=\"relu\")(Layer2)\n",
    "concat=keras.layers.Concatenate()([Layer1,Layer2])\n",
    "\n",
    "BN=keras.layers.BatchNormalization()(concat)\n",
    "op=keras.layers.Dense(10,activation=\"softmax\")(BN)\n",
    "\n",
    "model=keras.Model(inputs=[Layer1],outputs=[op])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"functional_1\"\n",
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "input_1 (InputLayer)            [(None, 784)]        0                                            \n",
      "__________________________________________________________________________________________________\n",
      "dense (Dense)                   (None, 80)           62800       input_1[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "concatenate (Concatenate)       (None, 864)          0           input_1[0][0]                    \n",
      "                                                                 dense[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization (BatchNorma (None, 864)          3456        concatenate[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "dense_2 (Dense)                 (None, 10)           8650        batch_normalization[0][0]        \n",
      "==================================================================================================\n",
      "Total params: 74,906\n",
      "Trainable params: 73,178\n",
      "Non-trainable params: 1,728\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n",
      "1125/1125 [==============================] - 5s 5ms/step - loss: 0.2933 - accuracy: 0.8903 - val_loss: 0.3461 - val_accuracy: 0.8782\n",
      "Epoch 2/20\n",
      "1125/1125 [==============================] - 4s 4ms/step - loss: 0.2899 - accuracy: 0.8915 - val_loss: 0.3439 - val_accuracy: 0.8802\n",
      "Epoch 3/20\n",
      "1125/1125 [==============================] - 4s 4ms/step - loss: 0.2897 - accuracy: 0.8925 - val_loss: 0.3463 - val_accuracy: 0.8787\n",
      "Epoch 4/20\n",
      "1125/1125 [==============================] - 4s 4ms/step - loss: 0.2851 - accuracy: 0.8933 - val_loss: 0.3804 - val_accuracy: 0.8641\n",
      "Epoch 5/20\n",
      "1125/1125 [==============================] - 4s 4ms/step - loss: 0.2849 - accuracy: 0.8926 - val_loss: 0.3642 - val_accuracy: 0.8745\n",
      "Epoch 6/20\n",
      "1125/1125 [==============================] - 4s 4ms/step - loss: 0.2816 - accuracy: 0.8946 - val_loss: 0.3445 - val_accuracy: 0.8784\n",
      "Epoch 7/20\n",
      "1125/1125 [==============================] - 4s 4ms/step - loss: 0.2813 - accuracy: 0.8944 - val_loss: 0.3707 - val_accuracy: 0.8712\n",
      "Epoch 8/20\n",
      "1125/1125 [==============================] - 4s 4ms/step - loss: 0.2795 - accuracy: 0.8962 - val_loss: 0.3787 - val_accuracy: 0.8657\n",
      "Epoch 9/20\n",
      "1125/1125 [==============================] - 4s 4ms/step - loss: 0.2788 - accuracy: 0.8964 - val_loss: 0.3628 - val_accuracy: 0.8740\n",
      "Epoch 10/20\n",
      "1125/1125 [==============================] - 4s 4ms/step - loss: 0.2763 - accuracy: 0.8970 - val_loss: 0.3569 - val_accuracy: 0.8783\n",
      "Epoch 11/20\n",
      "1125/1125 [==============================] - 4s 4ms/step - loss: 0.2740 - accuracy: 0.8984 - val_loss: 0.3881 - val_accuracy: 0.8675\n",
      "Epoch 12/20\n",
      "1125/1125 [==============================] - 4s 4ms/step - loss: 0.2723 - accuracy: 0.8967 - val_loss: 0.4398 - val_accuracy: 0.8543\n",
      "Epoch 13/20\n",
      "1125/1125 [==============================] - 5s 4ms/step - loss: 0.2720 - accuracy: 0.8983 - val_loss: 0.3717 - val_accuracy: 0.8740\n",
      "Epoch 14/20\n",
      "1125/1125 [==============================] - 4s 4ms/step - loss: 0.2679 - accuracy: 0.9001 - val_loss: 0.3954 - val_accuracy: 0.8700\n",
      "Epoch 15/20\n",
      "1125/1125 [==============================] - 4s 4ms/step - loss: 0.2684 - accuracy: 0.8996 - val_loss: 0.4062 - val_accuracy: 0.8628\n",
      "Epoch 16/20\n",
      "1125/1125 [==============================] - 4s 4ms/step - loss: 0.2692 - accuracy: 0.8978 - val_loss: 0.3639 - val_accuracy: 0.8773\n",
      "Epoch 17/20\n",
      "1125/1125 [==============================] - 4s 4ms/step - loss: 0.2680 - accuracy: 0.8996 - val_loss: 0.3666 - val_accuracy: 0.8791\n",
      "Epoch 18/20\n",
      "1125/1125 [==============================] - 4s 4ms/step - loss: 0.2639 - accuracy: 0.9010 - val_loss: 0.3639 - val_accuracy: 0.8764\n",
      "Epoch 19/20\n",
      "1125/1125 [==============================] - 4s 4ms/step - loss: 0.2633 - accuracy: 0.9008 - val_loss: 0.3682 - val_accuracy: 0.8763\n",
      "Epoch 20/20\n",
      "1125/1125 [==============================] - 4s 4ms/step - loss: 0.2623 - accuracy: 0.9021 - val_loss: 0.3779 - val_accuracy: 0.8692\n"
     ]
    }
   ],
   "source": [
    "model.compile(loss=\"sparse_categorical_crossentropy\",optimizer=\"adam\",metrics=[\"accuracy\"])\n",
    "\n",
    "opdata=model.fit(X_train_full,y_train_full,epochs=20,validation_split=0.40)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Here we performed both batch normalization and dropout on the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "Layer1=keras.layers.Input(shape=(784,))\n",
    "Layer2=keras.layers.Dense(80,activation=\"relu\")(Layer1)\n",
    "Layer3=keras.layers.Dense(320,activation=\"relu\")(Layer2)\n",
    "concat=keras.layers.Concatenate()([Layer1,Layer2])\n",
    "\n",
    "BN=keras.layers.BatchNormalization()(concat)\n",
    "\n",
    "Drop=keras.layers.Dropout(0.5)(BN)\n",
    "op=keras.layers.Dense(10,activation=\"softmax\")(Drop)\n",
    "model=keras.Model(inputs=[Layer1],outputs=[op])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"functional_5\"\n",
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "input_3 (InputLayer)            [(None, 784)]        0                                            \n",
      "__________________________________________________________________________________________________\n",
      "dense_6 (Dense)                 (None, 80)           62800       input_3[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "concatenate_2 (Concatenate)     (None, 864)          0           input_3[0][0]                    \n",
      "                                                                 dense_6[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_2 (BatchNor (None, 864)          3456        concatenate_2[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "dropout (Dropout)               (None, 864)          0           batch_normalization_2[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "dense_8 (Dense)                 (None, 10)           8650        dropout[0][0]                    \n",
      "==================================================================================================\n",
      "Total params: 74,906\n",
      "Trainable params: 73,178\n",
      "Non-trainable params: 1,728\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n",
      "1313/1313 [==============================] - 6s 5ms/step - loss: 0.7367 - accuracy: 0.7554 - val_loss: 0.5018 - val_accuracy: 0.8216\n",
      "Epoch 2/20\n",
      "1313/1313 [==============================] - 6s 4ms/step - loss: 0.5250 - accuracy: 0.8170 - val_loss: 0.4677 - val_accuracy: 0.8320\n",
      "Epoch 3/20\n",
      "1313/1313 [==============================] - 6s 4ms/step - loss: 0.4921 - accuracy: 0.8275 - val_loss: 0.4347 - val_accuracy: 0.8477\n",
      "Epoch 4/20\n",
      "1313/1313 [==============================] - 6s 4ms/step - loss: 0.4746 - accuracy: 0.8340 - val_loss: 0.4357 - val_accuracy: 0.8441\n",
      "Epoch 5/20\n",
      "1313/1313 [==============================] - 6s 4ms/step - loss: 0.4658 - accuracy: 0.8365 - val_loss: 0.4346 - val_accuracy: 0.8451\n",
      "Epoch 6/20\n",
      "1313/1313 [==============================] - 6s 5ms/step - loss: 0.4554 - accuracy: 0.8395 - val_loss: 0.4225 - val_accuracy: 0.8511\n",
      "Epoch 7/20\n",
      "1313/1313 [==============================] - 6s 4ms/step - loss: 0.4534 - accuracy: 0.8386 - val_loss: 0.4160 - val_accuracy: 0.8518\n",
      "Epoch 8/20\n",
      "1313/1313 [==============================] - 6s 4ms/step - loss: 0.4452 - accuracy: 0.8434 - val_loss: 0.4046 - val_accuracy: 0.8582\n",
      "Epoch 9/20\n",
      "1313/1313 [==============================] - 6s 5ms/step - loss: 0.4411 - accuracy: 0.8452 - val_loss: 0.4089 - val_accuracy: 0.8558\n",
      "Epoch 10/20\n",
      "1313/1313 [==============================] - 6s 4ms/step - loss: 0.4348 - accuracy: 0.8474 - val_loss: 0.4006 - val_accuracy: 0.8570\n",
      "Epoch 11/20\n",
      "1313/1313 [==============================] - 6s 4ms/step - loss: 0.4374 - accuracy: 0.8467 - val_loss: 0.3997 - val_accuracy: 0.8576\n",
      "Epoch 12/20\n",
      "1313/1313 [==============================] - 6s 4ms/step - loss: 0.4318 - accuracy: 0.8482 - val_loss: 0.4114 - val_accuracy: 0.8532\n",
      "Epoch 13/20\n",
      "1313/1313 [==============================] - 6s 4ms/step - loss: 0.4307 - accuracy: 0.8470 - val_loss: 0.4000 - val_accuracy: 0.8577\n",
      "Epoch 14/20\n",
      "1313/1313 [==============================] - 6s 4ms/step - loss: 0.4266 - accuracy: 0.8495 - val_loss: 0.3927 - val_accuracy: 0.8617\n",
      "Epoch 15/20\n",
      "1313/1313 [==============================] - 6s 4ms/step - loss: 0.4252 - accuracy: 0.8515 - val_loss: 0.3908 - val_accuracy: 0.8603\n",
      "Epoch 16/20\n",
      "1313/1313 [==============================] - 6s 5ms/step - loss: 0.4213 - accuracy: 0.8497 - val_loss: 0.4207 - val_accuracy: 0.8464\n",
      "Epoch 17/20\n",
      "1313/1313 [==============================] - 6s 4ms/step - loss: 0.4174 - accuracy: 0.8523 - val_loss: 0.3925 - val_accuracy: 0.8611\n",
      "Epoch 18/20\n",
      "1313/1313 [==============================] - 6s 4ms/step - loss: 0.4169 - accuracy: 0.8506 - val_loss: 0.3833 - val_accuracy: 0.8630\n",
      "Epoch 19/20\n",
      "1313/1313 [==============================] - 6s 4ms/step - loss: 0.4192 - accuracy: 0.8510 - val_loss: 0.3808 - val_accuracy: 0.8641\n",
      "Epoch 20/20\n",
      "1313/1313 [==============================] - 6s 5ms/step - loss: 0.4124 - accuracy: 0.8530 - val_loss: 0.3848 - val_accuracy: 0.8639\n"
     ]
    }
   ],
   "source": [
    "model.compile(loss=\"sparse_categorical_crossentropy\",optimizer=\"adam\",metrics=[\"accuracy\"])\n",
    "\n",
    "opdata=model.fit(X_train_full,y_train_full,epochs=20,validation_split=0.30)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "PROCEDURE:\n",
    "\n",
    "1.Here we are taking the MNIST fashion data set and using tensorflow libraries.\n",
    "\n",
    "Tensorflow used for:\n",
    "It is more powerful than the sequential API in the sense branching or sharing of layers is allowed here. And also it can have multiple inputs and outputs\n",
    "\n",
    "Dataset:\n",
    "Fashion-MNIST is a dataset of Zalando's article images—consisting of a training set of 60,000 examples and a test set of 10,000 examples. Each example is a 28x28 grayscale image, associated with a label from 10 classes. Zalando intends Fashion-MNIST to serve as a direct drop-in replacement for the original MNIST dataset for benchmarking machine learning algorithms. It shares the same image size and structure of training and testing splits.\n",
    "\n",
    "2.There are two methods in training the DEEP LEARNING MODEL ,those are sequential method  and functional api\n",
    "\n",
    "3.In sequential , we cannot have multiple input at a time , it is designed layer by layer\n",
    "\n",
    "4.First we did batchnormalization and dropout layering for sequential methods, we got accuracy of 86 and 90 % respectively,\n",
    "\n",
    "5.Functional api : It is more advanced than the sequential API in the sense branching or sharing of layers is allowed here and also it can have multiple inputs and outputs.\n",
    "\n",
    "Optimizer:Here we used Adam optimizer, it is a replacement optimization algorithm for stochastic gradient descent for training deep learning models. \n",
    "\n",
    "6.After adding the batchnormaliation and drop out layers , we compile the model and fit the model to run epochs with set limit.\n",
    "\n",
    "\n",
    "AActivation function: it defines the what node/nueron has to do .\n",
    "\n",
    "\n",
    "7.So after the fit we got 90.8 % and 87% accuracy in functional api.\n",
    "\n",
    "\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "name": "NeuN_Part_I.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}