NLP-with NLTK.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction\n",
    "Natural Language Processing (NLP) is the task of making computers understand and produce human languages."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# What is a Corpus?\n",
    "There are many corpora (plural of corpus) available in NLTK, lets start with an English one call the Brown corpus.\n",
    "\n",
    "When using a new corpus in NLTK for the first time, downloads the corpus with the nltk.download()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package brown to\n",
      "[nltk_data]     C:\\Users\\500062812\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package brown is already up-to-date!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import nltk\n",
    "nltk.download('brown')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.corpus import brown"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['The', 'Fulton', 'County', 'Grand', 'Jury', 'said', ...]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "brown.words() # Returns a list of strings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1161192"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(brown.words()) # No. of words in the corpus"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['The', 'Fulton', 'County', 'Grand', 'Jury', 'said', 'Friday', 'an', 'investigation', 'of', \"Atlanta's\", 'recent', 'primary', 'election', 'produced', '``', 'no', 'evidence', \"''\", 'that', 'any', 'irregularities', 'took', 'place', '.'], ['The', 'jury', 'further', 'said', 'in', 'term-end', 'presentments', 'that', 'the', 'City', 'Executive', 'Committee', ',', 'which', 'had', 'over-all', 'charge', 'of', 'the', 'election', ',', '``', 'deserves', 'the', 'praise', 'and', 'thanks', 'of', 'the', 'City', 'of', 'Atlanta', \"''\", 'for', 'the', 'manner', 'in', 'which', 'the', 'election', 'was', 'conducted', '.'], ...]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "brown.sents() # Returns a list of list of strings "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['ca01', 'ca02', 'ca03', 'ca04', 'ca05', 'ca06', 'ca07', 'ca08', 'ca09', 'ca10', 'ca11', 'ca12', 'ca13', 'ca14', 'ca15', 'ca16', 'ca17', 'ca18', 'ca19', 'ca20', 'ca21', 'ca22', 'ca23', 'ca24', 'ca25', 'ca26', 'ca27', 'ca28', 'ca29', 'ca30', 'ca31', 'ca32', 'ca33', 'ca34', 'ca35', 'ca36', 'ca37', 'ca38', 'ca39', 'ca40', 'ca41', 'ca42', 'ca43', 'ca44', 'cb01', 'cb02', 'cb03', 'cb04', 'cb05', 'cb06', 'cb07', 'cb08', 'cb09', 'cb10', 'cb11', 'cb12', 'cb13', 'cb14', 'cb15', 'cb16', 'cb17', 'cb18', 'cb19', 'cb20', 'cb21', 'cb22', 'cb23', 'cb24', 'cb25', 'cb26', 'cb27', 'cc01', 'cc02', 'cc03', 'cc04', 'cc05', 'cc06', 'cc07', 'cc08', 'cc09', 'cc10', 'cc11', 'cc12', 'cc13', 'cc14', 'cc15', 'cc16', 'cc17', 'cd01', 'cd02', 'cd03', 'cd04', 'cd05', 'cd06', 'cd07', 'cd08', 'cd09', 'cd10', 'cd11', 'cd12']\n"
     ]
    }
   ],
   "source": [
    "print(brown.fileids()[:100]) # First 100 sources."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['The', 'Fulton', 'County', 'Grand', 'Jury', 'said', 'Friday', 'an', 'investigation', 'of', \"Atlanta's\", 'recent', 'primary', 'election', 'produced', '``', 'no', 'evidence', \"''\", 'that', 'any', 'irregularities', 'took', 'place', '.'], ['The', 'jury', 'further', 'said', 'in', 'term-end', 'presentments', 'that', 'the', 'City', 'Executive', 'Committee', ',', 'which', 'had', 'over-all', 'charge', 'of', 'the', 'election', ',', '``', 'deserves', 'the', 'praise', 'and', 'thanks', 'of', 'the', 'City', 'of', 'Atlanta', \"''\", 'for', 'the', 'manner', 'in', 'which', 'the', 'election', 'was', 'conducted', '.'], ...]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "brown.sents(fileids='ca01') # You can access a specific file with `fileids` argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['Austin', ',', 'Texas'], ['--', 'Committee', 'approval', 'of', 'Gov.', 'Price', \"Daniel's\", '``', 'abandoned', 'property', \"''\", 'act', 'seemed', 'certain', 'Thursday', 'despite', 'the', 'adamant', 'protests', 'of', 'Texas', 'bankers', '.'], ...]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "brown.sents(fileids='ca02') # You can access a specific file with `fileids` argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['Several', 'defendants', 'in', 'the', 'Summerdale', 'police', 'burglary', 'trial', 'made', 'statements', 'indicating', 'their', 'guilt', 'at', 'the', 'time', 'of', 'their', 'arrest', ',', 'Judge', 'James', 'B.', 'Parsons', 'was', 'told', 'in', 'Criminal', 'court', 'yesterday', '.'], ['The', 'disclosure', 'by', 'Charles', 'Bellows', ',', 'chief', 'defense', 'counsel', ',', 'startled', 'observers', 'and', 'was', 'viewed', 'as', 'the', 'prelude', 'to', 'a', 'quarrel', 'between', 'the', 'six', 'attorneys', 'representing', 'the', 'eight', 'former', 'policemen', 'now', 'on', 'trial', '.'], ...]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "brown.sents(fileids='ca03') # You can access a specific file with `fileids` argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "500"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(brown.fileids()) # 500 sources, each file is a source."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Assembly/nn-hl session/nn-hl brought/vbd-hl much/ap-hl good/nn-hl \n",
      "The/at General/jj-tl Assembly/nn-tl ,/, which/wdt adjourns/vbz today/nr ,/, has/hvz performed/vbn in/in an/at atmosphere/nn of/in crisis/nn and/cc struggle/nn from/in the/at day/nn it/pps convened/vbd ./.\n",
      "It/pps was/bedz faced/vbn immediately/rb with/in a/at showdown/nn on/in the/at schools/nns ,/, an/at issue/nn which/wdt was/bedz met/vbn squarely/rb in/in conjunction/nn with/in the/at governor/nn with/in a/at decision/nn not/* to/to risk/vb abandoning/vbg public/nn education/nn ./.\n",
      "\n",
      "\n",
      "\tThere/ex followed/vbd the/at historic/jj appropriations/nns and/cc budget/nn fight/nn ,/, in/in which/wdt the/at General/jj-tl Assembly/nn-tl decided/vbd to/to tackle/vb executive/nn powers/nns ./.\n",
      "The/at final/jj decision/nn went/vbd to/in the/at executive/nn but/cc a/at way/nn has/hvz been/ben opened/vbn for/in strengthening/vbg budgeting/vbg procedures/nns and/cc to/to provide/vb legislators/nns information/nn they/ppss need/vb ./.\n",
      "\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(brown.raw('cb01').strip()[:1000]) # First 1000 characters."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You will see that each word comes with a slash and a label and unlike normal text, we see that punctuations are separated from the word that comes before it,"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we also see that the each sentence is separated by a newline:\n",
    "    \n",
    "That brings us to the next point on sentence tokenization and word tokenization."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tokenization\n",
    "\n",
    "Sentence tokenization is the process of splitting up strings into “sentences”\n",
    "Word tokenization is the process of splitting up “sentences” into “words”\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, downoad the data with nltk.download():"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package webtext to\n",
      "[nltk_data]     C:\\Users\\500062812\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package webtext is already up-to-date!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nltk.download('webtext')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.corpus import webtext"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['firefox.txt',\n",
       " 'grail.txt',\n",
       " 'overheard.txt',\n",
       " 'pirates.txt',\n",
       " 'singles.txt',\n",
       " 'wine.txt']"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "webtext.fileids()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0:\t25 SEXY MALE, seeks attrac older single lady, for discreet encounters.\n",
      "1:\t35YO Security Guard, seeking lady in uniform for fun times.\n",
      "2:\t40 yo SINGLE DAD, sincere friendly DTE seeks r/ship with fem age open S/E\n",
      "3:\t44yo tall seeks working single mum or lady below 45 fship rship. Nat Open\n",
      "4:\t6.2 35 yr old OUTGOING M seeks fem 28-35 for o/door sports - w/e away\n",
      "5:\tA professional business male, late 40s, 6 feet tall, slim build, well groomed, great personality, home owner, interests include the arts travel and all things good, Ringwood area, is seeking a genuine female of similar age or older, in same area or surrounds, for a meaningful long term rship. Looking forward to hearing from you all.\n",
      "6:\tABLE young man seeks, sexy older women. Phone for fun ready to play\n",
      "7:\tAFFECTIONATE LADY Sought by generous guy, 40s, mutual fulfillment\n",
      "8:\tARE YOU ALONE or lost in a r/ship too, with no hope in sight? Maybe we could explore new beginnings together? Im 45 Slim/Med build, GSOH, high needs and looking for someone similar. You WONT be disappointed.\n",
      "9:\tAMIABLE 43 y.o. gentleman with European background, 170 cm, medium build, employed, never married, no children. Enjoys sports, music, cafes, beach &c. Seeks an honest, attractive lady with a European background, without children, who would like to get married and have chil dren in the future. 29-39 y.o. Prefer non-smoker and living in Adelaide.\n",
      "10:\tARE YOU A COPPER REDHEAD? I am 36 y.o. and looking for companionship/friendship. I enjoy the AFL, animals and dining out.\n"
     ]
    }
   ],
   "source": [
    "# Each line is one advertisement.\n",
    "for i, line in enumerate(webtext.raw('singles.txt').split('\\n')):\n",
    "    if i > 10: # Lets take a look at the first 10 ads.\n",
    "        break\n",
    "    print(str(i) + ':\\t' + line)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lets zoom in on candidate no. 8"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ARE YOU ALONE or lost in a r/ship too, with no hope in sight? Maybe we could explore new beginnings together? Im 45 Slim/Med build, GSOH, high needs and looking for someone similar. You WONT be disappointed.\n"
     ]
    }
   ],
   "source": [
    "single_no8 = webtext.raw('singles.txt').split('\\n')[8]\n",
    "print(single_no8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sentence Tokenization\n",
    "\n",
    "In NLTK, sent_tokenize() the default tokenizer function that you can use to split strings into \"sentences\". "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk import sent_tokenize, word_tokenize"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt to\n",
      "[nltk_data]     C:\\Users\\500062812\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package punkt is already up-to-date!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nltk.download('punkt')#It is using the Punkt algortihm from Kiss and Strunk (2006)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['ARE YOU ALONE or lost in a r/ship too, with no hope in sight?',\n",
       " 'Maybe we could explore new beginnings together?',\n",
       " 'Im 45 Slim/Med build, GSOH, high needs and looking for someone similar.',\n",
       " 'You WONT be disappointed.']"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sent_tokenize(single_no8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['ARE', 'YOU', 'ALONE', 'or', 'lost', 'in', 'a', 'r/ship', 'too', ',', 'with', 'no', 'hope', 'in', 'sight', '?']\n",
      "['Maybe', 'we', 'could', 'explore', 'new', 'beginnings', 'together', '?']\n",
      "['Im', '45', 'Slim/Med', 'build', ',', 'GSOH', ',', 'high', 'needs', 'and', 'looking', 'for', 'someone', 'similar', '.']\n",
      "['You', 'WONT', 'be', 'disappointed', '.']\n"
     ]
    }
   ],
   "source": [
    "for sent in sent_tokenize(single_no8):\n",
    "    print(word_tokenize(sent))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lowercasing\n",
    "We can simply lowercase them after we do sent_tokenize() and word_tokenize(). \n",
    "The tokenizers uses the capitalization as cues to know when to split so removing them before the calling the functions would be sub-optimal."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['ARE YOU ALONE or lost in a r/ship too, with no hope in sight?',\n",
       " 'Maybe we could explore new beginnings together?',\n",
       " 'Im 45 Slim/Med build, GSOH, high needs and looking for someone similar.',\n",
       " 'You WONT be disappointed.']"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sent_tokenize(single_no8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['are', 'you', 'alone', 'or', 'lost', 'in', 'a', 'r/ship', 'too', ',', 'with', 'no', 'hope', 'in', 'sight', '?']\n",
      "['maybe', 'we', 'could', 'explore', 'new', 'beginnings', 'together', '?']\n",
      "['im', '45', 'slim/med', 'build', ',', 'gsoh', ',', 'high', 'needs', 'and', 'looking', 'for', 'someone', 'similar', '.']\n",
      "['you', 'wont', 'be', 'disappointed', '.']\n"
     ]
    }
   ],
   "source": [
    "for sent in sent_tokenize(single_no8):\n",
    "    # It's a little in efficient to loop through each word,\n",
    "    # after but sometimes it helps to get better tokens.\n",
    "    print([word.lower() for word in word_tokenize(sent)])\n",
    "    # Alternatively:\n",
    "    #print(list(map(str.lower, word_tokenize(sent))))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['ARE', 'YOU', 'ALONE', 'or', 'lost', 'in', 'a', 'r/ship', 'too', ',', 'with', 'no', 'hope', 'in', 'sight', '?', 'Maybe', 'we', 'could', 'explore', 'new', 'beginnings', 'together', '?', 'Im', '45', 'Slim/Med', 'build', ',', 'GSOH', ',', 'high', 'needs', 'and', 'looking', 'for', 'someone', 'similar', '.', 'You', 'WONT', 'be', 'disappointed', '.']\n"
     ]
    }
   ],
   "source": [
    "print(word_tokenize(single_no8))  # Treats the whole line as one document."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tangential Note\n",
    "Punkt is a statistical model so it applies the knowledge it has learnt from previous data. \n",
    "Generally, it works for most cases on well-formed texts but if your data is different e.g. user-generated noisy texts, you might have to retrain a new model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "E.g. if we look at candidate no. 9, we see that it's splitting on y.o. (its thinking that its the end of the sentnence) and not splitting on &c. (its thinking that its an abbreviation, e.g. Mr., Inc.)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['AMIABLE 43 y.o.',\n",
       " 'gentleman with European background, 170 cm, medium build, employed, never married, no children.',\n",
       " 'Enjoys sports, music, cafes, beach &c. Seeks an honest, attractive lady with a European background, without children, who would like to get married and have chil dren in the future.',\n",
       " '29-39 y.o.',\n",
       " 'Prefer non-smoker and living in Adelaide.']"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "single_no9 = webtext.raw('singles.txt').split('\\n')[9]\n",
    "sent_tokenize(single_no9)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Stopwords\n",
    "Stopwords are non-content words that primarily has only grammatical function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package stopwords to\n",
      "[nltk_data]     C:\\Users\\500062812\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Unzipping corpora\\stopwords.zip.\n",
      "['i', 'me', 'my', 'myself', 'we', 'our', 'ours', 'ourselves', 'you', \"you're\", \"you've\", \"you'll\", \"you'd\", 'your', 'yours', 'yourself', 'yourselves', 'he', 'him', 'his', 'himself', 'she', \"she's\", 'her', 'hers', 'herself', 'it', \"it's\", 'its', 'itself', 'they', 'them', 'their', 'theirs', 'themselves', 'what', 'which', 'who', 'whom', 'this', 'that', \"that'll\", 'these', 'those', 'am', 'is', 'are', 'was', 'were', 'be', 'been', 'being', 'have', 'has', 'had', 'having', 'do', 'does', 'did', 'doing', 'a', 'an', 'the', 'and', 'but', 'if', 'or', 'because', 'as', 'until', 'while', 'of', 'at', 'by', 'for', 'with', 'about', 'against', 'between', 'into', 'through', 'during', 'before', 'after', 'above', 'below', 'to', 'from', 'up', 'down', 'in', 'out', 'on', 'off', 'over', 'under', 'again', 'further', 'then', 'once', 'here', 'there', 'when', 'where', 'why', 'how', 'all', 'any', 'both', 'each', 'few', 'more', 'most', 'other', 'some', 'such', 'no', 'nor', 'not', 'only', 'own', 'same', 'so', 'than', 'too', 'very', 's', 't', 'can', 'will', 'just', 'don', \"don't\", 'should', \"should've\", 'now', 'd', 'll', 'm', 'o', 're', 've', 'y', 'ain', 'aren', \"aren't\", 'couldn', \"couldn't\", 'didn', \"didn't\", 'doesn', \"doesn't\", 'hadn', \"hadn't\", 'hasn', \"hasn't\", 'haven', \"haven't\", 'isn', \"isn't\", 'ma', 'mightn', \"mightn't\", 'mustn', \"mustn't\", 'needn', \"needn't\", 'shan', \"shan't\", 'shouldn', \"shouldn't\", 'wasn', \"wasn't\", 'weren', \"weren't\", 'won', \"won't\", 'wouldn', \"wouldn't\"]\n"
     ]
    }
   ],
   "source": [
    "from nltk.corpus import stopwords\n",
    "import nltk\n",
    "nltk.download('stopwords')\n",
    "\n",
    "stopwords_en = stopwords.words('english')\n",
    "print(stopwords_en)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Let's try to remove the stopwords using the English stopwords list in NLTK"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['are', 'you', 'alone', 'or', 'lost', 'in', 'a', 'r/ship', 'too', ',', 'with', 'no', 'hope', 'in', 'sight', '?', 'maybe', 'we', 'could', 'explore', 'new', 'beginnings', 'together', '?', 'im', '45', 'slim/med', 'build', ',', 'gsoh', ',', 'high', 'needs', 'and', 'looking', 'for', 'someone', 'similar', '.', 'you', 'wont', 'be', 'disappointed', '.']\n"
     ]
    }
   ],
   "source": [
    "#For instance, lets go back to the our single_no8\n",
    "# Treat the multiple sentences as one document (no need to sent_tokenize)\n",
    "# Tokenize and lowercase\n",
    "single_no8_tokenized_lowered = list(map(str.lower, word_tokenize(single_no8)))\n",
    "print(single_no8_tokenized_lowered)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['alone', 'lost', 'r/ship', ',', 'hope', 'sight', '?', 'maybe', 'could', 'explore', 'new', 'beginnings', 'together', '?', 'im', '45', 'slim/med', 'build', ',', 'gsoh', ',', 'high', 'needs', 'looking', 'someone', 'similar', '.', 'wont', 'disappointed', '.']\n"
     ]
    }
   ],
   "source": [
    "stopwords_en = set(stopwords.words('english')) # Set checking is faster in Python than list.\n",
    "\n",
    "# List comprehension.\n",
    "print([word for word in single_no8_tokenized_lowered if word not in stopwords_en])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Often, we want to remove the punctuations from the documents too."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "From string.punctuation: <class 'str'> !\"#$%&'()*+,-./:;<=>?@[\\]^_`{|}~\n"
     ]
    }
   ],
   "source": [
    "from string import punctuation\n",
    "# It's a string so we have to them into a set type\n",
    "print('From string.punctuation:', type(punctuation), punctuation)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Combining the punctuation with the stopwords from NLTK."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'your', 'no', 'as', 'once', \"needn't\", \"wouldn't\", 'who', '*', 'himself', 'very', 'am', \"mightn't\", \"wasn't\", 'been', 'against', 'few', 'do', '`', 'by', 'if', 'below', '~', 'nor', 'd', '+', 'a', 'where', 'is', 'each', \"shan't\", 'm', 'me', \"couldn't\", 'through', 'wouldn', 'herself', 'hers', 'but', 'than', 'hasn', 'needn', 'any', 'when', '(', 'weren', 'mustn', \"that'll\", 'whom', 'myself', 'in', 'during', 'the', 'same', 'up', '!', '^', 'mightn', 'again', '\"', 'does', 'hadn', 't', 'they', \"weren't\", 'o', 'can', 'doing', '&', '>', 'have', 'was', \"it's\", \"should've\", 'after', 'were', 'of', 'about', 're', 'ourselves', 'she', '{', 'before', 'doesn', \"won't\", '.', '-', 'shan', 'them', 'with', 'its', 'aren', '[', 'ain', 'her', '<', 'theirs', 'why', 'down', 'here', ';', 'these', 'that', \"she's\", 'ma', 'further', 'under', ',', 'or', 'couldn', \"aren't\", \"you'll\", 'such', 'own', 'my', 's', '@', 'it', 'both', 'not', 'are', 'from', 'so', 'yourselves', 'having', 'an', \"'\", 'there', 'above', \"haven't\", '|', 'themselves', ')', '\\\\', 'did', 'and', 'at', 'out', 'i', '?', '/', 'until', 'while', 'other', 'had', 'now', 'don', 'he', \"mustn't\", 'wasn', 'more', '}', ']', 'yourself', \"shouldn't\", '=', '_', \"don't\", 'ours', 'all', 'only', 'being', \"you've\", 'because', 'then', 'won', 'him', 'our', 'those', \"you're\", 'for', 'isn', 'too', ':', 'll', '$', 'be', 'just', 'on', 'off', '#', 'should', 'yours', 'shouldn', 'this', 'has', 'what', 'you', \"hasn't\", 'most', 'didn', \"isn't\", '%', \"doesn't\", 'into', 'between', 'their', 'we', 've', \"you'd\", 'some', 'haven', 'to', 'will', 'over', 'itself', 'how', \"didn't\", 'which', 'his', \"hadn't\", 'y'}\n"
     ]
    }
   ],
   "source": [
    "stopwords_en_withpunct = stopwords_en.union(set(punctuation))\n",
    "print(stopwords_en_withpunct)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Removing stopwords with punctuations from Single no. 8"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['alone', 'lost', 'r/ship', 'hope', 'sight', 'maybe', 'could', 'explore', 'new', 'beginnings', 'together', 'im', '45', 'slim/med', 'build', 'gsoh', 'high', 'needs', 'looking', 'someone', 'similar', 'wont', 'disappointed']\n"
     ]
    }
   ],
   "source": [
    "print([word for word in single_no8_tokenized_lowered if word not in stopwords_en_withpunct])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}