Quiz/m5_financial_statements/beautifulSoup.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Parsing an HTML File\n",
    "\n",
    "To parse an HTML or XML document, we need to pass the document into the BeautifulSoup constructor. The BeautifulSoup constructor, `BeautifulSoup(file, 'parser')`, parses the given `file` using the given `parser` and returns a BeautifulSoup object. We can pass our `file` to the constructor either as a string or as an open filehandle, while the `parser` is a string that indicates the parser we want to use.\n",
    "\n",
    "The BeautifulSoup constructor will transform the HTML or XML file into a complex tree of Python objects. One of this objects is the BeautifulSoup object returned by the constructor. The BeautifulSoup object itself represents the document as a whole and can be searched using various methods, as we will see in the following lessons. \n",
    "\n",
    "In the code below, we start by importing the BeautifulSoup library by using the statement:\n",
    "\n",
    "```python\n",
    "from bs4 import BeautifulSoup\n",
    "```\n",
    "\n",
    "We then open the `sample.html` file and pass the open filehandle `f` to the `BeautifulSoup` constructor. We also set the `parser` in the constructor to `lxml` to indicate that we want to use lxml’s HTML parser. The `BeautifulSoup` constructor will return a BeautifulSoup object that we will save in the `page_content` variable. We then print the BeautifulSoup object to see what it looks like."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<!DOCTYPE html>\n",
      "<html lang=\"en-US\">\n",
      "<head>\n",
      "<title>AI For Trading</title>\n",
      "<meta charset=\"utf-8\"/>\n",
      "<link href=\"./teststyle.css\" rel=\"stylesheet\"/>\n",
      "<style>.h2style {background-color: tomato;color: white;padding: 10px;}</style>\n",
      "</head>\n",
      "<body>\n",
      "<h1 id=\"intro\">Get Help From Peers and Mentors</h1>\n",
      "<div class=\"section\">\n",
      "<h2 class=\"h2style\" id=\"hub\">Student Hub</h2>\n",
      "<p>Student Hub is our real time collaboration platform where you can work with peers and mentors. You will find Community rooms with other students and alumni.</p>\n",
      "</div>\n",
      "<hr/>\n",
      "<div class=\"section\">\n",
      "<h2 class=\"h2style\" id=\"know\">Knowledge</h2>\n",
      "<p>Search or ask questions in <a href=\"https://knowledge.udacity.com/\">Knowledge</a></p>\n",
      "</div>\n",
      "<div class=\"outro\">\n",
      "<h3 id=\"know\">Good Luck</h3>\n",
      "<p>Good luck and we hope you enjoy the course</p>\n",
      "</div>\n",
      "</body>\n",
      "</html>\n"
     ]
    }
   ],
   "source": [
    "# Import BeautifulSoup\n",
    "from bs4 import BeautifulSoup\n",
    "\n",
    "# Open the HTML file and create a BeautifulSoup Object\n",
    "with open('./sample.html') as f:\n",
    "    page_content = BeautifulSoup(f, 'lxml')\n",
    "\n",
    "# Print the BeautifulSoup Object\n",
    "print(page_content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, `page_content`, holds the entire contents of our `sample.html` file. Notice that when we print the BeautifulSoup object, it is not printed in a nice format and it is very hard to read.\n",
    "\n",
    "Luckily, the BeautifulSoup object has the `.prettify()` method that allows our BeautifulSoup object to be printed with all the tags nicely indented. Let's see how this works:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<!DOCTYPE html>\n",
      "<html lang=\"en-US\">\n",
      " <head>\n",
      "  <title>\n",
      "   AI For Trading\n",
      "  </title>\n",
      "  <meta charset=\"utf-8\"/>\n",
      "  <link href=\"./teststyle.css\" rel=\"stylesheet\"/>\n",
      "  <style>\n",
      "   .h2style {background-color: tomato;color: white;padding: 10px;}\n",
      "  </style>\n",
      " </head>\n",
      " <body>\n",
      "  <h1 id=\"intro\">\n",
      "   Get Help From Peers and Mentors\n",
      "  </h1>\n",
      "  <div class=\"section\">\n",
      "   <h2 class=\"h2style\" id=\"hub\">\n",
      "    Student Hub\n",
      "   </h2>\n",
      "   <p>\n",
      "    Student Hub is our real time collaboration platform where you can work with peers and mentors. You will find Community rooms with other students and alumni.\n",
      "   </p>\n",
      "  </div>\n",
      "  <hr/>\n",
      "  <div class=\"section\">\n",
      "   <h2 class=\"h2style\" id=\"know\">\n",
      "    Knowledge\n",
      "   </h2>\n",
      "   <p>\n",
      "    Search or ask questions in\n",
      "    <a href=\"https://knowledge.udacity.com/\">\n",
      "     Knowledge\n",
      "    </a>\n",
      "   </p>\n",
      "  </div>\n",
      "  <div class=\"outro\">\n",
      "   <h3 id=\"know\">\n",
      "    Good Luck\n",
      "   </h3>\n",
      "   <p>\n",
      "    Good luck and we hope you enjoy the course\n",
      "   </p>\n",
      "  </div>\n",
      " </body>\n",
      "</html>\n"
     ]
    }
   ],
   "source": [
    "# Print the BeautifulSoup Object with prettify\n",
    "print(page_content.prettify())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, the `.prettify()` method makes the object easier to read and also it also allows us to identify tags more readily. In the next lesson, we will see how to access the information contained in each tag."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Navigating The Parse Tree\n",
    "\n",
    "The most straightforward way of navigating the parse tree created by BeautifulSoup is by accessing the HTML or XML tags. We can access the tags as if they were attributes of the BeautifulSoup object. Let's see how this works.\n",
    "\n",
    "In the code below, we will access the `<head>` tag in our `page_content` object by using the statement:\n",
    "\n",
    "```python\n",
    "page_content.head\n",
    "```\n",
    "\n",
    "Whenever we access a tag in this manner, we get a **Tag** object. Consequently, the above statement returns a Tag object that we will save in the `page_head` variable. We then print the Tag object to see what it looks like."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<head>\n",
      " <title>\n",
      "  AI For Trading\n",
      " </title>\n",
      " <meta charset=\"utf-8\"/>\n",
      " <link href=\"./teststyle.css\" rel=\"stylesheet\"/>\n",
      " <style>\n",
      "  .h2style {background-color: tomato;color: white;padding: 10px;}\n",
      " </style>\n",
      "</head>\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Import BeautifulSoup\n",
    "from bs4 import BeautifulSoup\n",
    "\n",
    "# Open the HTML file and create a BeautifulSoup Object\n",
    "with open('./sample.html') as f:\n",
    "    page_content = BeautifulSoup(f, 'lxml')\n",
    "\n",
    "# Access the head tag\n",
    "page_head = page_content.head\n",
    "\n",
    "# Print the Tag Object\n",
    "print(page_head.prettify())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, the `page_head` object has the entire contents of the `<head>` tag only, including all the opening and closing tags within it. These sub tags are known as children of the `<head>` tag. For example, the `<title>` tag is a child of the `<head>` tag. \n",
    "\n",
    "We can access these child tags within the `<head>` tag as if they were attributes of the `page_head` object. For example, if we wanted to access the `<title>` tag within the `<head>` tag, we can use:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<title>AI For Trading</title>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Access the title tag within the head tag from the page_head object\n",
    "page_head.title"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the above statement, `page_head.title`, is equivalent to the statement:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<title>AI For Trading</title>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Access the title tag within the head tag from the page_content object\n",
    "page_content.head.title"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Access The Meta Tag\n",
    "\n",
    "In the cell below, access the `<meta>` tag contained within the `<head>` tag. Start by importing BeautifulSoup, then open the `sample.html` file and pass the open filehandle to the BeautifulSoup constructor and use the `lxml` parser. Save the BeautifulSoup object returned by the constructor in a variable called `page_content`. Then access the `<meta>` tag contained within the `<head>` tag from the `page_content` object. Save the Tag object in a variable called `page_meta` and then print it. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<meta charset=\"utf-8\"/>\n"
     ]
    }
   ],
   "source": [
    "# Import BeautifulSoup\n",
    "from bs4 import BeautifulSoup \n",
    "\n",
    "# Open the HTML file and create a BeautifulSoup Object\n",
    "with open('./sample.html') as f:\n",
    "    page_content = BeautifulSoup(f, 'lxml')\n",
    "\n",
    "# Access the head tag\n",
    "page_meta = page_content.head.meta\n",
    "\n",
    "# Print the Tag Object\n",
    "print(page_meta)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Access The `<h1>` Tag\n",
    "\n",
    "In the cell below, access the `<h1>` tag contained within the `<body>` tag. Start by importing BeautifulSoup, then open the `sample.html` file and pass the open filehandle to the BeautifulSoup constructor and use the `lxml` parser. Save the BeautifulSoup object returned by the constructor in a variable called `page_content`. Then access the `<h1>` tag contained within the `<body>` tag from the `page_content` object. Save the Tag object in a variable called `page_h1` and then print it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<h1 id=\"intro\">Get Help From Peers and Mentors</h1>\n"
     ]
    }
   ],
   "source": [
    "# Import BeautifulSoup\n",
    "from bs4 import BeautifulSoup\n",
    "\n",
    "# Open the HTML file and create a BeautifulSoup Object\n",
    "with open('./sample.html') as f:\n",
    "    page_content = BeautifulSoup(f, 'lxml')\n",
    "\n",
    "# Access the h1 tag\n",
    "page_h1 = page_content.body.h1\n",
    "\n",
    "# Print the Tag Object\n",
    "print(page_h1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Getting Text \n",
    "\n",
    "Let's get the `<title>` tag within the `<head>` tag, again:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<title>AI For Trading</title>\n"
     ]
    }
   ],
   "source": [
    "# Print the title tag within the head tag\n",
    "print(page_head.title)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see from this example, and the ones above it, Tag objects contain the beginning and end tags as well as the text within them. In most cases however we do not want the tags, but rather, only the text contained within the tags. For example, let's suppose we only wanted to get the text `AI for Trading` contained within the `title` tags. In these cases we can use the `.get_text()` method. The `.get_text()` method only gets the text part of a document or tag. Let's see how it works:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "AI For Trading\n"
     ]
    }
   ],
   "source": [
    "# Print only the text in the title tag within the head tag\n",
    "print(page_head.title.get_text())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Remove HTML Tags\n",
    "\n",
    "In the cell below, use the `.get_text()` method to remove all the HTML tags from the `sample.html` document. In other words, just print the entire text in the document, with no HTML tags.\n",
    "\n",
    "**HINT:** Use the `.get_text()` method on the `page_content` object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "AI For Trading\n",
      "\n",
      "\n",
      ".h2style {background-color: tomato;color: white;padding: 10px;}\n",
      "\n",
      "\n",
      "Get Help From Peers and Mentors\n",
      "\n",
      "Student Hub\n",
      "Student Hub is our real time collaboration platform where you can work with peers and mentors. You will find Community rooms with other students and alumni.\n",
      "\n",
      "\n",
      "\n",
      "Knowledge\n",
      "Search or ask questions in Knowledge\n",
      "\n",
      "\n",
      "Good Luck\n",
      "Good luck and we hope you enjoy the course\n",
      "\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Import BeautifulSoup\n",
    "from bs4 import BeautifulSoup\n",
    "\n",
    "# Open the HTML file and create a BeautifulSoup Object\n",
    "with open('./sample.html') as f:\n",
    "    page_content = BeautifulSoup(f, 'lxml')\n",
    "\n",
    "# Print only the text in the whole document\n",
    "print(page_content.get_text())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Getting Attributes\n",
    "\n",
    "An HTML or XML tag can have many attributes. For example, the tag:\n",
    "\n",
    "```python\n",
    "<h1 id='intro'>\n",
    "```\n",
    "has the attribute `id` whose value is `'intro'`. BeautifulSoup allows us to get the value of a tag’s attribute by treating the tag like a dictionary. For example, in the code below we get the value of the `id` attribute of the `<h1>` tag by using:\n",
    "\n",
    "```python\n",
    "page_h1['id']\n",
    "```\n",
    "\n",
    "where `page_h1` is the Tag object that holds the contents of the `<h1>` tag. Let's see how this works in the code below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "intro\n"
     ]
    }
   ],
   "source": [
    "# Import BeautifulSoup\n",
    "from bs4 import BeautifulSoup\n",
    "\n",
    "# Open the HTML file and create a BeautifulSoup Object\n",
    "with open('./sample.html') as f:\n",
    "    page_content = BeautifulSoup(f, 'lxml')\n",
    "\n",
    "# Access the h1 tag\n",
    "page_h1 = page_content.body.h1\n",
    "\n",
    "# Get the value of the id attribute from the h1 tag\n",
    "h1_id_attr = page_h1['id']\n",
    "\n",
    "# Print the value of the id attribute\n",
    "print(h1_id_attr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get The Website Address\n",
    "\n",
    "In this exercise, you will get the website address in a hyperlink tag. Hyperlinks are defined by the `<a>` tag. In our `sample.html` document we only have one hyperlink:\n",
    "\n",
    "```python\n",
    "<a href=\"https://knowledge.udacity.com/\">Knowledge</a>\n",
    "```\n",
    "Hyperlinks are used to link webpages together. The `href` attribute in the `<a>` tag, indicates the link's destination, *i.e.* a website address.\n",
    "\n",
    "In the cell below, open the `sample.html` file and pass the open filehandle to the BeautifulSoup constructor and use the `lxml` parser. Save the BeautifulSoup object returned by the constructor in a variable called `page_content`. Then access the `<a>` tag from the `page_content` object. Save the Tag object in a variable called `page_hyperlink`. Then get the `href` attribute from the `page_hyperlink` object and save it into a variable called `href_attr`. Finally, print the `href_attr` variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://knowledge.udacity.com/\n"
     ]
    }
   ],
   "source": [
    "# Import BeautifulSoup\n",
    "from bs4 import BeautifulSoup\n",
    "\n",
    "# Open the HTML file and create a BeautifulSoup Object\n",
    "with open('./sample.html') as f:\n",
    "    page_content = BeautifulSoup(f, 'lxml')\n",
    "\n",
    "# Access the a tag\n",
    "page_hyperlink = page_content.a\n",
    "\n",
    "# Get the href attribute from the a tag\n",
    "href_attr = page_hyperlink['href']\n",
    "\n",
    "# Print the href attribute\n",
    "print(href_attr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Finding All Tags\n",
    "\n",
    "Let's take a look at our `sample.html` file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<!DOCTYPE html>\n",
      "<html lang=\"en-US\">\n",
      " <head>\n",
      "  <title>\n",
      "   AI For Trading\n",
      "  </title>\n",
      "  <meta charset=\"utf-8\"/>\n",
      "  <link href=\"./teststyle.css\" rel=\"stylesheet\"/>\n",
      "  <style>\n",
      "   .h2style {background-color: tomato;color: white;padding: 10px;}\n",
      "  </style>\n",
      " </head>\n",
      " <body>\n",
      "  <h1 id=\"intro\">\n",
      "   Get Help From Peers and Mentors\n",
      "  </h1>\n",
      "  <div class=\"section\">\n",
      "   <h2 class=\"h2style\" id=\"hub\">\n",
      "    Student Hub\n",
      "   </h2>\n",
      "   <p>\n",
      "    Student Hub is our real time collaboration platform where you can work with peers and mentors. You will find Community rooms with other students and alumni.\n",
      "   </p>\n",
      "  </div>\n",
      "  <hr/>\n",
      "  <div class=\"section\">\n",
      "   <h2 class=\"h2style\" id=\"know\">\n",
      "    Knowledge\n",
      "   </h2>\n",
      "   <p>\n",
      "    Search or ask questions in\n",
      "    <a href=\"https://knowledge.udacity.com/\">\n",
      "     Knowledge\n",
      "    </a>\n",
      "   </p>\n",
      "  </div>\n",
      "  <div class=\"outro\">\n",
      "   <h3 id=\"know\">\n",
      "    Good Luck\n",
      "   </h3>\n",
      "   <p>\n",
      "    Good luck and we hope you enjoy the course\n",
      "   </p>\n",
      "  </div>\n",
      " </body>\n",
      "</html>\n"
     ]
    }
   ],
   "source": [
    "# Import BeautifulSoup\n",
    "from bs4 import BeautifulSoup\n",
    "\n",
    "# Open the HTML file and create a BeautifulSoup Object\n",
    "with open('./sample.html') as f:\n",
    "    page_content = BeautifulSoup(f, 'lxml')\n",
    "    \n",
    "print(page_content.prettify())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that our `sample.html` file has two `<h2>` tags:\n",
    "\n",
    "```python\n",
    "<h2 class=\"h2style\" id=\"hub\">Student Hub</h2>\n",
    "```\n",
    "\n",
    "and\n",
    "\n",
    "```python\n",
    "<h2 class=\"h2style\" id=\"know\">Knowledge</h2>\n",
    "```\n",
    "\n",
    "So let's try to access these `<h2>` tags as we did before: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<h2 class=\"h2style\" id=\"hub\">Student Hub</h2>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get h2 tag\n",
    "page_content.body.h2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, we only get the first `<h2>` tag and not both. This is because, when we access a tag as an attribute, we only get the first occurrence of that tag in the document. \n",
    "\n",
    "In order to get all the `<h2>` tags we need to use the `.find_all()` search method, which is the topic of our next lesson."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Searching The Parse Tree\n",
    "\n",
    "BeautifulSoup provides a number of methods for searching the parse tree, but we will only cover the `.find_all()` method in this lesson. You can learn about other search methods in the [BeautifulSoup Documentation](https://www.crummy.com/software/BeautifulSoup/bs4/doc/).\n",
    "\n",
    "The `.find_all(filter)` method will search an entire document for the given `filter`. The `filter` can be a string containing the HTML or XML tag name, a tag attribute, or even a regular expression. In this notebook we will see examples of these cases. \n",
    "\n",
    "So let's begin by using the `.find_all()` method to find all `<h2>` tags in our `sample.html` file. To do this, we will pass the string `'h2'` to the `.find_all()` method as shown in the code below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[<h2 class=\"h2style\" id=\"hub\">Student Hub</h2>, <h2 class=\"h2style\" id=\"know\">Knowledge</h2>]\n"
     ]
    }
   ],
   "source": [
    "# Import BeautifulSoup\n",
    "from bs4 import BeautifulSoup\n",
    "\n",
    "# Open the HTML file and create a BeautifulSoup Object\n",
    "with open('./sample.html') as f:\n",
    "    page_content = BeautifulSoup(f, 'lxml')\n",
    "\n",
    "# Find all the h2 tags\n",
    "h2_list = page_content.find_all('h2')\n",
    "\n",
    "# Print the h2_list\n",
    "print(h2_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, the `.find_all()` method returns a list with all the `<h2>` tags. Since lists are iterables, we can loop through the `h2_list` to print each tag, as shown below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<h2 class=\"h2style\" id=\"hub\">Student Hub</h2>\n",
      "<h2 class=\"h2style\" id=\"know\">Knowledge</h2>\n"
     ]
    }
   ],
   "source": [
    "# Print each tag in the h2_list\n",
    "for tag in h2_list:\n",
    "    print(tag)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Searching For Multiple Tags\n",
    "\n",
    "We can also search for more than one tag at a time by passing a list to the `.find_all()` method. Let's see an example.\n",
    "\n",
    "Let's suppose we wanted to search for all the `<h2>` and `<p>` tags in our `sample.html` file. Instead of using two statements (one for each tag):\n",
    "\n",
    "```python\n",
    "h2_list = page_content.find_all('h2')\n",
    "p_list = page_content.find_all('p')\n",
    "```\n",
    "\n",
    "we can just pass the list `['h2', 'p']` to the `.find_all()` method, as shown in the code below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<h2 class=\"h2style\" id=\"hub\">\n",
      " Student Hub\n",
      "</h2>\n",
      "\n",
      "<p>\n",
      " Student Hub is our real time collaboration platform where you can work with peers and mentors. You will find Community rooms with other students and alumni.\n",
      "</p>\n",
      "\n",
      "<h2 class=\"h2style\" id=\"know\">\n",
      " Knowledge\n",
      "</h2>\n",
      "\n",
      "<p>\n",
      " Search or ask questions in\n",
      " <a href=\"https://knowledge.udacity.com/\">\n",
      "  Knowledge\n",
      " </a>\n",
      "</p>\n",
      "\n",
      "<p>\n",
      " Good luck and we hope you enjoy the course\n",
      "</p>\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Import BeautifulSoup\n",
    "from bs4 import BeautifulSoup\n",
    "\n",
    "# Open the HTML file and create a BeautifulSoup Object\n",
    "with open('./sample.html') as f:\n",
    "    page_content = BeautifulSoup(f, 'lxml')\n",
    "    \n",
    "# Print all the h2 and p tags\n",
    "for tag in page_content.find_all(['h2', 'p']):\n",
    "    print(tag.prettify())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Searching For Tags With Particular Attributes\n",
    "\n",
    "The `.find_all()` method also allows us to pass some arguments, such as the attribute of a tag, so that we can search the entire document for the exact tag we want. For example, in our `sample.html` file, we have two `<h2>` tags:\n",
    "\n",
    "1. `<h2 class=\"h2style\" id=\"hub\">Student Hub</h2>`\n",
    "\n",
    "2. `<h2 class=\"h2style\" id=\"know\">Knowledge</h2>`\n",
    "\n",
    "We can see that the first `<h2>` tag has the attribute `id=\"hub\"`, while the second `<h2>` tag has the attribute `id=\"know\"`. Let's suppose, we only wanted to search our `sample.html` document for the `<h2>` tag that had `id=\"know\"`. To do this, we will add the `id` attribute to the `.find_all()` method as shown below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<h2 class=\"h2style\" id=\"know\">Knowledge</h2>\n"
     ]
    }
   ],
   "source": [
    "# Import BeautifulSoup\n",
    "from bs4 import BeautifulSoup\n",
    "\n",
    "# Open the HTML file and create a BeautifulSoup Object\n",
    "with open('./sample.html') as f:\n",
    "    page_content = BeautifulSoup(f, 'lxml')\n",
    "\n",
    "# Find the h2 tag with id = know\n",
    "h2_know = page_content.find_all('h2', id = 'know')\n",
    "\n",
    "# Print each item in the h2_know\n",
    "for tag in h2_know:\n",
    "    print(tag)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see, the list returned by the `.find_all()` method only has one element, and it corresponds to the `<h2>` tag that has `id=\"know\"`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Searching For Attributes Directly\n",
    "\n",
    "The `.find_all()` method also allows us to search for tag attributes directly. For example, we can search for all the tags that have the attribute `id=\"intro\"` by only passing this attribute to the `.find_all()` method, as shown below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<h1 id=\"intro\">Get Help From Peers and Mentors</h1>\n"
     ]
    }
   ],
   "source": [
    "# Import BeautifulSoup\n",
    "from bs4 import BeautifulSoup\n",
    "\n",
    "# Open the HTML file and create a BeautifulSoup Object\n",
    "with open('./sample.html') as f:\n",
    "    page_content = BeautifulSoup(f, 'lxml')\n",
    "\n",
    "# Print all the tags with id = intro\n",
    "for tag in page_content.find_all(id = 'intro'):\n",
    "    print(tag)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Searching by `class`\n",
    "\n",
    "Let's suppose we wanted to find all the tags that had the attribute `class=\"h2style\"`. Unfortunately, in this case, we can't simply pass this attribute to the `.find_all()` method. The reason is that the **CSS** attribute, `class`, is a reserved word in Python. Therefore, using `class` as a keyword argument in the `.find_all()` method, will give you a syntax error. To get around this problem, BeautifulSoup has implemented the keyword `class_` (notice the underscore at the end) that can be used to search for the `class` attribute. Let's see how this works.\n",
    "\n",
    "In the code below, we will use the `.find_all()` method to search for all the tags in our `sample.html` file that have the attribute `class=\"h2style\"`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<h2 class=\"h2style\" id=\"hub\">Student Hub</h2>\n",
      "<h2 class=\"h2style\" id=\"know\">Knowledge</h2>\n"
     ]
    }
   ],
   "source": [
    "# Import BeautifulSoup\n",
    "from bs4 import BeautifulSoup\n",
    "\n",
    "# Open the HTML file and create a BeautifulSoup Object\n",
    "with open('./sample.html') as f:\n",
    "    page_content = BeautifulSoup(f, 'lxml')\n",
    "    \n",
    "# Print the tags that have the attribute class_ = 'h2style'\n",
    "for tag in page_content.find_all(class_ = 'h2style'):\n",
    "    print(tag)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that we get the two `<h2>` tags since they are the only ones in our document that have the attribute `class=\"h2style\"`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Searching With Regular Expressions\n",
    "\n",
    "We can also pass a regular expression object to the `.find_all()` method. Let's see an example. The code below uses a regular expression to find all the tags whose names contain the letter `i`. Remember that in order to use regular expressions we must import the `re` module. In this particular example we will be only interested in printing the tag name and not its entire content. In order to do this, we will use the `.name` attribute of the `Tag` object to only print the name of tag itself, as shown below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "title\n",
      "link\n",
      "div\n",
      "div\n",
      "div\n"
     ]
    }
   ],
   "source": [
    "# Import BeautifulSoup\n",
    "from bs4 import BeautifulSoup\n",
    "\n",
    "# Import the re module\n",
    "import re \n",
    "\n",
    "# Open the HTML file and create a BeautifulSoup Object\n",
    "with open('./sample.html') as f:\n",
    "    page_content = BeautifulSoup(f, 'lxml')\n",
    "    \n",
    "# Print only the tag names of all the tags whose name contain the letter i\n",
    "for tag in page_content.find_all(re.compile(r'i')):\n",
    "    print(tag.name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tag's Children\n",
    "\n",
    "We saw in an earlier lesson that tags may contain other tags and strings within in them. These elements are known as the tag’s children. For simplicity, in the following examples we will use a simpler HTML file named `sample2.html`. So let's print it to see what it looks like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<!DOCTYPE html>\n",
      "<html lang=\"en-US\">\n",
      " <head>\n",
      "  <title>\n",
      "   AI For Trading\n",
      "  </title>\n",
      "  <meta charset=\"utf-8\"/>\n",
      "  <link href=\"./teststyle.css\" rel=\"stylesheet\"/>\n",
      "  <style>\n",
      "   .h2style {background-color: tomato;color: white;padding: 10px;}\n",
      "  </style>\n",
      " </head>\n",
      "</html>\n"
     ]
    }
   ],
   "source": [
    "# Import BeautifulSoup\n",
    "from bs4 import BeautifulSoup\n",
    "\n",
    "# Open the HTML file and print the head tag\n",
    "with open('./sample2.html') as f:\n",
    "    print(BeautifulSoup(f, 'lxml').prettify())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, the `<html>` tag contains some child tags. For example, the `<head>` tag is a direct child of the `<html>` tag. Similarly, the `<title>` tag is a direct child of the `<head>` tag. We also see that the `<title>` tag itself has a child, namely the string `'AI For Trading'`. BeautifulSoup provides a lot of different attributes for navigating over a tag’s children. We already saw that we can access child tags as if they were attributes of the parent tag. For example, we can access the string `'AI For Trading'` from our `soup` object by using:\n",
    "\n",
    "```python\n",
    "soup.head.title.get_text()\n",
    "```\n",
    "as shown in the code below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "AI For Trading\n"
     ]
    }
   ],
   "source": [
    "# Import BeautifulSoup\n",
    "from bs4 import BeautifulSoup\n",
    "\n",
    "# Open the HTML file and print the text in the title tag within the head tag\n",
    "with open('./sample2.html') as f:\n",
    "    print(BeautifulSoup(f, 'lxml').head.title.get_text())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can view a tag's children by using the `.contents` attribute of the Tag object. The `.contents` attribute returns a list with all the tag's children. By counting the number of elements in this list we can see how many children a parent tag has. In the code below we print the children of the `<head>` tag and we also print the number children the `<head>` tag has:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[<title>AI For Trading</title>, <meta charset=\"utf-8\"/>, <link href=\"./teststyle.css\" rel=\"stylesheet\"/>, <style>.h2style {background-color: tomato;color: white;padding: 10px;}</style>]\n",
      "\n",
      "The <head> tag contains 4 children\n"
     ]
    }
   ],
   "source": [
    "# Import BeautifulSoup\n",
    "from bs4 import BeautifulSoup\n",
    "\n",
    "# Open the HTML file and create a BeautifulSoup Object\n",
    "with open('./sample2.html') as f:\n",
    "    page_content = BeautifulSoup(f, 'lxml')\n",
    "\n",
    "# Access the head tag\n",
    "page_head = page_content.head\n",
    "\n",
    "# Print the children of the head tag\n",
    "print(page_head.contents)\n",
    "\n",
    "# Print the number of children of the head tag\n",
    "print('\\nThe <head> tag contains {} children'.format(len(page_head.contents)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The Recursive Argument\n",
    "\n",
    "If we use the `.find_all()` method on a Tag object, `tag.find_all()`,then the `find_all()` method will search all the tag's children, its children’s children, and so on. However, there will be times where you only want BeautifulSoup to search a tag's direct children. To do this, we can pass the `recursive=False` argument, to the `.find_all()` method. Let's see an example.\n",
    "\n",
    "Let's start by printing our `sample2.html` file again to see what its structure looks like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<!DOCTYPE html>\n",
      "<html lang=\"en-US\">\n",
      " <head>\n",
      "  <title>\n",
      "   AI For Trading\n",
      "  </title>\n",
      "  <meta charset=\"utf-8\"/>\n",
      "  <link href=\"./teststyle.css\" rel=\"stylesheet\"/>\n",
      "  <style>\n",
      "   .h2style {background-color: tomato;color: white;padding: 10px;}\n",
      "  </style>\n",
      " </head>\n",
      "</html>\n"
     ]
    }
   ],
   "source": [
    "# Import BeautifulSoup\n",
    "from bs4 import BeautifulSoup\n",
    "\n",
    "# Open the HTML file and print the head tag\n",
    "with open('./sample2.html') as f:\n",
    "    print(BeautifulSoup(f, 'lxml').prettify())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that the `<head>` tag is directly beneath the `<html>` tag and that the `<title>` tag is directly beneath the `<head>` tag. Even though the `<title>` tag is beneath the `<html>` tag, it’s **not** directly beneath it, because the `<head>` tag is in between them. \n",
    "\n",
    "Now, if we search the `<html>` tag for the `<title>` tag, using the `.find_all()` method, BeautifulSoup will find it because it is searching in all of the descendants of the `<html>` tag:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<title>AI For Trading</title>\n"
     ]
    }
   ],
   "source": [
    "# Import BeautifulSoup\n",
    "from bs4 import BeautifulSoup\n",
    "\n",
    "# Open the HTML file and create a BeautifulSoup Object\n",
    "with open('./sample2.html') as f:\n",
    "    page_content = BeautifulSoup(f, 'lxml')\n",
    "    \n",
    "# Search the html tag for the title tag\n",
    "for tag in page_content.html.find_all('title'):\n",
    "    print(tag)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that the we get a match.\n",
    "\n",
    "Now, let's restrict our search to only look at the `<html>` tag’s direct children, by using the `recursive=False` argument:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import BeautifulSoup\n",
    "from bs4 import BeautifulSoup\n",
    "\n",
    "# Open the HTML file and create a BeautifulSoup Object\n",
    "with open('./sample2.html') as f:\n",
    "    page_content = BeautifulSoup(f, 'lxml')\n",
    "\n",
    "# Search the html tag's direct children for the title tag\n",
    "for tag in page_content.html.find_all('title', recursive = False):\n",
    "    print(tag)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that now we don't get any matches because the `<title>` tag is **not** a direct descendent of the `<html>` tag."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Coding Exercise: Get Headers and Paragraphs\n",
    "\n",
    "In the cell below, use what you learned in the previous lessons to print only the text from all the `<h2>` and `<p>` tags inside the `<div>` tags that have a `class=\"section\"` attribute. For this quiz use the `sampe.html` file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<h2 class=\"h2style\" id=\"hub\">Student Hub</h2>\n",
      "<p>Student Hub is our real time collaboration platform where you can work with peers and mentors. You will find Community rooms with other students and alumni.</p>\n",
      "<h2 class=\"h2style\" id=\"know\">Knowledge</h2>\n",
      "<p>Search or ask questions in <a href=\"https://knowledge.udacity.com/\">Knowledge</a></p>\n"
     ]
    }
   ],
   "source": [
    "# Import BeautifulSoup\n",
    "from bs4 import BeautifulSoup\n",
    "\n",
    "# Open the HTML file and create a BeautifulSoup Object\n",
    "with open('./sample.html') as f:\n",
    "    page_content = BeautifulSoup(f, 'lxml')\n",
    "\n",
    "# Print only the text from all the <h2> and <p> tags inside the <div> tags that have a class=\"section\" attribute\n",
    "for tag in page_content.find_all(['div'], class_ = 'section'):\n",
    "    print(tag.h2)\n",
    "    print(tag.p)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}