3-principles-of-machine-learning-for-named-entities.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Principles of Machine Learning for Named Entities\n",
    "---\n",
    "---\n",
    "\n",
    "## Machine Learning Model Predictions\n",
    "\n",
    "A machine learning (ML) algorithm \"learns\" from past experience by storing that experience in a **statistical model**. This happens by training the algorithm on a large amount of relevant data. Then, when it encounters new data, it applies the existing statistical model, makes the best **predictions** it can and calculates a **degree of certainty** in those predictions.\n",
    "\n",
    "When a language model creates a named entity label for a particular token (or span of tokens) it is actually making predictions (or \"guesses\", if you like) and recording the one with the highest probability.\n",
    "\n",
    "<a href=\"https://museums.cam.ac.uk/research/cambridge-university-herbarium\"><img src=\"https://museums.cam.ac.uk/sites/default/files/inline-images/Herbarium%202.jpg\" alt=\"Cambridge University Herbarium\" title=\"Cambridge University Herbarium\"></a>\n",
    "<p style=\"text-align: center; font-style: italic;\">Cambridge University Herbarium</p>\n",
    "\n",
    "For example, let's consider the word \"Herbarium\" in its context within this sentence in a Henslow letter (see the [last notebook](2-named-entity-recognition-of-henslow-data.ipynb#NER-in-Practice:-A-Letter-from-William-Christy,-Jr.,--to-John-Henslow)):\n",
    "\n",
    "> _\"I am only anxious to shew you every opportunity of benefiting your Herbarium.\"_\n",
    "\n",
    "A model might calculate for all possible outcomes a probability of occurrence, which could look something like this:\n",
    "\n",
    "Named Entities:\n",
    "* \"PRODUCT\": 44%\n",
    "* \"ORG\": 41%\n",
    "* \"WORK_OF_ART\": 9%\n",
    "* \"PERSON\": 5%\n",
    "* (All others... 1% in total)\n",
    "\n",
    "So, while we as humans can tell \"PRODUCT\" is not accurate in the context of this letter, it has the highest probability (44%) of being correct as far as the model is concerned, based on what it has learnt in the past.\n",
    "\n",
    "---\n",
    "> I am only anxious to shew you every opportunity of benefiting your \n",
    "<mark class=\"entity\" style=\"background: #bfeeb7; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
    "    Herbarium\n",
    "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">PRODUCT</span>\n",
    "</mark>\n",
    ".\n",
    "\n",
    "---\n",
    "\n",
    "Of course, I have made up these figures for the purpose of my explanation. This is not actually how spaCy's algorithm works in detail, and it's not really possible to access raw scores like this from spaCy in a comparable way, but it helps illustrate my point that a model's output is probabilistic.\n",
    "\n",
    "---\n",
    "---\n",
    "\n",
    "## Lifecycle of Machine Learning\n",
    "\n",
    "Clearly, one pass over some training data may not be sufficient in many cases. It's necessary to train the model, check the model, re-train the model, check again, and so on, iterating on the model to achieve the most acceptable result with the time and resources available. This is often referred to as a lifecycle.\n",
    "\n",
    "* **Train** the model: feed the algorithm a large set of correctly labelled data.\n",
    "* **Validate** the model: test its accuracy by asking for predictions on a subset of labelled data that has been reserved for this purpose.\n",
    "* **Re-train** the model: if necessary, update the model to make better predictions.\n",
    "* **Apply** the model: run the main body of your novel data through the algorithm and get the predictions.\n",
    " \n",
    "As you work through a ML project, you may need to repeat and finesse these steps.\n",
    "\n",
    "### Catastrophic Forgetting\n",
    "\n",
    "Choosing which examples to train or update a model with is a skilled task. For example, if you don't include examples in your training data of named entities in a particular context the model has already seen and can predict correctly, you may find that the model stops bothering to predict those entities. This is called **catastrophic forgetting** and is something you have to work hard to avoid!\n",
    "\n",
    "![Caïn venant de tuer son frère Abel, by Henri Vidal in Tuileries Garden in Paris, France. Alex E. Proimos: CC BY 2.0](https://upload.wikimedia.org/wikipedia/commons/thumb/3/3b/Paris_Tuileries_Garden_Facepalm_statue.jpg/320px-Paris_Tuileries_Garden_Facepalm_statue.jpg \"Caïn venant de tuer son frère Abel, by Henri Vidal in Tuileries Garden in Paris, France. Alex E. Proimos: CC BY 2.0\")\n",
    "\n",
    "---\n",
    "---\n",
    "\n",
    "## Working with spaCy's Default Language Models\n",
    "\n",
    "As we have seen, spaCy provides some [pre-trained language models](https://spacy.io/usage/models) you can download and use for a small range of modern European languages. These have been trained on one or more large, high-quality datasets. \n",
    "\n",
    "These training sets may have limited relevance for projects that you hope to work with, such as:\n",
    "\n",
    "* language dialects\n",
    "* ancient or historical forms of language\n",
    "* context-specific language styles. \n",
    "\n",
    "For named entities, in particular, you may wish to recognise _new_ named entities or a _different set_ of named entities than provided by default.\n",
    "\n",
    "The alternatives are:\n",
    "\n",
    "* **Train a new model from scratch**: this may be appropriate in some cases, but it requires a lot of effort to create labels and it is time consuming computationally.\n",
    "* **Improve an existing model**: if you can just re-train a model to improve or modify it, this will be significantly easier and less time consuming.\n",
    "\n",
    "For an overview of the technical details, you can read more about [Training spaCy’s Statistical Models](https://spacy.io/usage/training#basics)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Support for Other Languages\n",
    "\n",
    "Support is currently being developed for many other languages, but they are at various stages of development. Many have some elements of a tokenizer and lemmatizer available, but no pipeline components for parts-of-speech tagging or named entity recognition.\n",
    "\n",
    "Below is code for how to load and run a blank model with a Russian tokenizer and feed in some example sentences. You can substitute Russian for any of the [supported languages listed](https://spacy.io/usage/models#languages) (with mixed results)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import spacy\n",
    "from spacy.lang.ru import Russian\n",
    "from spacy.lang.ru.examples import sentences\n",
    "\n",
    "# Create an empty Russian language model\n",
    "nlp = Russian()\n",
    "\n",
    "# Process list of example texts in Russian\n",
    "docs = nlp.pipe(sentences)\n",
    "\n",
    "# Print each sentence and its alphabetic tokens and lemmas\n",
    "for doc in docs:\n",
    "    print(f'\\nSentence: {doc.text}')\n",
    "    for token in doc:\n",
    "        if token.is_alpha:\n",
    "            print(f'Token: {token.text}, Lemma: {token.lemma_}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "---\n",
    "\n",
    "## Training Data Labelling and Format\n",
    "Before we starting training a model, we need to consider how we can acquire the training data we need. \n",
    "\n",
    "As we have seen, training data is a portion of your data that has been accurately **labelled** with the location of each named entity in the text and its entity type. Labelling is one type of **annotation** for machine learning.\n",
    "\n",
    "However, spaCy only accepts annotations in certain formats, either of:\n",
    "\n",
    "* A list of texts and named entity labels — when using the [`nlp.update()` method](https://spacy.io/api/language#update)\n",
    "* JSON format — when using spaCy's [`spacy train` command](https://spacy.io/usage/training#spacy-train-cli)\n",
    "\n",
    "For example, the `nlp.update()` method needs to receive training data like this:\n",
    "\n",
    "`[(\"Yours very sincerely | John Evans\", {\"entities\": [(23, 33, \"PERSON\")]}),]`\n",
    "\n",
    "It hardly needs to be said, this is not very user friendly for a human! 😫 \n",
    "\n",
    "We might need to manually add or correct hundreds of labels. **What can we do to make it less painful?**\n",
    "\n",
    "If we used displaCy to visualise this label it might look like this:\n",
    "\n",
    "---\n",
    "> Yours very sincerely | \n",
    "<mark class=\"entity\" style=\"background: #aa9cfc; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
    "    John Evans\n",
    "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">PERSON</span>\n",
    "    </mark>\n",
    "\n",
    "---\n",
    "\n",
    "This sort of visual interface would be ideal for annotating data as a human. In the next section we will look at software for doing just that."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "---\n",
    "---\n",
    "\n",
    "## Annotation as a Human\n",
    "\n",
    "### Labelling Tasks\n",
    "\n",
    "As a human, there are several different labelling tasks you may want to perform:\n",
    "* Labelling data from scratch.\n",
    "* Verifying or rejecting labels predicted by spaCy.\n",
    "* Labelling data for a new type of entity that spaCy doesn't know about yet.\n",
    "\n",
    "Which of these tasks you need to do depends on your project, but we will have the opportunity to try all three.\n",
    "\n",
    "### Annotation Using Doccano\n",
    "\n",
    "There are various software options for annotation as a human. Arguably the best for integration with spaCy is [Prodigy](https://prodi.gy/), which is a high-quality annotation tool made by the same company that created spaCy. It is a paid product, which means we can't use it for CDH Data School this year, but if you are serious about building any machine learning pipeline into your institution's workflow then you may wish to consider this professional annotation tool.\n",
    "\n",
    "For the moment we need a free and open-source alternative, of which there are many of varying quality, but [Doccano](https://doccano.herokuapp.com/) is perhaps the most polished, and it is collaborative, which means we can all edit the same documents.\n",
    "\n",
    "<img src=\"https://raw.githubusercontent.com/doccano/doccano/master/frontend/assets/icon.png\" alt=\"Doccano logo\" title=\"Doccano logo\" width=\"200\">\n",
    "\n",
    "The Doccano interface for annotating named entities looks something like this:\n",
    "\n",
    "![Doccano annotation interface with example text and named entities](assets/doccano-named-entities.png \"Doccano annotation interface with example text and named entities\")\n",
    "\n",
    "---\n",
    "> **EXERCISE**: Follow the instructions given to you by the trainer to open Doccano in your browser, log in and try the various tasks.\n",
    "\n",
    "> Note: If you are following this notebook outside the context of the CDH Data School 2020, then you can try the [official Doccano demo](https://doccano.herokuapp.com/demo/named-entity-recognition/).\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "---\n",
    "## Converting Between spaCy and Doccano Formats\n",
    "\n",
    "In order to load the named entities spaCy produces into Doccano, it is necessary to transform the spaCy output into **Doccano's JSONL format**, where each document sits on a newline:\n",
    "\n",
    "```\n",
    "{\"text\": \"We start with a letter Charles Lyell sent to Darwin in 1865 where he discusses the latest revision of his book Elements of Geology and relates to Darwin his discussions with various aquaintances about Darwin's own book On the Origin of Species. This letter has been transcribed and annotated in TEI-XML by editors from the DCP team in Cambridge.\", \"labels\": [[23, 36, \"PERSON\"], [45, 51, \"PERSON\"], [55, 59, \"DATE\"], [111, 130, \"WORK_OF_ART\"], [146, 152, \"PERSON\"], [201, 207, \"PERSON\"], [219, 243, \"WORK_OF_ART\"], [323, 326, \"ORG\"], [335, 344, \"GPE\"]]}\n",
    "```\n",
    "\n",
    "I have written a utility class called **`DoccanoNamedEnts`** to help us bootstrap a first-pass of named entity recognition into Doccano for annotation. You can browse the code in [doccano/doccano_named_ents.py](doccano/doccano_named_ents.py).\n",
    "\n",
    "NB: This script is written specifically for our usage with letters marked up in Cambridge-style TEI. To use it for your own projects you may need to make some modifications to extract relevant data from your XML files. It's also designed for educational rather than production use.\n",
    "\n",
    "Here is how to use it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the DoccanoNamedEnts class\n",
    "from doccano.doccano_named_ents import DoccanoNamedEnts\n",
    "\n",
    "# Create an instance with the whole set of Henslow data\n",
    "labels = DoccanoNamedEnts('data/henslow')\n",
    "\n",
    "# Print the NER labels in Doccano format if you want to copy and paste it\n",
    "labels.print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write the labels in Doccano format to JSONL file\n",
    "labels.to_file('output/doccano_ner_henslow.jsonl')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This JSONL file can now be uploaded to Doccano ready for annotation. \n",
    "\n",
    "> Doccano has an upload limit of 1MB on the file size so you may need to batch your output into multiple files. Make sure that all your XML documents have valid non-empty transcriptions as Doccano will reject any file that contains empty fields e.g. `{\"text\": \"\", \"labels\": []}`. You file must not contain any blank lines so check that before you try to upload.\n",
    "\n",
    "NB: On the Doccano set up for this workshop, you do not have access to the import feature. To try this out you will need to install your own version."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "---\n",
    "## Summary\n",
    "\n",
    "In this notebook:\n",
    "\n",
    "- A machine learning algorithm learns from the data it has been trained on and stores its experience as a **statistical model**. The trained model can make **predictions** for text it has never seen before. For example, it can predict which named entities are the most probable for a given token (or span) in a given context.\n",
    "- The model is not always correct in its predictions and may have to be **trained** again to improve its performance.\n",
    "- Training takes place by feeding the model **training data**, which is data that has been accurately **labelled** by humans.\n",
    "- We can use **text annotation software** to make it easier for humans to label training data by hand.\n",
    "- New predictions must be **evaluated** (validated) with a sub-set of training data reserved for the purpose.\n",
    "\n",
    "In the [next notebook](4-updating-the-model-on-henslow-data.ipynb) we will do a **codewalk** through an example of re-training the model with Henslow data. Using Python, we will clean up the tokenization, correct named entity predictions and add a new named entity type."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}