DataAnalysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Quality of food offered in the World\n",
    "\n",
    "In this notebook we explore the [Open Food Facts database](https://world.openfoodfacts.org/). After loading, observing and cleaning the data in part 1, we start our analysis in part 2 with the aim of updating our readme to see if what we wanted to do is feasable. In this second part, we uncover insights about the nutritional quality of food consumed around the world. Food habits vary among countries, we would like to point out which areas offer the least healthy options by observing the products offered there.\n",
    "\n",
    "More particularly, we will look at the following points:\n",
    "* How restrictions concerning additives in different countries affects the presence of these substances in foods around the world. We'll also look at whether banning an ingredient in a certain country reduces its presence in other places?\n",
    "* We will consider the amounts of sugar and fat availble in foods around the world.\n",
    "* We will look at how widespread ingredients like palm oil are."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Loading, Observing and Cleaning the Data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading the data and imports.\n",
    "\n",
    "First, we import the necessary packages for our analysis. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import json\n",
    "import folium as folium\n",
    "from wordcloud import WordCloud\n",
    "from matplotlib import pyplot\n",
    "import matplotlib.path as mplPath\n",
    "from os import listdir\n",
    "from os.path import isfile, join\n",
    "from ast import literal_eval\n",
    "import seaborn as sns\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# make figures larger for better viewing\n",
    "plt.rc('figure', figsize=[18,10])\n",
    "plt.style.use('fivethirtyeight')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, we load the data and take a first look at it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "ename": "FileNotFoundError",
     "evalue": "File b'./data/en.openfoodfacts.org.products.csv' does not exist",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mFileNotFoundError\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-3-85d38b3b239b>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[0mopen_food_facts_path\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mdata_path\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;34m'en.openfoodfacts.org.products.csv'\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 4\u001b[0;31m \u001b[0mdf\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mpd\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mread_csv\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mopen_food_facts_path\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msep\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'\\t'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlow_memory\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mencoding\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'utf-8'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      5\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshape\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      6\u001b[0m \u001b[0mdf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mhead\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pandas/io/parsers.py\u001b[0m in \u001b[0;36mparser_f\u001b[0;34m(filepath_or_buffer, sep, delimiter, header, names, index_col, usecols, squeeze, prefix, mangle_dupe_cols, dtype, engine, converters, true_values, false_values, skipinitialspace, skiprows, nrows, na_values, keep_default_na, na_filter, verbose, skip_blank_lines, parse_dates, infer_datetime_format, keep_date_col, date_parser, dayfirst, iterator, chunksize, compression, thousands, decimal, lineterminator, quotechar, quoting, escapechar, comment, encoding, dialect, tupleize_cols, error_bad_lines, warn_bad_lines, skipfooter, skip_footer, doublequote, delim_whitespace, as_recarray, compact_ints, use_unsigned, low_memory, buffer_lines, memory_map, float_precision)\u001b[0m\n\u001b[1;32m    707\u001b[0m                     skip_blank_lines=skip_blank_lines)\n\u001b[1;32m    708\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 709\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0m_read\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfilepath_or_buffer\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkwds\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    710\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    711\u001b[0m     \u001b[0mparser_f\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__name__\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pandas/io/parsers.py\u001b[0m in \u001b[0;36m_read\u001b[0;34m(filepath_or_buffer, kwds)\u001b[0m\n\u001b[1;32m    447\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    448\u001b[0m     \u001b[0;31m# Create the parser.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 449\u001b[0;31m     \u001b[0mparser\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mTextFileReader\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfilepath_or_buffer\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwds\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    450\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    451\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mchunksize\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0miterator\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pandas/io/parsers.py\u001b[0m in \u001b[0;36m__init__\u001b[0;34m(self, f, engine, **kwds)\u001b[0m\n\u001b[1;32m    816\u001b[0m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moptions\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'has_index_names'\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mkwds\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'has_index_names'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    817\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 818\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_make_engine\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mengine\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    819\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    820\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mclose\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pandas/io/parsers.py\u001b[0m in \u001b[0;36m_make_engine\u001b[0;34m(self, engine)\u001b[0m\n\u001b[1;32m   1047\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_make_engine\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mengine\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'c'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1048\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mengine\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m'c'\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1049\u001b[0;31m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_engine\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mCParserWrapper\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mf\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moptions\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1050\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1051\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mengine\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m'python'\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pandas/io/parsers.py\u001b[0m in \u001b[0;36m__init__\u001b[0;34m(self, src, **kwds)\u001b[0m\n\u001b[1;32m   1693\u001b[0m         \u001b[0mkwds\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'allow_leading_cols'\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mindex_col\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1694\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1695\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_reader\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mparsers\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTextReader\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msrc\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwds\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1696\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1697\u001b[0m         \u001b[0;31m# XXX\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32mpandas/_libs/parsers.pyx\u001b[0m in \u001b[0;36mpandas._libs.parsers.TextReader.__cinit__\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;32mpandas/_libs/parsers.pyx\u001b[0m in \u001b[0;36mpandas._libs.parsers.TextReader._setup_parser_source\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;31mFileNotFoundError\u001b[0m: File b'./data/en.openfoodfacts.org.products.csv' does not exist"
     ]
    }
   ],
   "source": [
    "data_path = './data/'\n",
    "open_food_facts_path = data_path + 'en.openfoodfacts.org.products.csv'\n",
    "\n",
    "df = pd.read_csv(open_food_facts_path, sep='\\t', low_memory = False, encoding='utf-8')\n",
    "print(df.shape)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have 692133 entries with 173 columns in the dataset. Everything fits in memory so we can easily handle all of the data in terms of size."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Removing empty columns."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Taking a brief look at the data, we notice that a lot of values are missing. Indeed, the data from open food facts is filled by regular users inputing information from the packaging of their food products. This leads to a lot of missing values as people do not generally fill all 173 columns available in the final database.\n",
    "\n",
    "Then what columns should we drop? We'll first look at what percentage of values are `nan` for each column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "number_of_rows = df.shape[0]\n",
    "amount_of_nan = df.isna().sum().apply(lambda x: x / number_of_rows * 100).sort_values(ascending = False)\n",
    "amount_of_nan.head(20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will drop columns whose nan percentage is too high. We find all columns that have more than 99% of nan values. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "threshold = 99\n",
    "cols_with_more_than_threshold = [col for col, percentage in amount_of_nan.iteritems() if percentage > threshold]\n",
    "print(cols_with_more_than_threshold)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "colors = amount_of_nan.apply(lambda percentage: 'b' if percentage < threshold else 'r')\n",
    "plt.rcParams.update({'font.size': 7})\n",
    "plt.xticks(rotation='vertical')\n",
    "plt.bar(amount_of_nan.index, amount_of_nan, color=colors)\n",
    "plt.title('percentage of Nan by column')\n",
    "plt.ylabel('percentage of Nan')\n",
    "plt.xlabel('colum names')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All these columns more than 99% empty (colored in red) will be dropped."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "df = df.drop(cols_with_more_than_threshold, axis=1)\n",
    "df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dropping uninteresting columns\n",
    "\n",
    "Open Food Facts offers a large array of columns per product. Some are not interesting considering the subject of our analysis. As such, we will drop the following columns:\n",
    "* `emb_codes` and `emb_codes_tags` refer to packaging numbers in France.\n",
    "* `pnns_groups_1` and `pnns_groups_2` only include data in French related to the 'PNNS' dietary guidelines.\n",
    "* `states`, `states_tags`and `states_en` refer to the state of the entry in the database (tags that still need to be completed).\n",
    "* `image_url`, `image_small_url`, `image_ingredients_url`, `image_ingredients_small_url`, `image_nutrition_url` and `image_nutrition_small_url` give urls to pictures of the food. We will not use this.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "uninteresting_columns_to_drop = ['emb_codes', 'emb_codes_tags', 'pnns_groups_1', 'pnns_groups_2', 'states', 'states_tags', 'states_en', 'image_url',\\\n",
    "                   'image_small_url', 'image_ingredients_url', 'image_ingredients_small_url',\\\n",
    "                   'image_nutrition_url', 'image_nutrition_small_url']\n",
    "\n",
    "df = df.drop(uninteresting_columns_to_drop, axis = 1)\n",
    "df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dealing with date types"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `created_t`, `created_datetime`, `last_modified_t` and `last_modified_datetime` columns deal with dates. We will drop `created_t` and `last_modified_t` as they contain redundant information already in the other two columns.\n",
    "\n",
    "We will parse the dates in `created_datetime` and `last_modified_datetime` into the datetime type for esier use in our analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['created_datetime'] = pd.to_datetime(df['created_datetime'], errors = 'coerce', infer_datetime_format = True)\n",
    "df['last_modified_datetime'] = pd.to_datetime(df['last_modified_datetime'], errors = 'coerce', infer_datetime_format = True)\n",
    "\n",
    "df = df.drop(['created_t', 'last_modified_t'], axis=1)\n",
    "df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Removing duplicate columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Even after all this cleaning, looking at the column names, it seems that a lot of them refer to the same thing. We will check for redundancy among similar column names and drop unnecessary values.\n",
    "\n",
    "We explore the content of the following groups of columns that seem to have redundant values:\n",
    "* `categories`, `categories_tags` and `categories_en` are all related to the kind of food the product belongs to.\n",
    "* `countries`, `countries_tags` and `countries_en` refer to the country of origin.\n",
    "* `packaging` and `packaging_tags` give info about the type of packaging.\n",
    "* `brands` and `brands_tags` contain the brand of the product.\n",
    "* `origins` and `origins_tags`\n",
    "* `manufacturing_places` and `manufacturing_places_tags`\n",
    "* `labels`, `labels_tags` and `labels_en`\n",
    "* `traces`, `traces_tags` and `traces_en`\n",
    "* `serving_size` and `serving_quantity`\n",
    "* `additives`, `additives_tags` and `additives_en`\n",
    "* `main_category` and `main_category_en`\n",
    "\n",
    "For each of these groups of column we will check that they indeed contain redundant information and remove all columns that are then useless."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This function will print the ranking of the given columns depending on their proportion of nan values\n",
    "# It also return a few values where none of the columns are nan to check for redundant data.\n",
    "def analyze_columns(list_of_columns):\n",
    "    print(amount_of_nan[list_of_columns].sort_values(ascending = False))\n",
    "    return df[list_of_columns].dropna().reset_index(drop = 'True').head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Category columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "analyze_columns(['categories', 'categories_tags', 'categories_en'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Values among these 3 columns seem largely redundant. We will only keep the `categories_tags` column for its english formatting and convert it to a list of strings for easier analysis later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "redundant_columns_to_drop = ['categories_en', 'categories']\n",
    "columns_to_convert = ['categories_tags']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Country columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "analyze_columns(['countries', 'countries_tags', 'countries_en'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These columns all cover the same amount of products. We will then only keep `countries_en` for its clearer formatting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "redundant_columns_to_drop += ['countries', 'countries_tags']\n",
    "columns_to_convert += ['countries_en']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Packaging columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "analyze_columns(['packaging', 'packaging_tags'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We keep `packaging` and convert it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "redundant_columns_to_drop += ['packaging_tags']\n",
    "columns_to_convert += ['packaging']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Brand columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "analyze_columns(['brands', 'brands_tags'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We keep `brands_tags` as it is cleaner than `brands` for a similar number of `nan` values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "redundant_columns_to_drop += ['brands']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Origin columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "analyze_columns(['origins', 'origins_tags'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We keep `origins`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "redundant_columns_to_drop += ['origins_tags']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Manufacturing columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "analyze_columns(['manufacturing_places', 'manufacturing_places_tags'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We keep only `manufacturing_places`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "redundant_columns_to_drop += ['manufacturing_places_tags']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Label columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "analyze_columns(['labels', 'labels_tags', 'labels_en'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`labels_tags` and `labels_en` cover the same rows, we will keep `labels_en` for its formatting and convert it. `labels` is unfortunately in French."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "redundant_columns_to_drop += ['labels', 'labels_tags']\n",
    "columns_to_convert += ['labels_en']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Traces columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "analyze_columns(['traces', 'traces_tags', 'traces_en'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`traces_tags` is better formatted and contains less `nan` values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "redundant_columns_to_drop += ['traces', 'traces_en']\n",
    "columns_to_convert += ['traces_tags']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Serving size columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "analyze_columns(['serving_size', 'serving_quantity'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, clearly `serving_quantity` is the right choice as it has less `nan` values and is formatted more simply."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "redundant_columns_to_drop += ['serving_size']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Additives columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "analyze_columns(['additives', 'additives_tags', 'additives_en'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looking at the proportion of `nan` values, the `additives` column seems to be the best choice. However, its content is not what we expected. We will instead keep `additives_tags`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "redundant_columns_to_drop += ['additives', 'additives_en']\n",
    "columns_to_convert += ['additives_tags']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Main category columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "analyze_columns(['main_category', 'main_category_en'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Their percentage of `nan` values are identical, we keep `main_category_en`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "redundant_columns_to_drop += ['main_category']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dropping columns and performing conversions\n",
    "\n",
    "Finally, we remove all the columns from the dataframe and we convert several columns from strings to list of strings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(redundant_columns_to_drop)\n",
    "print(columns_to_convert)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = df.drop(redundant_columns_to_drop, axis = 1)\n",
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for col in columns_to_convert:\n",
    "    df[col] = df[col].astype(str).apply(lambda s: s if s == 'nan' else s.split(','))\n",
    "    \n",
    "df.info(verbose = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Removing illogical entries\n",
    "\n",
    "Even after having cleaned all this we should be careful to remove any illogical entries. Indeed, some values are obvious wrong entries.\n",
    "\n",
    "First, we consider values of the form `carbohydrates_100g` (this indicates the amount of carbohydrates per 100g). This value must necessarely be between 0 and 100. We remove all entries not in this range."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "per_100g_columns = ['fat_100g', 'saturated-fat_100g', 'monounsaturated-fat_100g',\\\n",
    "                    'polyunsaturated-fat_100g', 'trans-fat_100g', 'cholesterol_100g',\\\n",
    "                    'carbohydrates_100g', 'sugars_100g', 'fiber_100g', 'proteins_100g',\\\n",
    "                    'salt_100g', 'sodium_100g']\n",
    "\n",
    "for col in per_100g_columns:\n",
    "    df = df[((0 <= df[col]) & (df[col] <= 100)) | df[col].isna()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `energy_100g` column indicates the amount of energy per 100g in kJ. Fat provides the most energy per gram with 37.7kJ in a gram of fat [\\[source\\]](http://www.nutritionaustralia.org/national/resource/balancing-energy-and-out). As such, a product containing only fat would provide 100 * 37.7 = 3770kJ of energy. We remove any entry with a value higher than this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = df[((0 <= df['energy_100g']) & (df['energy_100g'] <= 3770)) | df['energy_100g'].isna()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sugars are actually a subset of carbohydrates. This means that we cannot have more sugars per 100g than carbohydrates. Similarly, saturated fats are a subset of fat.\n",
    "\n",
    "We remove entries where the subset has a higher content than its category."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = df[df['sugars_100g'] <= df['carbohydrates_100g']]\n",
    "df = df[df['saturated-fat_100g'] <= df['fat_100g']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Looking at the cleaned data\n",
    "\n",
    "## For which countries do we have data?\n",
    "\n",
    "Our analysis had the aim of covering the whole world as Open Food Facts is not limited to any country. It is important that we check if this is possible by looking at how much data is availble in every country and if we have enough to do a world analysis (we'll see that this is not the case and we will have to pivot our analysis).\n",
    "\n",
    "We first define a function so that we can analyze different metrics aggregated per country."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# metric is a function row -> value. It might be any measurement we want to do (amount of fat, ...)\n",
    "# aggrate_function if a function list(value) -> score (median, sum, ...) leading to the final score for one country\n",
    "# we do not keep the values if the list has less than n_mini items, to deal with countries with not enough data\n",
    "def group_by_country_and_aggregate(df, metric, metric_name, aggregate_function, n_mini):\n",
    "    \n",
    "    # fill a dictionnary {country: [metric_product1, metric_product2, metric_product3, ...]}\n",
    "    per_country = {}\n",
    "\n",
    "    # helper function:\n",
    "    # for each country mentionned in row, add the value (using the metric) to the list of values for this country\n",
    "    def add_value(row):\n",
    "        \n",
    "        # if the metric cannot be computed, return\n",
    "        val = metric(row)\n",
    "        \n",
    "        try:\n",
    "            if pd.isnull(val):\n",
    "                return\n",
    "        except:\n",
    "            if not val:\n",
    "                return\n",
    "        \n",
    "        for country in row[\"countries_en\"]:\n",
    "            if not country in per_country.keys():\n",
    "                per_country[country] = []\n",
    "            per_country[country] += [val]\n",
    "\n",
    "    # calls the helper for all the rows\n",
    "    for _, row in df.iterrows():\n",
    "        add_value(row)\n",
    "    \n",
    "    # filter the lists that have les than n_mini elements\n",
    "    filtred_dict = {}\n",
    "    for k in per_country.keys():\n",
    "        l = per_country[k]\n",
    "        if len(l) >= n_mini:\n",
    "            filtred_dict[k] = l\n",
    "    \n",
    "    # get the list of countries\n",
    "    countries = list(filtred_dict.keys())\n",
    "    \n",
    "    # call our aggregation function for each list of more than n_mini elements \n",
    "    final_scores =  [aggregate_function(filtred_dict[c]) for c in countries]\n",
    "\n",
    "    # finally, build a DF with the result and return it\n",
    "    return pd.DataFrame({\"country\": countries, metric_name: final_scores}).set_index(\"country\").sort_values(metric_name, ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since we want to analyze the quality of foods in the world, it is crucial that we understand which countries have the most entries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "entries_per_country = group_by_country_and_aggregate(df, lambda x: 1, \"products per country\", np.sum, 1000)\n",
    "entries_per_country.plot.bar(logy=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The top 3 countries for which we have data are France, the US and Switzerland. Germany, Belgium, Spain and the UK have a lot of data points also but for other countries we have much fewer information.\n",
    "\n",
    "In particular, we don't have as many values as we expected for other areas in the world (Asia and Africa is especially absent). We might have to shift our analysis from the whole world to the set of countries for which we have data (North Amercia and Western Europe).\n",
    "\n",
    "It could be interesting to look at who are the most important contributors to the database also as this might explain the differences in quantity of data per country."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we do not need the whole dataframe for this\n",
    "mini_df = df[['creator','code']]\n",
    "mini_df.groupby('creator').count().sort_values('code', ascending=False).head(20).plot.bar(logy=True)\n",
    "plt.title('number of products per contributor')\n",
    "plt.xlabel('contributor')\n",
    "plt.ylabel('number of contributions')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Indeed it is very interesting to look at the contributors!\n",
    "\n",
    "We find explanations for the top 3 countries:\n",
    "* France: Open Food Facts started as a French project (which explains why we saw so many values in French during the cleaning part). On top of this, kiliweb is the most important contributor to the data. They are a [french web design and app developper](https://kiliweb.fr/), most notably for us they developped [Yuka](https://itunes.apple.com/fr/app/yuka-analyse-de-produits-alimentaires/id1092799236?mt=8) which is currently the number 1 health app on the French app store and allows for scanning of food product bar codes (it's basically a pretty front-end for Open Food Facts).\n",
    "* USA: The US department of agriculture gathers a lot of data on food products sold in the United States. They have transferred a lot of this data over to Open Food Facts.\n",
    "* Switzerland: Yuka is also the number one health app on the Swiss app store (so some of the kiliweb entries also transfer to Switzerland) and very interestingly the fourth biggest contributor is [openfood.ch](https://www.foodrepo.org/) (now renamed foodrepo.org) that is an Open Food Facts cloned focused on Switzerland that was actually [launched by EPFL last year](https://actu.epfl.ch/news/food-data-at-your-fingertips-4/)! They must have included their data to Open Food Facts.\n",
    "\n",
    "To get an even better idea of which regions are interesting for our analysis we visualize which countries have more than 1000 entries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "entries_per_country_dict = entries_per_country.to_dict()['products per country']\n",
    "entries_per_country = entries_per_country.reset_index()\n",
    "\n",
    "# here we add 0 values to all countries which have less than a 1000 entries and\n",
    "# 1 to countries with more than a 1000 entries\n",
    "# this section is needed because by default choropleth displays missing values as max value\n",
    "world_map = './data/world-countries.json'\n",
    "countries = []\n",
    "entries_count = []\n",
    "json_object = json.load(open(world_map))\n",
    "\n",
    "# we iterate over all countries in our world geo json\n",
    "for country in json_object['features']:\n",
    "    country_name = country['properties']['name']\n",
    "    countries.append(country_name)\n",
    "    if country_name in list(entries_per_country_dict.keys()):\n",
    "        entries_count.append(1)\n",
    "    else:\n",
    "        entries_count.append(0)\n",
    "\n",
    "# we recreate a dataframe with all countries\n",
    "full_entries_per_country_dict = pd.DataFrame({\"country\": countries, 'products per country': entries_count})\n",
    "\n",
    "map = folium.Map(location=[20, 0], zoom_start=2)\n",
    "map.choropleth(geo_data=world_map, data=full_entries_per_country_dict,\n",
    "             columns=['country', 'products per country'],\n",
    "             key_on='feature.properties.name',\n",
    "             fill_color='YlGn')\n",
    "map.save('map.html')\n",
    "map"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The map is available [here](map.html) for viewing on github.\n",
    "\n",
    "Clearly only North America and Europe are valid for our analysis.\n",
    "\n",
    "We will definitely update our README to shift our analysis to North America and Western Europe."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Is the data current?\n",
    "\n",
    "Open Food Facts was [launched in 2012](https://en.wikipedia.org/wiki/Open_Food_Facts). We would like to check whether the data on the wesite is still current as we wish to analyze foods that are available right now.\n",
    "\n",
    "Let's see how many new entries are added per year on the website."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['created_datetime'].dropna().map(lambda x: int(x.year)).value_counts().sort_index().plot.bar()\n",
    "plt.title('number of new entries per year')\n",
    "plt.xlabel('year')\n",
    "plt.ylabel('number of new entries')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is good news for our analysis! More and more data is added each year so we should not have to worry about only dealing with older products that might not even be available anymore. This is really not a concern when we see that alsmost all entries are actually from the last two years.\n",
    "\n",
    "On top of this, we can also check whether or not the data has been updated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['last_modified_datetime'].map(lambda x: x.year).value_counts().sort_index().plot.bar()\n",
    "plt.title('number of modified entries per year')\n",
    "plt.xlabel('year')\n",
    "plt.ylabel('number of modified entries')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We really don't have to worry about outdated data. Indeed, most updates were done this year."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A first look at the data points most interesting for our questions\n",
    "\n",
    "Let's now see if we can get basic insights on the colummns most interesting for the questions we want to consider."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Numerical values\n",
    "\n",
    "Let's see if our numerical values have any important correlations so that we can better understand what is inside our data (and our food)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "numeric_df = df.select_dtypes(include = 'float64')\n",
    "sns.heatmap(numeric_df.corr(), cmap ='hot', robust = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Most of our values are poorly correlated.\n",
    "\n",
    "Most of the fats (total trans, saturated, unsaturated), as well as the energy, are correlated, which is normal.\n",
    "\n",
    "Some of the vitamins are correlated together, mostly A <-> D and C <-> D. This correlation does not seem correct, since D vitamine is found mostly in fishes or animal products, whereas A and C vitamin is found mostly in vegetables.\n",
    "\n",
    "The general conclusion we can draw is that the values are not or poorly correlated, and that this simple heatmap is not sufficient to understand the correlations. We checked the relevence of some of the correlations on the web: it seems that the majority of the correlations are not real, but due to the fact that some columns contains almost only zeroes in the dataframe."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What is Palm Oil? How common is it in European foods? Is it bad for your health?\n",
    "Palm Oil is an oil extracted from the fruit of the trees called African Oil Palms. It is used in food products, soap and biodiesel. Palm Oil's high consumption and production has been at the center of multiple <a href=\"https://www.bbc.co.uk/newsround/39492207\">controversies</a> in Europe. We will study the extent of Palm Oil use is food products in European and North American countries and will try to understand where Palm Oil's bad reputation comes from."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This method returns the percentage of products containing palm oil per value in a particular column\n",
    "# The threshold argument indicates how many entries for a particular value are needed to be a valid category\n",
    "def percentage_of_palm_based_products(column_name, threshold):\n",
    "    # we only need 2 columns for this, the one we group by (column_name) and the\n",
    "    # one indicating whether palm oil is used in a product (ingredients_from_palm_oil_n)\n",
    "    df_by_brands = df[[column_name,'ingredients_from_palm_oil_n']]\n",
    "    df_by_brands = df_by_brands.dropna()\n",
    "    df_by_brands.ingredients_from_palm_oil_n = df_by_brands.ingredients_from_palm_oil_n\n",
    "    df_by_brands = df_by_brands.groupby([column_name])\n",
    "\n",
    "    values_dict = {column_name: [], 'percentages_of_palm_based_products':[]}\n",
    "    for name, group_ingredients_palm in df_by_brands:\n",
    "        number_of_products = len(group_ingredients_palm)\n",
    "        # check if this value of the column contains more than the threshold\n",
    "        if number_of_products > threshold:\n",
    "            n_products_containing_palm = group_ingredients_palm['ingredients_from_palm_oil_n'].astype(bool).sum()\n",
    "            percentage = n_products_containing_palm / number_of_products\n",
    "            # only keep values with high percentages of palm oil\n",
    "            if percentage > 0.02:\n",
    "                values_dict['percentages_of_palm_based_products'].append(percentage)\n",
    "                values_dict[column_name].append(name)\n",
    "    return pd.DataFrame.from_dict(values_dict).set_index(column_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How common is Palm Oil in European food products?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# here we try to find the number of products containing palm oil\n",
    "number_of_products_with_palm_oil_data = len(df.ingredients_from_palm_oil_n.dropna())\n",
    "number_of_products_with_palm_oil_based_ingredients = df.ingredients_from_palm_oil_n.dropna().astype(bool).sum()\n",
    "percentage_of_products_containing_palm_oil = 100 * number_of_products_with_palm_oil_based_ingredients / number_of_products_with_palm_oil_data\n",
    "\n",
    "print('percentage of products containing palm oil:', round(percentage_of_products_containing_palm_oil,2), '%')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see what brands in the dataset use palm oil the most."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# this function converts letter grades a - b - c - d - e to integer values 0 - 1 - 2 - 3 - 4\n",
    "def grade_to_score(x):\n",
    "    return ord(x) - ord('a') + 1\n",
    "\n",
    "nutrition_grade_fr_df = df.dropna(subset=['nutrition_grade_fr'])\n",
    "#print(nutrition_grade_fr_df.nutrition_grade_fr.unique())\n",
    "nutrition_grade_fr_df.nutrition_grade_fr = nutrition_grade_fr_df.nutrition_grade_fr.apply(grade_to_score)\n",
    "nutrition_grade_fr_df.boxplot(column='nutrition_grade_fr', by='ingredients_from_palm_oil_n')\n",
    "plt.xlabel(\"Number of palm oil based ingredients\")\n",
    "plt.ylabel(\"Nutrition grade (interval and quartiles)\")\n",
    "plt.yticks([a+1 for a in range(5)], [\"A\", \"B\", \"C\", \"D\", \"E\"])\n",
    "plt.title(\"Nutrition grade vs palm oil\")\n",
    "plt.suptitle(\"\")\n",
    "plt.savefig('nutrition_grade_palm_oil.png', transparent=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see here that products containing 1 or 2 ingredients containing palm oil are less healthy on average than products that do not contain palm oil. This is mostly caused by the categrories of products that use palm oil which are mostly unhealthy kinds of foods. Palm Oil itself is not particularly dangerous compared to other edible oils."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.rcParams.update({'font.size': 20})\n",
    "palm_broducs_by_brand = percentage_of_palm_based_products('main_category_en', 200)\n",
    "palm_broducs_by_brand.sort_values(by='percentages_of_palm_based_products', ascending=True).plot.barh()\n",
    "plt.ylabel('')\n",
    "plt.legend([\"% of palm oil based products\"])\n",
    "plt.title('products containing palm oil based ingredients vs product category')\n",
    "plt.savefig('images/palm_oil_per_category.png', transparent=True, bbox_inches='tight', dpi = 300)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Breakfast is the category with the highest palm oil percentage, this makes sense when we consider things like spreads (which unfortunately also have their own category, this is proaly a slight categorization problem). It's not surprising to see that snacks (whether sugary or salty )\n",
    "\n",
    "Data concerning palm oil seems to make sense here. This is something we can pursue in our analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "palm_broducs_by_brand = percentage_of_palm_based_products('brands_tags', 300)\n",
    "palm_broducs_by_brand.sort_values(by='percentages_of_palm_based_products', ascending=True)[-10:].plot.barh()\n",
    "plt.ylabel('')\n",
    "plt.legend([\"% of palm oil based products\"])\n",
    "plt.title('products containing palm oil vs product brand')\n",
    "plt.savefig('images/palm_oil_per_brand.png', transparent=True, bbox_inches='tight', dpi = 300)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The top 3 values are Lu, Knorr and Migros. This makes sense as the first brand is a mass producer of cakes and biscuits (both of which often contain inexpensive vegetable oils such as palm oil). Same for Knorr and Migros, they offer a large selection of prepared foods that make us of palm oil to cut down on costs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Why does palm oil have a bad reputation?\n",
    "### Where is it produced?  Economic situation of countries / regions where it is produced"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# use data containing 2017 country gross domestic product per capita downloaded here: https://data.worldbank.org/indicator/NY.GDP.MKTP.CD\n",
    "gdp_per_country_path = data_path + 'API_NY.GDP.PCAP.CD_DS2_en_csv_v2_10224851.csv'\n",
    "gdp_per_country = pd.read_csv(gdp_per_country_path, low_memory = False, encoding='utf-8').dropna().reset_index()\n",
    "\n",
    "# use data containing palm oil mill coordinates downloaded here: http://data.globalforestwatch.org/datasets/ed8d5951b2a4482a9e62c4fe0bc23b5f_27\n",
    "palm_oil_path = data_path + 'Palm_oil_mills.csv'\n",
    "palm_mills = pd.read_csv(palm_oil_path, low_memory = False, encoding='utf-8').dropna().reset_index()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# this file contains topologies for all countries in the world_map\n",
    "world_map = './data/world-countries.json'\n",
    "json_object = json.load(open(world_map))\n",
    "countries = []\n",
    "all_valid_countries = []\n",
    "gdps = []\n",
    "\n",
    "# we add 0 values to countries for which we have no data (small number of small countries)\n",
    "for country in json_object['features']:\n",
    "    country_name = country['properties']['name']\n",
    "    all_valid_countries.append(country_name)\n",
    "    if country_name not in gdp_per_country['Country Name'].unique():\n",
    "        gdps.append(0)\n",
    "        countries.append(country_name)\n",
    "\n",
    "# we discard all values above 85000 gdp per capita. The dataset contains higher values for some regions that are not\n",
    "# countries are not of interest to us\n",
    "gdp_per_country = gdp_per_country.loc[gdp_per_country['2017'] < 85000]\n",
    "gdp_per_country_unknown = pd.DataFrame({\"Country Name\": countries, '2017': gdps}).reset_index()\n",
    "\n",
    "# create map with colores countries based on their gdp\n",
    "map = folium.Map(location=[20, 0], zoom_start=2)\n",
    "map.choropleth(geo_data=world_map, data=gdp_per_country.append(gdp_per_country_unknown),\n",
    "             columns=['Country Name', '2017'],\n",
    "             key_on='feature.properties.name',\n",
    "             fill_color='BuGn')\n",
    "\n",
    "# add red circle indication Palm oil mills on the world map \n",
    "for index, row in palm_mills.iterrows():\n",
    "    folium.CircleMarker(\n",
    "    location=[row['latitude'], row['longitude']],\n",
    "    radius=1,\n",
    "    popup=row['mill_name_'],\n",
    "    color='red',\n",
    "    fill=True,\n",
    "    fill_color='#3186cc'\n",
    "    ).add_to(map)\n",
    "map.save('map.html')\n",
    "map"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we see in the map above, palm oil plantations and mills are mostly in South East Asia with some in Central and South America. The countries from these regions are less economically developed nations compared to European and North American countries.\n",
    "\n",
    "For example, Indonesia has more than 80% of the world's palm oil mills but also has a GDP per capita 20x smaller than Switzerland's"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# In this cell we use country geojson files to detect in which country a coordinate (longitude, latitude) is.\n",
    "# We could have used the Google Maps API but the free version is slow (1 request per second)\n",
    "directory = './data/countries'\n",
    "onlyfiles = [f for f in listdir(directory) if isfile(join(directory, f))]\n",
    "\n",
    "# here we read and store each countries geojson file and convert it to a mplPath polygon\n",
    "country_dict = {}\n",
    "for file in onlyfiles:\n",
    "\n",
    "    with open(directory + '/' + file) as f:\n",
    "        data = json.load(f)\n",
    "        country_name = data['features'][0]['properties']['name']\n",
    "        pos = (96.25,-75)\n",
    "        polygons = data['features'][0]['geometry']['coordinates']\n",
    "        type_polygon = data['features'][0]['geometry']['type']\n",
    "        inside_polygon = False\n",
    "        country_dict[country_name] = []\n",
    "        if type_polygon == 'Polygon':\n",
    "            polygon = polygons[0]\n",
    "            bbPath = mplPath.Path(polygon)\n",
    "            country_dict[country_name].append(bbPath)\n",
    "        else:\n",
    "            for poly in polygons:\n",
    "                poly = poly[0]\n",
    "                bbPath = mplPath.Path(poly)\n",
    "                country_dict[country_name].append(bbPath)\n",
    "                \n",
    "# this function is used to detect in which country a specific coordinate is\n",
    "# it returns 'Unknown' if no country is found\n",
    "def convert(lat, lon, country_dict):\n",
    "    for country_name, polygons in country_dict.items():\n",
    "        for polygon in polygons:\n",
    "            inside_polygon = polygon.contains_point((lon,lat))\n",
    "            if inside_polygon:\n",
    "                return country_name\n",
    "    return 'Unknown'\n",
    "\n",
    "palm_mills['country'] = palm_mills.apply(lambda row: convert(-row.latitude, row.longitude, country_dict), axis=1)\n",
    "counts_per_country = palm_mills['country'].value_counts()\n",
    "counts_per_country"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our suspicions are confirmed, Indonesia is by far the leading producer of palm oil. Some mills's countries were not found (hence the 511 Unknown values). This is due to South Asia having many small islands which are often missing from our countries geogson files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# this function return the row of a specific country\n",
    "def filter_(country_name):\n",
    "    return gdp_per_country[gdp_per_country['Country Name'] == country_name]\n",
    "\n",
    "Indonesia_gdp = filter_('Indonesia').iloc[0]['2017']\n",
    "Swiss_gdp = filter_('Switzerland').iloc[0]['2017']\n",
    "print('Indonesia GDP per capita:', Indonesia_gdp)\n",
    "print('Swiss GDP per capita:', Swiss_gdp)\n",
    "print('Indonesia GDP per capita is', int(Swiss_gdp/Indonesia_gdp), \"times smaller than Switzerland's\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see here the important economical differences between countries producing palm oil and those consuming it (North America and Europe)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Ok, but what's wrong with that?\n",
    "Palm oil is produced in countries which are less economically developed than the countries where the products containing it are sold. This often means the the main producers (Indonesia, Myanmar, ...) have less strict and enforced environmental policies than western countries. This leads palm oil producing companies to expand their plantations without control. The huge demand for palm oil in food products is the cause of a rapid expansion in palm oil plantation's land use. This is the reason why palm oil is blamed for:\n",
    "\n",
    "- Deforestation\n",
    "- Habitat destruction (orang-utans, tigers, elephants)\n",
    "- Air and water pollution\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### What are the solutions?\n",
    "As we saw earlier, many well know Western and Swiss companies (Nestle, Kellogs, Lindt, ...) massively use palm oil in their food products. Consumers should pressure these companies into choosing environmentally friendly palm oil producers which follow strict guidelines about producing palm oil sustainably."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sugar\n",
    "\n",
    "Similarly to palm oil, a lot of our questions were related to sugar. We will try and get basic insights as a sanity check to make sure we can follow through with this in the future. Let's start by looking at the median sugar content of products by country."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mean_sugar_per_country = group_by_country_and_aggregate(df,\\\n",
    "    lambda r: r[\"sugars_100g\"], \"median sugar per 100g\", np.median, 1000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "# this reindex operation reverts the serie\n",
    "mean_sugar_per_country.reindex(index=mean_sugar_per_country.index[::-1]).plot.barh()\n",
    "plt.ylabel(\"\")\n",
    "plt.savefig('images/median_sugar_per_country.png', transparent=True, bbox_inches='tight', dpi = 300)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(plt.style.available)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our top three is then:\n",
    "* Canada\n",
    "* The US\n",
    "* Switzerland\n",
    "\n",
    "Now that we have an overview of the countries offering a lot sugar, we want to specify by category. We find the categories with the most median sugar content."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "def q10(l):\n",
    "    return np.quantile(l,0.1)\n",
    "def q90(l):\n",
    "    return np.quantile(l,0.9)\n",
    "\n",
    "def err(bar):\n",
    "    return np.array([bar[''\"median\"] - bar[\"q10\"], bar[\"q90\"] - bar[\"median\"]])\n",
    "\n",
    "\n",
    "\n",
    "test_sugar = df.groupby(\"main_category_en\")[\"sugars_100g\"].aggregate([\"median\", \"count\", q10, q90])\n",
    "bar = test_sugar[test_sugar[\"count\"] > 1000].sort_values(by=\"median\", ascending=True)\n",
    "\n",
    "plt.title(\"Sugar rate of 80% of the food, by category\")\n",
    "bar[\"median\"].plot.barh(xerr=err(bar))\n",
    "plt.ylabel(\"\")\n",
    "plt.savefig('images/median_sugar_per_category.png', transparent=True, bbox_inches='tight', dpi = 300)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The three food categories with the most sugar are:\n",
    "1. Sugary Snacks\n",
    "2. Desserts\n",
    "3. Fruit juices\n",
    "\n",
    "Let's focus on the sugary snacks and compare countries with the top three highest median sugar content. We'll separate Switzerland and North America (USA + Canada)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "north_america_mask = df.apply(lambda r: \"United States\" in r[\"countries_en\"] or \"Canada\" in r[\"countries_en\"], axis=1)\n",
    "north_america = df[north_america_mask]\n",
    "\n",
    "switzerland_mask = df.apply(lambda r: \"Switzerland\" in r[\"countries_en\"], axis=1)\n",
    "switzerland = df[switzerland_mask]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's compare the two regions by median sugar content on a few product categories."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "compared_categories =  {\"Sugary snacks\", \"Fruit juices\", \"Beverages\", \"Groceries\"}\n",
    "\n",
    "sugar_switzerland = switzerland.groupby(\"main_category_en\")[\"sugars_100g\"].aggregate([\"median\", \"count\", q10, q90])\n",
    "sugar_switzerland = sugar_switzerland.sort_values(by=\"count\", ascending=False)\n",
    "\n",
    "selected_categories = sugar_switzerland.index.map(lambda r: r in compared_categories)\n",
    "median_sugar_switzerland = sugar_switzerland[selected_categories].sort_index(ascending=False)\n",
    "\n",
    "sugar_north_america = north_america.groupby(\"main_category_en\")[\"sugars_100g\"].aggregate([\"median\", \"count\", q10, q90])\n",
    "sugar_north_america = sugar_north_america.sort_values(by=\"count\", ascending=False)\n",
    "\n",
    "selected_categories = sugar_north_america.index.map(lambda r: r in compared_categories)\n",
    "median_sugar_north_america = sugar_north_america[selected_categories].sort_index(ascending=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots()\n",
    "\n",
    "ind = np.arange(4)    # the x locations for the groups\n",
    "width = 0.35         # the width of the bars\n",
    "\n",
    "\n",
    "p1 = ax.bar(ind, median_sugar_switzerland[\"median\"], width, bottom=0)#, yerr=err(median_sugar_switzerland))\n",
    "p2 = ax.bar(ind + width, median_sugar_north_america[\"median\"], width, bottom=0)\n",
    "\n",
    "ax.set_title('Sugar per category: Switzerland - North America')\n",
    "ax.set_xticks(ind + width / 2)\n",
    "ax.set_xticklabels(median_sugar_switzerland.index)\n",
    "\n",
    "ax.legend((p1[0], p2[0]), ('Switzerland', 'North America'))\n",
    "ax.autoscale_view()\n",
    "\n",
    "plt.savefig('images/america_swiss_sugar_comparaison.png', transparent=True, bbox_inches='tight', dpi = 300)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Surprisingly, Switzerland's sugary snacks have a much higher median sugar content than those sold in North America!\n",
    "\n",
    "To investigate this trend, let's consider the brands selling the most sugary foods."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_sugar = df.groupby(\"brands_tags\")[\"sugars_100g\"].aggregate([\"median\", \"count\"])\n",
    "test_sugar[test_sugar[\"count\"] > 1000].sort_values(by=\"median\", ascending=True)[\"median\"].plot.barh()\n",
    "plt.ylabel(\"\")\n",
    "plt.savefig('images/top_sugar_brands.png', transparent=True, bbox_inches='tight', dpi = 300)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nestlé comes in first with a median sugar content four times superior to Spartan in second place!\n",
    "\n",
    "Let's take a look at the words most present in the Nestlé sugary snacks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot(word_list, name=\"\"):\n",
    "    wlist = word_list.copy()\n",
    "    \n",
    "    # shuffle the list to avoid patterns\n",
    "    np.random.shuffle(wlist)    \n",
    "    words = \"\"\n",
    "    for word in wlist:\n",
    "        words += word\n",
    "        words += \" \" \n",
    "    wordcloud = WordCloud(width = 1920, height = 1080, min_font_size=30).generate(words)\n",
    "    print(wordcloud)\n",
    "    # Display the generated image:\n",
    "    plt.figure(figsize=(10, 10))\n",
    "    plt.title(name)\n",
    "    plt.imshow(wordcloud)\n",
    "    plt.axis(\"off\")\n",
    "    plt.margins(x=0, y=0)\n",
    "    plt.savefig(\"images/\" + str(name) + \" keywords.png\", transparent=True, bbox_inches='tight', dpi = 300)\n",
    "    plt.show()\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def stringify(df):\n",
    "    l = []\n",
    "    for name in df[\"product_name\"]:\n",
    "        for n in str(name).split():  \n",
    "            if (len(n) > 2 and n != \"aux\"):\n",
    "                l += [n]\n",
    "    return l"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sugary_snacks_df = df[df[\"main_category_en\"] == \"Sugary snacks\"]\n",
    "nestle_snacks = sugary_snacks_df[sugary_snacks_df[\"brands_tags\"] == \"nestle\"]\n",
    "plot(stringify(nestle_snacks), \"Nestle snacks\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot(stringify(switzerland[switzerland[\"main_category_en\"] == \"Sugary snacks\"]), \"Swiss snacks\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A clear pattern emerges and we see that chocolate stands out and appears a lot in sugary snack product names. This is probably due to some important Swiss chocolate companies, such as Nestlé. This might also explain the importance of Switzerland in the data.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot(stringify(north_america[north_america[\"main_category_en\"] == \"Sugary snacks\"]), \"US snacks\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The US sugary snacks, however, are a little more various and less linked to chocolate.\n",
    "\n",
    "Surprisingly, popcorn is the most common word in the dataset of US sugary snacks!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Additives"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Comparing the use of additives in the US and France\n",
    "Finding how restrictions affect the presence of additives in our food.\n",
    "\n",
    "Where do we have the greatest variety of additives?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def additives(row):\n",
    "    return row[\"additives_tags\"]\n",
    "\n",
    "def n_distinct_additives(additives_list):\n",
    "    total_list = set()\n",
    "    for  additives in additives_list:\n",
    "        for a in additives:\n",
    "            total_list.add(a)\n",
    "    return len(total_list)\n",
    "\n",
    "number_additives_per_country = group_by_country_and_aggregate(df, additives, \"number of different additives\", n_distinct_additives, 1000)\n",
    "number_additives_per_country.sort_values(by = 'number of different additives', ascending=True).plot.barh()\n",
    "plt.ylabel(\"\")\n",
    "plt.savefig('images/additives_per_country.png', transparent=True, bbox_inches='tight', dpi = 300)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We find that the countries with the greatest variety of additives on open food facts are:\n",
    "1. France\n",
    "2. USA\n",
    "3. Switzerland\n",
    "\n",
    "This makes sense considering that these are the countries with the greatest quantity of entries on the website. It would be interesting however to compare this variety of additives to the proportion of those considered dangerous. Do the same countries show up at the top if we consider the proportion of dangerous additives?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def rate_of_additives_in(additives_list, additives_set):   \n",
    "    l = 0     \n",
    "    for additives in additives_list:\n",
    "        for a in additives:\n",
    "            if a in [\"n\", \"a\"]:\n",
    "                break\n",
    "            if a[3:].lower() in additives_set:           \n",
    "                l += 1\n",
    "    return l / len(additives_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def rate_of_dangerous_additives(additives_list):\n",
    "    # source: https://www.foodmatters.com/article/top-10-food-additives-to-avoid\n",
    "    dangerous_additives = [\"e951\", \"e621\", \"e133\", \"e124\", \"e110\", \"e102\", \"e221\", \"e320\", \"e220\"]\n",
    "    return rate_of_additives_in(additives_list, dangerous_additives)\n",
    " \n",
    "name = \"at least one unpopular additive\"\n",
    "danger_additives_per_country = group_by_country_and_aggregate(df, additives, name, rate_of_dangerous_additives, 1000)\n",
    "danger_additives_per_country.sort_values(name, ascending=True).plot.barh()\n",
    "plt.ylabel(\"\")\n",
    "plt.savefig('images/dangerous_additives_per_country.png', transparent=True, bbox_inches='tight', dpi = 300)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Interestingly, only the US is also in the top of countries with a proportion of dangerous additives. France, while it was the country with the largest variety of additives, does not have a very high proportion of dangerous additives.\n",
    "\n",
    "How do France and the US differ in terms of additives legislation? We first try to get an idea by looking at additives present in US foods but not available in France."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def distinct_additives(additives_list):\n",
    "    total_list = set()\n",
    "    for  additives in additives_list:\n",
    "        for a in additives:\n",
    "            a = str(a)\n",
    "            if a != \"n\" and a != \"a\":          \n",
    "                total_list.add(a[3:])\n",
    "    return total_list\n",
    "\n",
    "additives_df = group_by_country_and_aggregate(df, additives, \"additives\", distinct_additives, 100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "set_us_additives = additives_df[additives_df.index == \"United States\"][\"additives\"].iloc[0]\n",
    "set_fr_additives = additives_df[additives_df.index == \"France\"][\"additives\"].iloc[0]\n",
    "\n",
    "set_us_additives - set_fr_additives"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "22 additives are present in at least one US product but never appear in France. The following are actually foridden in France:\n",
    "* E105\n",
    "* E107\n",
    "* E908\n",
    "* E924a\n",
    "* E927a, it is also forbidden in the rest of the EU.\n",
    "* E928, it is also forbidden in the rest of the EU, Australia and New Zealand.\n",
    "* E409, it is also forbidden in the rest of the EU.\n",
    "* E488\n",
    "* E539, it is also forbidden in the rest of the EU.\n",
    "\n",
    "E924a is particularly interesting as it is a floor bleaching agent forbidden in the whole EU but it has been used extensively by big brands such as [Subway and Wendy's](https://en.wikipedia.org/wiki/Azodicarbonamide).\n",
    "\n",
    "While the following additives are allowed in France, there is no entry that contains these products in the French dataset:\n",
    "* E161G [since 1997](https://www.legifrance.gouv.fr/affichTexte.do;jsessionid=DB38A4996122419ED4EC649263CEFEAA.tpdjo17v_3?cidTexte=LEGITEXT000022192579&dateTexte=20130215)\n",
    "* E343i [since 2007](https://www.legifrance.gouv.fr/affichTexte.do;jsessionid=DB38A4996122419ED4EC649263CEFEAA.tpdjo17v_3?cidTexte=LEGITEXT000022192579&dateTexte=20130215)\n",
    "* E352 [since 2001](https://www.legifrance.gouv.fr/affichTexte.do;jsessionid=DB38A4996122419ED4EC649263CEFEAA.tpdjo17v_3?cidTexte=LEGITEXT000022192579&dateTexte=20130215)\n",
    "* E403 [since 2009](https://www.legifrance.gouv.fr/affichTexte.do;jsessionid=DB38A4996122419ED4EC649263CEFEAA.tpdjo17v_3?cidTexte=LEGITEXT000022192579&dateTexte=20130215), but with harsh restrictions on the kind of products allowed to contain this additive.\n",
    "* E432 [since 2004](https://www.legifrance.gouv.fr/affichTexte.do;jsessionid=DB38A4996122419ED4EC649263CEFEAA.tpdjo17v_3?cidTexte=LEGITEXT000022192579&dateTexte=20130215), but it is only allowed in some pastry products.\n",
    "* E450vi [since 2003](https://www.legifrance.gouv.fr/affichTexte.do;jsessionid=DB38A4996122419ED4EC649263CEFEAA.tpdjo17v_3?cidTexte=LEGITEXT000022192579&dateTexte=20130215)\n",
    "* E523 [since 2001](https://www.legifrance.gouv.fr/affichTexte.do;jsessionid=DB38A4996122419ED4EC649263CEFEAA.tpdjo17v_3?cidTexte=LEGITEXT000022192579&dateTexte=20130215)\n",
    "* E557 unknown when\n",
    "* E622 [since 2001](https://www.legifrance.gouv.fr/affichTexte.do;jsessionid=DB38A4996122419ED4EC649263CEFEAA.tpdjo17v_3?cidTexte=LEGITEXT000022192579&dateTexte=20130215)\n",
    "* E624 [since 2001](https://www.legifrance.gouv.fr/affichTexte.do;jsessionid=DB38A4996122419ED4EC649263CEFEAA.tpdjo17v_3?cidTexte=LEGITEXT000022192579&dateTexte=20130215)\n",
    "* E634 [since 2001](https://www.legifrance.gouv.fr/affichTexte.do;jsessionid=DB38A4996122419ED4EC649263CEFEAA.tpdjo17v_3?cidTexte=LEGITEXT000022192579&dateTexte=20130215)\n",
    "* E905b [since 1999](https://www.legifrance.gouv.fr/affichTexte.do;jsessionid=DB38A4996122419ED4EC649263CEFEAA.tpdjo17v_3?cidTexte=LEGITEXT000022192579&dateTexte=20130215)\n",
    "\n",
    "The absence of some of these can be easily explained. E161g for exmaple, is only allowed for use in sausages from Strasbourg. This makes it quite rare and explains why it is not in the Open Food Facts database"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "name='proportion of products containing additives which are illegal in France'\n",
    "\n",
    "def rate_of_forbidden_additives(additives_list): \n",
    "    return rate_of_additives_in(additives_list, [\"e105\", \"e107\", \"e908\", \"e924a\", \"e927a\", \"e928\", \"e409\", \"e488\", \"e539\"])\n",
    "\n",
    "rate_of_forbidden_additives = group_by_country_and_aggregate(df, additives, name, rate_of_forbidden_additives, 1000)\n",
    "rate_of_forbidden_additives[name] *= 10000\n",
    "rate_of_forbidden_additives.sort_values(name, ascending=False).plot.bar()\n",
    "plt.savefig('images/forbidden_additives_per_country.png', transparent=True, bbox_inches='tight', dpi = 300)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conclusion\n",
    "It's interesting to see how differently the US and France are concerning additives. Despite France having a larger variety of additives in the dataset, very few dangerous ones appear in French food.\n",
    "\n",
    "Looking at the difference between additives available in France and the US, we found that several of these absences could be explained by the fact that France has stricter regulations on additives. HHowever, these additives allowed in the US ut not France are only present in 0.4% of products"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}