07-big-data-dashboards.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "9988a010-b1d5-44e4-9eea-bf6e4466b4a7",
   "metadata": {},
   "source": [
    "<img src=\"images/panel_logo.png\" width=\"10%\" align=\"right\">\n",
    "\n",
    "\n",
    "# Big data dashboarding with Panel\n",
    "\n",
    "In this notebook, we are going to put our visualizations together into a dashboard. We will also introduce a new tool from the HoloViz suite, Panel - a high-level app and dashboarding solution for Python.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f23406d-8642-4da0-ac18-bf6211a15f2b",
   "metadata": {},
   "source": [
    "## Reconnect to Dask Cluster"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6f3eae75-0829-4bca-bebc-6d49c8ba0050",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import dask_gateway\n",
    "import dask.dataframe as dd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e9265dde-8387-4c27-8702-1cb9015045c3",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "gateway = dask_gateway.Gateway()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "168b1ded-076a-4894-a3ad-64dceefbe3c9",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "if len(running_clusters := gateway.list_clusters())>0:\n",
    "    cluster = gateway.connect(running_clusters[0].name)\n",
    "else:\n",
    "    cluster = gateway.new_cluster(conda_environment=\"scipy/scipy-data-of-unusual-size\", profile=\"Medium Worker\")\n",
    "    cluster.adapt(1,10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f895739f-ff0f-41e8-a933-6057b138b212",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "cluster"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "993ca5bf-59d8-472d-aac3-85ea367e175c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "client = cluster.get_client()\n",
    "client"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2a0751ae-8ebb-46f1-920b-8786c5a14d78",
   "metadata": {},
   "source": [
    "## Load the airline on-time performance dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1f27e8e6-0066-46c3-819c-209d22cb065f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "columns = [\n",
    "    'YEAR', 'MONTH', 'DAY_OF_MONTH', 'DAY_OF_WEEK', 'FL_DATE', 'OP_CARRIER', \n",
    "    'TAIL_NUM', 'OP_CARRIER_FL_NUM', 'ORIGIN', 'DEST', 'CRS_DEP_TIME', \n",
    "    'DEP_TIME', 'DEP_DELAY', 'ARR_TIME', 'ARR_DELAY', 'CANCELLED', \n",
    "    'CANCELLATION_CODE', 'DIVERTED', 'AIR_TIME', 'FLIGHTS', 'DISTANCE',\n",
    "    'CARRIER_DELAY', 'WEATHER_DELAY', 'NAS_DELAY', 'SECURITY_DELAY', \n",
    "    'LATE_AIRCRAFT_DELAY', 'DIV_ARR_DELAY'\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1c7ef3af-90e2-4205-ae7e-95a5cb76752a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "flights = dd.read_parquet(\n",
    "    f\"gcs://quansight-datasets/airline-ontime-performance/sorted/full_dataset.parquet\", \n",
    "    columns=columns\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0aab7630-0a18-4bed-b17d-8adff3401720",
   "metadata": {},
   "source": [
    "## Load the latitude-longitude data for the airports\n",
    "\n",
    "We have a file `prep/airports.csv` that has the lat/lon information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bd8633a4-8781-4d8a-8fcc-b0f2e6f10de3",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import panel as pn\n",
    "import pandas as pd\n",
    "\n",
    "pn.extension()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4324e4f0-022d-47da-9829-b18f5a4ac149",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import hvplot.dask\n",
    "import hvplot.pandas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ddd27d12-622f-4159-98d0-5b36f6979fdb",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import warnings\n",
    "\n",
    "warnings.filterwarnings('ignore') # Ignore some HoloviewsDeprecationWarning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e1a153bb-f15e-4f44-a70a-92889976973c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "airports = pd.read_csv(\"prep/airports.csv\", \n",
    "                        usecols=[\"AIRPORT\", \"DISPLAY_AIRPORT_NAME\", \"LATITUDE\", \"LONGITUDE\"], \n",
    "                        dtype={\n",
    "                            \"AIRPORT\": \"string\", \n",
    "                            \"DISPLAY_AIRPORT_NAME\": \"string\", \n",
    "                            \"LATITUDE\":\"float\", \n",
    "                            \"LONGITUDE\":\"float\",\n",
    "                        }\n",
    "                      ).set_index('AIRPORT')\n",
    "\n",
    "# drop duplicates keeping the last entry\n",
    "airports = airports[~airports.index.duplicated(keep='last')]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d7549961-55ed-4eec-8eb7-a25d823ddaba",
   "metadata": {},
   "source": [
    "Let's quickly plot the map of airports:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1d0045e4-1be4-43b2-8ea9-7935affc0821",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "airports.hvplot.points('LONGITUDE', 'LATITUDE',  geo=True, \n",
    "                       color='red', alpha=0.2, hover_cols=['AIRPORT'],\n",
    "                       tiles='CartoLight')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "51047775-ad15-472d-8926-dae2c7e311a7",
   "metadata": {},
   "source": [
    "## Build a Panel dashboard\n",
    "\n",
    "Panel has 3 methods for building out interactive dashboards:\n",
    "\n",
    "- hvPlot `.interactive` turns any of your DataFrame processing pipelines into a dashboard (great if you want to explore a dataset!);\n",
    "- Panel `.bind` binds your widgets with your interactive plot (great if you want to build an arbitrary app!);\n",
    "- Param encapsulates your dashboard as self-contained classes (great if you want to build a complex codebase supporting both GUI and non-GUI usage).\n",
    "\n",
    "For more details, read this excellent blog post: [3 ways to build a panel visualization dashboard](https://towardsdatascience.com/3-ways-to-build-a-panel-visualization-dashboard-6e14148f529d).\n",
    "\n",
    "We will use the `hvplot.interactive` method here, but first, let's build a small `pandas` data pipeline:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "019c6f98-db75-4528-b3b0-95678a6e87c3",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "pipeline = (\n",
    "    flights[\n",
    "        (flights['FL_DATE'] > \"2020\") &\n",
    "        (flights['FL_DATE'] <= \"2021\")\n",
    "    ]\n",
    "    .groupby('DAY_OF_WEEK')[\"ARR_DELAY\"].agg(how=\"mean\")\n",
    "    .rename(columns={\"how\": f\"ARR_DELAY - mean\"})\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "56a53604-b3f1-4d47-9bc3-0efafc580f84",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d7ddd5f9-aaff-464b-9963-3013d134a8f8",
   "metadata": {},
   "source": [
    "## Move from static pipeline to a dynamic version\n",
    "\n",
    "In the data pipeline above, let's use variables to represent the quantities we want to select in our dashboard: \n",
    "\n",
    "- `daterange` - start and end dates to filter by\n",
    "- `groupby` - variable we wish to groupby\n",
    "- `field` - data field we wish to plot\n",
    "- `method` - statistic we wish to calculate (min, max, mean etc)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fffc44e7-a9f1-48ca-9fb5-d0491f5fcfbb",
   "metadata": {},
   "source": [
    "### Pick some Panel widgets for each of these variables\n",
    "\n",
    "Further reading: [Panel documentation](https://panel.holoviz.org/reference/index.html#widgets)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "890ea4af-e020-4c34-aa21-ac943c0dff6c",
   "metadata": {},
   "source": [
    "**We're picking the `DateRangeSlider` widget for the `daterange` variable:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "446c7179-639c-4386-bee4-6af7b29740e6",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import datetime as dt\n",
    "\n",
    "daterange = pn.widgets.DateRangeSlider(\n",
    "    name='Date Range Slider',\n",
    "    start=dt.datetime(2003, 1, 1), end=dt.datetime(2022, 12, 31),\n",
    "    value=(dt.datetime(2022, 1, 1), dt.datetime(2022, 12, 31)),\n",
    "    step=24*3600*2*1000,\n",
    "    bar_color = \"green\",\n",
    "    width=800\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "24d722ed-bd37-41b7-ae06-a94c311645ae",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "daterange"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5e6a0315-cbbf-4cbc-af0b-700969b01dd6",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "daterange.value"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0feb6f05-bdab-4f70-bba6-6b66fb628f34",
   "metadata": {},
   "source": [
    "**We're picking the `RadioButtonGroup` widget for the `groupby` variable:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5eafb8c3-c7d5-489e-8a9a-5a316d790344",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "groupby = pn.widgets.RadioButtonGroup(\n",
    "    name='Period', \n",
    "    options=['YEAR', 'MONTH', 'DAY_OF_MONTH', 'OP_CARRIER'],\n",
    "    value='MONTH',\n",
    ")\n",
    "groupby"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7ceee046-b5b3-4f8d-b380-bad15af99310",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "groupby.value"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42a08771-cff2-4622-9a5e-193bc1222948",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-04-18T12:15:35.192126Z",
     "iopub.status.busy": "2023-04-18T12:15:35.191670Z",
     "iopub.status.idle": "2023-04-18T12:15:35.197283Z",
     "shell.execute_reply": "2023-04-18T12:15:35.196007Z",
     "shell.execute_reply.started": "2023-04-18T12:15:35.192088Z"
    },
    "tags": []
   },
   "source": [
    "### 💻 Your turn: Choose an appropriate widget for `field` and `method`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "76f94173-aa52-4528-9b05-14fdba51e145",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Your code here. When ready, click on the three dots below for the solutions."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a544804d-e8fb-4756-8ff6-e3ee24aefd4b",
   "metadata": {
    "jupyter": {
     "source_hidden": true
    },
    "tags": []
   },
   "source": [
    "**We're picking the `RadioButtonGroup` widget for the `groupby` variable:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c8a2ad9a-44fd-433e-88ff-69a3107410ef",
   "metadata": {
    "jupyter": {
     "source_hidden": true
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "method = pn.widgets.Select(\n",
    "    name='Method', \n",
    "    options=['min', 'max', 'mean', 'count'],\n",
    "    value='mean',\n",
    ")\n",
    "method"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3446fb5f-2682-47bb-b021-a45ccb11d6a8",
   "metadata": {
    "jupyter": {
     "source_hidden": true
    },
    "tags": []
   },
   "source": [
    "**We're picking the `RadioButtonGroup` widget for the `groupby` variable:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "50d60619-0059-468b-ae42-59705fce5050",
   "metadata": {
    "jupyter": {
     "source_hidden": true
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "field = pn.widgets.RadioBoxGroup(\n",
    "    name='Field', \n",
    "    options=['DEP_DELAY', 'ARR_DELAY'],\n",
    ")\n",
    "field"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d13b24f-d116-487f-ad2c-1d3dfb1580c8",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-04-18T12:43:31.042106Z",
     "iopub.status.busy": "2023-04-18T12:43:31.041671Z",
     "iopub.status.idle": "2023-04-18T12:43:31.047693Z",
     "shell.execute_reply": "2023-04-18T12:43:31.046401Z",
     "shell.execute_reply.started": "2023-04-18T12:43:31.042072Z"
    },
    "tags": []
   },
   "source": [
    "## Put everything together with `hvplot.interactive`\n",
    "\n",
    "First, we need to make the DataFrame into an interactive DataFrame:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f623852a-1c2b-41c2-847d-1e848831d5d7",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "iflights = flights.interactive()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "95412123-52f4-4f23-bee1-dcd692eb2f83",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-04-18T12:44:36.546015Z",
     "iopub.status.busy": "2023-04-18T12:44:36.545142Z",
     "iopub.status.idle": "2023-04-18T12:44:36.552594Z",
     "shell.execute_reply": "2023-04-18T12:44:36.551144Z",
     "shell.execute_reply.started": "2023-04-18T12:44:36.545977Z"
    },
    "tags": []
   },
   "source": [
    "Then, we can make an interactive pipeline using our widgets as variables\n",
    "\n",
    "For reference, here's our pipeline code from before:\n",
    "\n",
    "```python\n",
    "pipeline = (\n",
    "    flights[\n",
    "        (flights['FL_DATE'] > \"2020\") &\n",
    "        (flights['FL_DATE'] <= \"2021\")\n",
    "    ]\n",
    "    .groupby('DAY_OF_WEEK')[\"ARR_DELAY\"].agg(how=\"mean\")\n",
    ")\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1e33c549-067d-4068-948d-5d1f7d602886",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Combine pipeline and widgets\n",
    "\n",
    "ipipeline = (\n",
    "    iflights[\n",
    "        (iflights['FL_DATE'] > daterange.value[0]) &\n",
    "        (iflights['FL_DATE'] <= daterange.value[1])\n",
    "    ]\n",
    "    .groupby(groupby)[field]\n",
    "    .agg(how=method)\n",
    "    .rename(columns={\"how\": f\"{field} - {method}\"})\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e5e75de4-a30f-4ec3-9473-9f65e3a46253",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "ipipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d53ea8fa-9d8a-4815-b9b9-19491fb0c5e1",
   "metadata": {},
   "source": [
    "We can now use the interactive pipeline object in several ways."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6619c7d5-b4e9-411c-8857-79335fb14c1d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "data_plot = ipipeline.hvplot()\n",
    "data_plot"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3fd208db-bc2a-4902-99cd-00eac72ddba0",
   "metadata": {},
   "source": [
    "### 💻 Your turn: Create an interactive version of the airport map plot, but color the airports based on data values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b0cbc740-2295-48f8-95a1-c647cc63dbe7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code here. When ready, click on the three dots below for the solutions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7a4b51b3-7cfc-419e-b3f1-67435bceaf71",
   "metadata": {
    "jupyter": {
     "source_hidden": true
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "flight_delays = (\n",
    "    iflights[\n",
    "        (iflights['FL_DATE'] > daterange.value[0]) &\n",
    "        (iflights['FL_DATE'] <= daterange.value[1])\n",
    "    ]\n",
    "    .groupby('ORIGIN')[field]\n",
    "    .agg(how=method)\n",
    "    .join(airports)\n",
    "    .rename(columns={\"how\": f\"{field} - {method}\"})\n",
    ")\n",
    "\n",
    "flight_delays"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8691f453-ef9d-4839-a76b-4d3f4ff9bb8b",
   "metadata": {
    "jupyter": {
     "source_hidden": true
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "map_plot = flight_delays.hvplot.points('LONGITUDE', 'LATITUDE', geo=True, color=f\"{field} - {method}\", \n",
    "                 hover_cols=['ORIGIN', f\"{field} - {method}\"],\n",
    "                 xlim=(-180, -30), ylim=(-20, 75), \n",
    "                 cmap='viridis', tiles='CartoLight')\n",
    "\n",
    "# Uncomment the next line to see the map plot (note: this will take a bit of time)\n",
    "# map_plot"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5500502d-8e06-46e3-96c9-11f75eaef5fa",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Panel's Row/Column Grid system\n",
    "\n",
    "Panel also has a customizable template system that allows you to build apps that have a header, sidebar, main area and popup windows. For details, see: https://panel.holoviz.org/user_guide/Templates.html\n",
    "\n",
    "Let's re-arrange the components of our dashboard with this system:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cdc123ca-645a-4a61-a35d-924214c8d82f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "pn.Column(\n",
    "    daterange,\n",
    "    pn.Row(field, pn.Column(groupby, method)),\n",
    "    data_plot.panel(),\n",
    "    map_plot.panel(),\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a17b00f-5471-4cac-a718-91d53290ffd9",
   "metadata": {},
   "source": [
    "Close your Dask cluster:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e773c686-c9a5-4ff8-a432-8b4ca80bf5a2",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "cluster.close(shutdown=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4ebb075b-d1de-4fb4-bfdd-e20d1d63a943",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Next →\n",
    "\n",
    "[Big data application pipeline](./08-big-data-application-pipeline.ipynb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3ef96708-abd8-4d46-afd6-3e7412789fcb",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "scipy-scipy-data-of-unusual-size",
   "language": "python",
   "name": "conda-env-scipy-scipy-data-of-unusual-size-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}