manage_assignments.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Intro to managing assignments"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All of the work managing assignments is completed in the nbgrader `<course_directory>`, while logged in as the user \"instructor\".  For the 2015-fall-big_data class, the `<course_directory>` is `/home/instructor/nbgrader/courses/2015-fall-big_data`.  This IPython notebook, `manage_assignments.ipynb`, should be run from the `<course_directory>`.  Do not try to run it in any other directory.\n",
    "\n",
    "This notebook contains:\n",
    "\n",
    "- instructions on initializing the course's gradebook database.\n",
    "- an overview of the process of making, releasing, collecting, and grading assignments.\n",
    "- tips for troubleshooting problems with assignments.\n",
    "\n",
    "## Table of Contents\n",
    "\n",
    "- [Initialize gradebook database](#Initialize-gradebook-database)\n",
    "\n",
    "    - [Add students to the gradebook](#Add-students-to-the-gradebook)\n",
    "    - [Add an assignment to the gradebook](#Add-an-assignment-to-the-gradebook)\n",
    "    \n",
    "- [Assignment workflow after adding to gradebook](#Assignment-workflow-after-adding-to-gradebook)\n",
    "\n",
    "    - [Implementing assignments](#Implementing-assignments)\n",
    "    - [1. render student version - `nbgrader assign \"<assignment>\"`](#1.-render-student-version---nbgrader-assign-\"<assignment>\")\n",
    "    - [2. release student version - `nbgrader release \"<assignment>\"`](#2.-release-student-version---nbgrader-release-\"<assignment>\")\n",
    "    - [3. STUDENT - Find assignments - `nbgrader list`](#3.-STUDENT---Find-assignments---nbgrader-list)\n",
    "    - [4. STUDENT - Fetch assignment - `nbgrader fetch \"<assignment>\"`](#4.-STUDENT---Fetch-assignment---nbgrader-fetch-\"<assignment>\")\n",
    "    - [5. STUDENT - Submit assignment - `nbgrader submit \"<assignment>\"`](#5.-STUDENT---Submit-assignment---nbgrader-submit-\"<assignment>\")\n",
    "    - [6a. STUDENT - List submitted assignments - `nbgrader list --cached`](#6a.-STUDENT---List-submitted-assignments---nbgrader-list---cached)\n",
    "    - [6b. List assignments with submissions - `nbgrader list --inbound`](#6b.-List-assignments-with-submissions---nbgrader-list---inbound)\n",
    "    - [7. Collect assignments - `nbgrader collect \"<assignment>\"`](#7.-Collect-assignments---nbgrader-collect-\"<assignment>\")\n",
    "    - [8. Auto-grade assignments - `nbgrader autograde \"<assignment>\"`](#8.-Auto-grade-assignments---nbgrader-autograde-\"<assignment>\")\n",
    "    - [9. Grade using the Formgrade application - `nbgrader formgrade`](#9.-Grade-using-the-Formgrade-Application---nbgrader-formgrade)\n",
    "    \n",
    "        - [Setting up, starting, and stopping the Formgrade application](#Setting-up,-starting,-and-stopping-the-Formgrade-application)\n",
    "    \n",
    "    - [10. Generating feedback for students - `nbgrader feedback \"<assignment>\"`](#10.-Generating-feedback-for-students---nbgrader-feedback-\"<assignment>\")\n",
    "    - [11. Getting feedback back to student](#11.-Getting-feedback-back-to-student)\n",
    "    \n",
    "- [Troubleshooting](#Troubleshooting)\n",
    "\n",
    "    - [`nbgrader formgrade` issues](#nbgrader-formgrade-issues)\n",
    "    \n",
    "        - [CONFIGPROXY_AUTH_TOKEN mismatch between jupyterhub and formgrade](#CONFIGPROXY_AUTH_TOKEN-mismatch-between-jupyterhub-and-formgrade)\n",
    "        - [\"attempt to write a readonly database\"](#\"attempt-to-write-a-readonly-database\")\n",
    "        - [\"unable to open database file\" api_tokens](#\"unable-to-open-database-file\"-api_tokens)\n",
    "        - [URLs on formgrade pages refer to proxy IP address, not public IP or domain name](#URLs-on-formgrade-pages-refer-to-proxy-IP-address,-not-public-IP-or-domain-name)\n",
    "        - [Notebook file not found when you try to grade a given submission](#Notebook-file-not-found-when-you-try-to-grade-a-given-submission)\n",
    "        \n",
    "    - [notebook issues](#notebook-issues)\n",
    "    \n",
    "        - [nbgrader solution can not be in a normal cell](#nbgrader-solution-can-not-be-in-a-normal-cell)\n",
    "        - [grade cell point value must be greater than 0](#grade-cell-point-value-must-be-greater-than-0)\n",
    "        - [make sure you run code cells that just contain a function definition before calling the function](#make-sure-you-run-code-cells-that-just-contain-a-function-definition-before-calling-the-function)\n",
    "        \n",
    "    - [gradebook database issues](#gradebook-database-issues)\n",
    "    \n",
    "        - [corruption of tables that hold auto-grading](#corruption-of-tables-that-hold-auto-grading)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Initialize gradebook database\n",
    "\n",
    "- Back to the [Table of Contents](#Table-of-Contents)\n",
    "\n",
    "A given course's students, assignments, and grades are stored in that course's gradebook database.  This database file is stored at `<course_directory>/gradebook.db`.  Before you do any work with assignments, you need to initialize the gradebook database by adding students and assignments to it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Add students to the gradebook\n",
    "\n",
    "- Back to the [Table of Contents](#Table-of-Contents)\n",
    "\n",
    "Before you do anything else, you'll need to add all your students to the gradebook.\n",
    "\n",
    "Assumptions:\n",
    "\n",
    "- as long as the \"source\" directory is in the same directory as this page, everything should be fine.\n",
    "- you'll need to make one line per student.  If you forget a student and then try to grade, there will be errors, so enter all your students right from the start."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# create a connection to the db using the nbgrader API\n",
    "from nbgrader.api import Gradebook\n",
    "\n",
    "# connect to the database.\n",
    "gb = Gradebook(\"sqlite:///gradebook.db\")\n",
    "\n",
    "# add some students to the database\n",
    "# template:\n",
    "# gb.add_student( \"<unix_username>\", first_name = \"<first_name>\", last_name = \"<last_name>\", email = \"<email>\" )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Add an assignment to the gradebook\n",
    "\n",
    "- Back to the [Table of Contents](#Table-of-Contents)\n",
    "\n",
    "Use the code below to add an assignment to the gradebook.  You should just need to set the assignment_name and assignment_due_date fields.  More notes on other options TK.\n",
    "\n",
    "Assumptions:\n",
    "\n",
    "- as long as the \"source\" directory is in the same directory as this page, everything should be fine.\n",
    "- should not need to list out the ipython notebooks in the assignment - you are just telling it the name of the assignment (which is also the name of the folder in \"source\" in which the assignment's notebooks live)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "# remove an existing database\n",
    "#if os.path.exists(\"gradebook.db\"):\n",
    "#    os.remove(\"gradebook.db\")\n",
    "\n",
    "# create a connection to the db using the nbgrader API\n",
    "from nbgrader.api import Gradebook\n",
    "gb = Gradebook(\"sqlite:///gradebook.db\")\n",
    "\n",
    "#==================================\n",
    "# set up assignment call variables\n",
    "#==================================\n",
    "\n",
    "# assignment_name - the name of the folder inside the \"source\" folder that contains\n",
    "#    the ipython notebooks for a given assignment.\n",
    "# example: assignment_name = \"08. Machine Learning\"\n",
    "\n",
    "# assignment_due_date - the date the assignment is due, in format \"YYYY-MM-DD HH:MM:SS:MMMMMM TZ\"\n",
    "# example: assignment_due_date = \"2015-02-01 17:00:00.000000 PST\"\n",
    "\n",
    "# add the assignment to the database and specify a due date\n",
    "gb.add_assignment( assignment_name, duedate = assignment_due_date )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Assignment workflow after adding to gradebook\n",
    "\n",
    "- Back to the [Table of Contents](#Table-of-Contents)\n",
    "\n",
    "Once you've run the code above to add an assignment to the gradebook and put your assignments in the `source` folder, you'll use the `nbgrader` command to:\n",
    "\n",
    "- make a student version of the assignment\n",
    "- release it to the students\n",
    "- check for submissions\n",
    "- gather submissions for grading\n",
    "- autograde all assignments (even those with no auto-grade questions, just so you can use the form grade app).\n",
    "- start the formgrade app to grade assignments that can not be auto-graded.\n",
    "\n",
    "Details on each step of this process follow, using an example assignment named either \"06. Networks\" or \"02. Database Basics\"."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Implementing assignments\n",
    "\n",
    "- Back to the [Table of Contents](#Table-of-Contents)\n",
    "\n",
    "Before starting the formal process of assigning and grading an assignment, you'll need to get the assignment code into the `source` directory inside the course directory.  For our class, the source folder is a github repository ( [https://github.com/CSSIP-AIR/Big-Data-Workbooks](https://github.com/CSSIP-AIR/Big-Data-Workbooks) ), cloned into the course folder with its root directory renamed to `source`.\n",
    "\n",
    "To initially clone this repository, you'd run the following command in the course folder:\n",
    "\n",
    "    git clone https://github.com/CSSIP-AIR/Big-Data-Workbooks.git source\n",
    "    \n",
    "As assignments are developed, updates should be committed to the github repository.  Once they are ready to be processed by nbgrader, go into the `source` folder in a command shell and run:\n",
    "\n",
    "    git pull\n",
    "    \n",
    "This will pull down the latest code from the repository.\n",
    "\n",
    "The source folder is structured so each _assignment_ is in a folder inside `source`, and then each _exercise_ for that _assignment_ is a Jupyter notebook within the _assignment_ folder.  You can include other files and folders in the _assignment_ folder, and they will be distributed to students as part of the assignment.\n",
    "\n",
    "To develop an assignment, you'll at the least need to download the github repository and then install nbgrader and the nbgrader toolbar in your development computer's jupyter setup.  First get the notebook as you want it, then turn on the nbgrader \"Create assignment\" toolbar and go through and set up the question and answer cells as needed.  Once you are done:\n",
    "\n",
    "- Clear out all cell output --> In the \"Cell\" menu, choose \"All Output\", then \"Clear\".\n",
    "- In \"File\" menu, \"Save and Checkpoint\" your assignment.\n",
    "- In \"File\" meny, \"Close and Halt\" the notebook.\n",
    "- in command shell, cd into the \"`source`\" directory.\n",
    "- use git to add and commit changes, then pull, then push your changes to github, so they can be pulled down by the instructor.\n",
    "\n",
    "More information on nbgrader-izing an assignment:\n",
    "\n",
    "- [http://nbgrader.readthedocs.org/en/stable/user_guide/02_developing_assignments.html](http://nbgrader.readthedocs.org/en/stable/user_guide/02_developing_assignments.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. render student version - `nbgrader assign \"<assignment>\"`\n",
    "\n",
    "- Back to the [Table of Contents](#Table-of-Contents)\n",
    "\n",
    "Use the `nbgrader assign \"<assignment>\"` command to render the student version of the notebook:\n",
    "\n",
    "    nbgrader assign \"06. Networks\"\n",
    "    \n",
    "- Example output:\n",
    "    \n",
    "        $ nbgrader assign \"06. Networks\"\n",
    "        Networks\" --IncludeHeaderFooter.header=source/Style\\ Guide.ipynb\n",
    "        [AssignApp | INFO] Converting notebook source/./06. Networks/networks_exercise.ipynb to notebook\n",
    "        [AssignApp | INFO] Writing 11791 bytes to release/./06. Networks/networks_exercise.ipynb\n",
    "        [AssignApp | INFO] Setting destination file permissions to 644  \n",
    "        \n",
    "After the first time you do this, to re-render, then overwrite the existing student version of the assignment:\n",
    "    \n",
    "    nbgrader assign \"06. Networks\" --force\n",
    "        \n",
    "- Example output:\n",
    "    \n",
    "        $ nbgrader assign \"06. Networks\" --force\n",
    "        [AssignApp | WARNING] Removing existing assignment: release/06. Networks\n",
    "        [AssignApp | INFO] Converting notebook source/./06. Networks/networks_exercise.ipynb to notebook\n",
    "        [AssignApp | INFO] Writing 11791 bytes to release/./06. Networks/networks_exercise.ipynb\n",
    "        [AssignApp | INFO] Setting destination file permissions to 644  \n",
    "        \n",
    "- Example output if you forget the `--force`:\n",
    "\n",
    "        $ nbgrader assign \"06. Networks\"\n",
    "        [AssignApp | INFO] Skipping existing assignment: release/06. Networks\n",
    "\n",
    "If you have a header (we don't at the moment):\n",
    "    \n",
    "    nbgrader assign \"06. Networks\" --IncludeHeaderFooter.header=source/Style\\ Guide.ipynb\n",
    "    \n",
    "Running `nbgrader assign \"<assignment>\"` on an assignment renders student versions of the assignment's notebooks, then  places the student versions in a directory named the same as the assignment in the `<course_folder>/release` folder."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. release student version - `nbgrader release \"<assignment>\"`\n",
    "\n",
    "- Back to the [Table of Contents](#Table-of-Contents)\n",
    "\n",
    "Use the `nbgrader release \"<assignment>\"` command to place the assignment in the `exchange` folder:\n",
    "\n",
    "    nbgrader release \"08. Machine Learning\"\n",
    "    \n",
    "- Example output:\n",
    "\n",
    "        $ nbgrader release \"06. Networks\"\n",
    "        [ReleaseApp | INFO] Source: /home/instructor/nbgrader/courses/2015-fall-big_data/release/06. Networks\n",
    "        [ReleaseApp | INFO] Destination: /srv/nbgrader/exchange/2015-fall-big_data/outbound/06. Networks\n",
    "        [ReleaseApp | INFO] Released as: 2015-fall-big_data 06. Networks\n",
    "        \n",
    "After the first time you do this, to overwrite the existing copy of the assignment in the exchange:\n",
    "    \n",
    "    nbgrader release \"06. Networks\" --force\n",
    "    \n",
    "- Example output:\n",
    "\n",
    "        $ nbgrader release \"06. Networks\" --force\n",
    "        [ReleaseApp | INFO] Overwriting files: 2015-fall-big_data 06. Networks\n",
    "        [ReleaseApp | INFO] Source: /home/instructor/nbgrader/courses/2015-fall-big_data/release/06. Networks\n",
    "        [ReleaseApp | INFO] Destination: /srv/nbgrader/exchange/2015-fall-big_data/outbound/06. Networks\n",
    "        [ReleaseApp | INFO] Released as: 2015-fall-big_data 06. Networks\n",
    "\n",
    "- Example output if you forget the `--force`:\n",
    "\n",
    "        $ nbgrader release \"06. Networks\"\n",
    "        [ReleaseApp | ERROR] Destination already exists, add --force to overwrite: 2015-fall-big_data 06. Networks\n",
    "        \n",
    "Releasing an assignment places all of that assignment's notebooks in `<exchange_folder>/<class>/outbound/<assignment>/`.  So, for the default exchange directory path (\"/srv/nbgrader/exchange\"), class \"2015-fall-big_data\" and assignment \"06. Networks\", the assignment's notebooks would be placed in:\n",
    "\n",
    "    /srv/nbgrader/exchange/2015-fall-big_data/outbound/06. Networks/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. STUDENT - Find assignments - `nbgrader list`\n",
    "\n",
    "- Back to the [Table of Contents](#Table-of-Contents)\n",
    "\n",
    "Once the assignment has been released, the student can see an assignment has been released by either:\n",
    "\n",
    "- looking in the \"Released assignments\" section of the \"Assignments\" tab.\n",
    "    \n",
    "- OR checking the output of the nbgrader `nbgrader list` command:\n",
    "\n",
    "        nbgrader list\n",
    "        \n",
    "    - Example output:\n",
    "    \n",
    "            $ nbgrader list\n",
    "            [ListApp | INFO] Released assignments:\n",
    "            [ListApp | INFO] 2015-fall-big_data 06. Networks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. STUDENT - Fetch assignment - `nbgrader fetch \"<assignment>\"`\n",
    "\n",
    "- Back to the [Table of Contents](#Table-of-Contents)\n",
    "\n",
    "Students can download a released assignment to work on it by either:\n",
    "\n",
    "- clicking on the \"Fetch\" button next to the asignment in the **\"Released assignments\"** section of the **\"Assignments\"** tab.\n",
    "\n",
    "- OR running the `nbgrader fetch \"<assignment>\"` command:\n",
    "    \n",
    "        nbgrader fetch \"08. Machine Learning\"\n",
    "     \n",
    "    - Example output:\n",
    "    \n",
    "            $ nbgrader fetch \"06. Networks\"\n",
    "            [FetchApp | INFO] Source: /srv/nbgrader/exchange/2015-fall-big_data/outbound/06. Networks\n",
    "            [FetchApp | INFO] Destination: /home/jmorgan/Downloads/06. Networks\n",
    "            [FetchApp | INFO] Fetched as: 2015-fall-big_data 06. Networks\n",
    "            \n",
    "If you use the `nbgrader fetch` command, your assignments will be downloaded into the folder in which you run the command.  The \"Fetch\" button will always download your assignments into your home folder.  If you don't want the assignments downloaded directly into your home folder, you must use the command line command rather than the Assignments tab, as the Assignments tab doesn't know to look anywhere other than your home folder for assignments."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. STUDENT - Submit assignment - `nbgrader submit \"<assignment>\"`\n",
    "\n",
    "- Back to the [Table of Contents](#Table-of-Contents)\n",
    "\n",
    "Once the student has completed the notebook(s) that make up the assignment, they can turn the assignment in by either:\n",
    "\n",
    "- clicking on the \"Submit\" button next to the assignment in the \"Downloaded assignments\" section of the \"Assignments\" tab.\n",
    "- OR running the `nbgrader submit \"<assignment>\"` command:\n",
    "    \n",
    "        nbgrader submit \"06. Networks\"\n",
    "            \n",
    "If you used the `nbgrader fetch \"<assignment>\"` command to fetch an assignment to a folder other than your home folder, you will have to use the `nbgrader submit \"<assignment>\"` command in the folder where you downloaded the assignment to submit it, again because the Assignments tab doesn't know to look anywhere other than your home folder for assignments.\n",
    "\n",
    "After a student submits their assignment, the updated notebook(s) are stored in `<exchange_folder>/inbound`, in a folder named `<student_user>+<assignment>+<timestamp>` (example: `jmorgan+06. Networks+2015-08-29 23:33:29 UTC`).  Inside, each notebook submitted by the student for the assignment is stored, as well as a file named `timestamp.txt` that contains the same time stamp as is appended to the folder name."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6a. STUDENT - List submitted assignments - `nbgrader list --cached`\n",
    "\n",
    "- Back to the [Table of Contents](#Table-of-Contents)\n",
    "\n",
    "A student can see which assignments they have submitted using the `nbgrader list` command:\n",
    "\n",
    "    nbgrader list --cached\n",
    "        \n",
    "- Example output (course, then username, then assignment, then submission date):\n",
    "\n",
    "        $ nbgrader list --cached\n",
    "        [ListApp | INFO] Submitted assignments:\n",
    "        [ListApp | INFO] 2015-fall-big_data jmorgan 06. Networks 2015-08-29 23:33:29 UTC"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6b. List assignments with submissions - `nbgrader list --inbound`\n",
    "\n",
    "- Back to the [Table of Contents](#Table-of-Contents)\n",
    "\n",
    "To see which assignments have submissions which can be graded, in the courses directory, the instructor can use the `nbgrader list` command:\n",
    "\n",
    "    nbgrader list --inbound\n",
    "        \n",
    "- Example output (course, then username, then assignment, then submission date):\n",
    "\n",
    "        $ nbgrader list --inbound\n",
    "        [ListApp | INFO] Submitted assignments:\n",
    "        [ListApp | INFO] 2015-fall-big_data jmorgan 06. Networks 2015-08-29 23:33:29 UTC\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7. Collect assignments - `nbgrader collect \"<assignment>\"`\n",
    "\n",
    "- Back to the [Table of Contents](#Table-of-Contents)\n",
    "\n",
    "To collect submitted assignments for grading, after you see submissions in `nbgrader list --inbound`, use the `nbgrader collect \"<assignment>\"` command to collect assignments back into the instructor's grading area from the exchange folder:\n",
    "\n",
    "     nbgrader collect \"06. Networks\"\n",
    "        \n",
    "- Example output:\n",
    "\n",
    "        $ nbgrader collect \"06. Networks\"\n",
    "        [CollectApp | INFO] Collecting submission: jmorgan 06. Networks\n",
    "        \n",
    "After running `nbgrader collect \"<assignment>\"`, the collected assignments are stored in the `<course_directory>/submitted` folder.  Inside, each student has a directory that matches their unix username.\n",
    "\n",
    "Each student's directory contains a folder for each assignment.  Inside each assignment folder are the notebooks that were collected and a timestamp file that holds the date and time the assignment was submitted.\n",
    "\n",
    "After the first time you do this, if students have resubmitted, the output will look like this:\n",
    "\n",
    "    $ nbgrader collect \"02. Database Basics\"\n",
    "    [CollectApp | INFO] Submission already exists, use --update to update: kwilken 02. Database Basics\n",
    "    [CollectApp | INFO] Submission already exists, use --update to update: wchang 02. Database Basics\n",
    "    [CollectApp | INFO] Submission already exists, use --update to update: acaporaso 02. Database Basics\n",
    "    [CollectApp | INFO] Collecting submission: adelano 02. Database Basics\n",
    "    [CollectApp | INFO] Submission already exists, use --update to update: smcdonald 02. Database Basics\n",
    "    [CollectApp | INFO] Submission already exists, use --update to update: eross 02. Database Basics\n",
    "    [CollectApp | INFO] Collecting submission: pmiller 02. Database Basics\n",
    "    [CollectApp | INFO] Submission already exists, use --update to update: jmorgan 02. Database Basics\n",
    "    [CollectApp | INFO] Collecting submission: ggraham 02. Database Basics\n",
    "    [CollectApp | INFO] Submission already exists, use --update to update: thill 02. Database Basics\n",
    "    [CollectApp | INFO] Collecting submission: dglennon 02. Database Basics\n",
    "    [CollectApp | INFO] Collecting submission: coslund 02. Database Basics\n",
    "    [CollectApp | INFO] Submission already exists, use --update to update: wkingkade 02. Database Basics\n",
    "    [CollectApp | INFO] Submission already exists, use --update to update: nczaplicki 02. Database Basics\n",
    "    [CollectApp | INFO] Collecting submission: azotti 02. Database Basics\n",
    "    [CollectApp | INFO] Submission already exists, use --update to update: jmenza 02. Database Basics\n",
    "    [CollectApp | INFO] Collecting submission: dbryant 02. Database Basics\n",
    "    [CollectApp | INFO] Submission already exists, use --update to update: mbenetsky 02. Database Basics\n",
    "    [CollectApp | INFO] Submission already exists, use --update to update: ebyerly 02. Database Basics\n",
    "    [CollectApp | INFO] Collecting submission: cwurster 02. Database Basics\n",
    "\n",
    "\n",
    "To overwrite the existing copy of the assignment with the latest:\n",
    "\n",
    "    nbgrader collect \"06. Networks\" --update\n",
    "\n",
    "- Example output:\n",
    "\n",
    "        $ nbgrader collect \"02. Database Basics\" --update\n",
    "        [CollectApp | INFO] No newer submission to collect: thill 02. Database Basics\n",
    "        [CollectApp | INFO] Updating submission: eross 02. Database Basics\n",
    "        [CollectApp | INFO] No newer submission to collect: dbryant 02. Database Basics\n",
    "        [CollectApp | INFO] No newer submission to collect: cwurster 02. Database Basics\n",
    "        [CollectApp | INFO] No newer submission to collect: azotti 02. Database Basics\n",
    "        [CollectApp | INFO] No newer submission to collect: wkingkade 02. Database Basics\n",
    "        [CollectApp | INFO] No newer submission to collect: pmiller 02. Database Basics\n",
    "        [CollectApp | INFO] Updating submission: smcdonald 02. Database Basics\n",
    "        [CollectApp | INFO] No newer submission to collect: jmorgan 02. Database Basics\n",
    "        [CollectApp | INFO] No newer submission to collect: ebyerly 02. Database Basics\n",
    "        [CollectApp | INFO] No newer submission to collect: dglennon 02. Database Basics\n",
    "        [CollectApp | INFO] No newer submission to collect: acaporaso 02. Database Basics\n",
    "        [CollectApp | INFO] No newer submission to collect: jmenza 02. Database Basics\n",
    "        [CollectApp | INFO] No newer submission to collect: coslund 02. Database Basics\n",
    "        [CollectApp | INFO] No newer submission to collect: wchang 02. Database Basics\n",
    "        [CollectApp | INFO] No newer submission to collect: kwilken 02. Database Basics\n",
    "        [CollectApp | INFO] No newer submission to collect: mbenetsky 02. Database Basics\n",
    "        [CollectApp | INFO] No newer submission to collect: adelano 02. Database Basics\n",
    "        [CollectApp | INFO] No newer submission to collect: ggraham 02. Database Basics\n",
    "        [CollectApp | INFO] No newer submission to collect: nczaplicki 02. Database Basics\n",
    "        \n",
    "As you can see, just because a student has an existing submission doesn't mean that they have submitted a new version.  Update is smart enough to only update if there is a submission newer than the one already stored for the student."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 8. Auto-grade assignments - `nbgrader autograde \"<assignment>\"`\n",
    "\n",
    "- Back to the [Table of Contents](#Table-of-Contents)\n",
    "\n",
    "After you have collected some assignments, you will autograde them using the `nbgrader autograde \"<assignment>\"` command:\n",
    "\n",
    "    nbgrader autograde \"06. Networks\"\n",
    "    \n",
    "- Example output:\n",
    "\n",
    "        [AutogradeApp | INFO] Copying submitted/jmorgan/06. Networks/timestamp.txt -> autograded/jmorgan/06. Networks/timestamp.txt\n",
    "        [AutogradeApp | INFO] SubmittedAssignment<06. Networks for jmorgan> submitted at 2015-08-29 23:33:29\n",
    "        [AutogradeApp | INFO] Overwriting files with master versions from the source directory\n",
    "        [AutogradeApp | INFO] Sanitizing submitted/jmorgan/06. Networks/networks_exercise.ipynb\n",
    "        [AutogradeApp | INFO] Converting notebook submitted/jmorgan/06. Networks/networks_exercise.ipynb to notebook\n",
    "        [AutogradeApp | INFO] Writing 8008 bytes to autograded/jmorgan/06. Networks/networks_exercise.ipynb\n",
    "        [AutogradeApp | INFO] Autograding autograded/jmorgan/06. Networks/networks_exercise.ipynb\n",
    "        [AutogradeApp | INFO] Converting notebook autograded/jmorgan/06. Networks/networks_exercise.ipynb to notebook\n",
    "        [AutogradeApp | INFO] Executing notebook with kernel: python2\n",
    "        [AutogradeApp | INFO] Writing 10097 bytes to autograded/jmorgan/06. Networks/networks_exercise.ipynb\n",
    "        [AutogradeApp | INFO] Setting destination file permissions to 444\n",
    "\n",
    "You must auto-grade all assignments, even those without auto-graded questions, since the form grade app also depends on the output of this command.\n",
    "\n",
    "Once assignments are auto-graded, they are stored in the `<course_directory>/autograded` folder.  Inside, each student has a directory that matches their unix username.\n",
    "\n",
    "Each student's directory contains a folder for each assignment.  Inside each assignment folder are the notebooks that were collected and a timestamp file that holds the date and time the assignment was auto-graded.\n",
    "\n",
    "**_ NOTE: For classes with 10 or more students, auto-grading can take a few minutes.  If you are SSHed in to the server on which you are grading, it is probably a good idea to run the autograde process in a GNU screen session, so it will continue if you are disconnected._**\n",
    "\n",
    "After the first time you do this, to re-autograde (overwrite the existing auto-graded version of the assignment):\n",
    "    \n",
    "    nbgrader autograde \"07. mini Machine Learning\" --force\n",
    "        \n",
    "- Example output:\n",
    "    \n",
    "        instructor@ip-172-31-36-239:~/nbgrader/courses/2015-fall-big_data$ nbgrader autograde \"07. mini Machine Learning\" --force\n",
    "        [AutogradeApp | WARNING] Removing existing assignment: autograded/jmorgan/07. mini Machine Learning\n",
    "        [AutogradeApp | INFO] Copying submitted/jmorgan/07. mini Machine Learning/timestamp.txt -> autograded/jmorgan/07. mini Machine Learning/timestamp.txt\n",
    "        [AutogradeApp | INFO] Copying submitted/jmorgan/07. mini Machine Learning/asgt7_cleaned.msg -> autograded/jmorgan/07. mini Machine Learning/asgt7_cleaned.msg\n",
    "        [AutogradeApp | INFO] SubmittedAssignment<07. mini Machine Learning for jmorgan> submitted at 2015-11-09 17:40:07\n",
    "        [AutogradeApp | INFO] Overwriting files with master versions from the source directory\n",
    "        [AutogradeApp | INFO] Sanitizing submitted/jmorgan/07. mini Machine Learning/Machine Learning.ipynb\n",
    "        [AutogradeApp | INFO] Converting notebook submitted/jmorgan/07. mini Machine Learning/Machine Learning.ipynb to notebook\n",
    "        [AutogradeApp | INFO] Writing 71417 bytes to autograded/jmorgan/07. mini Machine Learning/Machine Learning.ipynb\n",
    "        [AutogradeApp | INFO] Autograding autograded/jmorgan/07. mini Machine Learning/Machine Learning.ipynb\n",
    "        [AutogradeApp | INFO] Converting notebook autograded/jmorgan/07. mini Machine Learning/Machine Learning.ipynb to notebook\n",
    "        [AutogradeApp | INFO] Executing notebook with kernel: python2\n",
    "        [AutogradeApp | INFO] Writing 100207 bytes to autograded/jmorgan/07. mini Machine Learning/Machine Learning.ipynb\n",
    "        [AutogradeApp | INFO] Setting destination file permissions to 444\n",
    "        \n",
    "- Example output if you forget the `--force`:\n",
    "\n",
    "        instructor@ip-172-31-36-239:~/nbgrader/courses/2015-fall-big_data$ nbgrader autograde \"07. mini Machine Learning\"\n",
    "        [AutogradeApp | INFO] Skipping existing assignment: autograded/jmorgan/07. mini Machine Learning\n",
    "\n",
    "### Troubleshooting\n",
    "\n",
    "If your students create backup copies of notebooks in their assignment folders, `nbgrader autograde ...` will crash when it encounters an ipython notebook in the directory that it isn't expecting.  It might also crash for other reasons...  To troubleshoot:\n",
    "\n",
    "#### `--notebook`\n",
    "\n",
    "One troubleshooting strategy is to explicitly limit auto-grading to specific notebooks.  For example, for an assignment \"02. Database Basics\", which in its initial form had one nbgrader notebook (`Data_and_database.ipynb`) and lots of other files and folders, and for which some students who struggled create multiple backup copies of this IPython Notebook with different names, all that is fine as long as you use the `--notebook` parameter to tell nbgrader to just grade `Data_and_databases.ipynb`.  \n",
    "\n",
    "Syntax:\n",
    "\n",
    "    nbgrader autograde \"<assignment_name>\" --notebook \"<notebook_name_pattern>\"\n",
    "    \n",
    "WHERE:\n",
    "\n",
    "- `<assignment_name>` is the assignment you are grading.\n",
    "- `<notebook_name_pattern>` is a file system name match pattern like those used in unix where an asterisk ( \"`*`\" ) matches zero to many characters anywhere you place it in a pattern string.  This will be compared to the complete name of each file in the assignment directory (so must account for the extension as well - like `*.ipynb`).  Example: only files that start with \"Data\" and end with \".ipynb\" - \"`Data*.ipynb`\".\n",
    "    \n",
    "Example command for scenario above (assignment \"02. Database Basics\", limit to notebook \"Data_and_databases.ipynb\"):\n",
    "\n",
    "    nbgrader autograde \"02. Database Basics\" --notebook \"Data_and_databases.ipynb\"\n",
    "    \n",
    "#### `--student`\n",
    "\n",
    "Another troubleshooting strategy is to just autograde a given student, using the `--student` parameter.\n",
    "\n",
    "Syntax:\n",
    "\n",
    "    nbgrader autograde \"<assignment_name>\" --student \"<student_username>\"\n",
    "    \n",
    "WHERE:\n",
    "\n",
    "- `<student_username>` is the username of the student, from when they were added to the gradebook.\n",
    "\n",
    "Example:\n",
    "\n",
    "    nbgrader autograde \"02. Database Basics\" --student \"dglennon\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 9. Grade using the Formgrade Application - `nbgrader formgrade`\n",
    "\n",
    "- Back to the [Table of Contents](#Table-of-Contents)\n",
    "\n",
    "To access the formgrade application, you will combine the URL you use to access the base jupyterhub and combine it with the path from the \"Proxying\" line above.  For example, for jupyterhub configured to be non-ssl, with domain of \"bigdataforsocialscience.com\" and hub listening on port 8000:\n",
    "\n",
    "- [http://bigdataforsocialscience.com:8000/hub/nbgrader/2015-fall-big_data](http://bigdataforsocialscience.com:8000/hub/nbgrader/2015-fall-big_data)\n",
    "\n",
    "For the same, but with SSL certificates enabled:\n",
    "\n",
    "- [https://bigdataforsocialscience.com:8000/hub/nbgrader/2015-fall-big_data](https://bigdataforsocialscience.com:8000/hub/nbgrader/2015-fall-big_data)\n",
    "\n",
    "The formgrade application must be started and run by the instructor user.  Once it is up and running, though, any user whose username is in the list of `c.HubAuth.graders` in `<course_folder>/nbgrader_config.py` can use the URL to connect and grade.\n",
    "\n",
    "Configuring the formgrade application can be tricky.  For more information on troubleshooting, see \"`nbgrader formgrade` issues\" in the troubleshooting section below.\n",
    "\n",
    "For more information on using the formgrade application, see the help located within the formgrade application itself.\n",
    "\n",
    "### Setting up, starting, and stopping the Formgrade application\n",
    "\n",
    "- Back to the [Table of Contents](#Table-of-Contents)\n",
    "\n",
    "In general, you'll leave the Formgrade application running on your server.  Initially, though, you'll need to configure it and set it up.  Here's how to do that.\n",
    "\n",
    "Before you start the formgrade app, you need to have its properties configured in the nbgrader_config.py file that is in the same folder as this file.  For more details, see [http://nbgrader.readthedocs.org/en/stable/user_guide/11_jupyterhub_config.html#configuring-nbgrader-formgrade](http://nbgrader.readthedocs.org/en/stable/user_guide/11_jupyterhub_config.html#configuring-nbgrader-formgrade).\n",
    "\n",
    "To start the formgrade app while logged in as the instructor user:\n",
    "\n",
    "- open a screen session in which you'll let the app run:\n",
    "\n",
    "        screen\n",
    "        \n",
    "- make sure you are in the `<course_directory>`.\n",
    "\n",
    "        pwd\n",
    "\n",
    "- export the same `CONFIGPROXY_AUTH_TOKEN` environment variable value as is used to start the jupyterhub itself:\n",
    "\n",
    "        export CONFIGPROXY_AUTH_TOKEN='<CONFIGPROXY_AUTH_TOKEN>'\n",
    "    \n",
    "- run `nbgrader formgrade`:\n",
    "\n",
    "        nbgrader formgrade\n",
    "        \n",
    "    - example output:\n",
    "    \n",
    "            $ nbgrader formgrade\n",
    "            [FormgradeApp | INFO] Proxying /hub/nbgrader/2015-fall-big_data --> http://127.0.0.1:9000\n",
    "            [FormgradeApp | INFO] Serving MathJax from /usr/local/lib/python3.4/dist-packages/notebook/static/components/MathJax/\n",
    "            [FormgradeApp | INFO] Form grader running at http://127.0.0.1:9000/\n",
    "            [FormgradeApp | INFO] Use Control-C to stop this server\n",
    "            * Running on http://127.0.0.1:9000/ (Press CTRL+C to quit)\n",
    "\n",
    "- detach from the screen session by pressing \"Control\" and \"A\" together, then pressing \"D\" once you release Control and A.\n",
    "\n",
    "At this point, the FormGrade application is running in the background in this screen session.\n",
    "\n",
    "To stop it:\n",
    "\n",
    "- log into the server as the instructor user.\n",
    "- reconnect to the screen session:\n",
    "\n",
    "        screen -r\n",
    "        \n",
    "- Press \"Control\" and \"C\" together to send the kill signal to the application.\n",
    "\n",
    "To start it again:\n",
    "\n",
    "- run `nbgrader formgrade`:\n",
    "\n",
    "        nbgrader formgrade\n",
    "\n",
    "If your server is restarted, you'll need to go through all of the startup steps again to start it up."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 10. Generating feedback for students - `nbgrader feedback \"<assignment>\"`\n",
    "\n",
    "- Back to the [Table of Contents](#Table-of-Contents)\n",
    "\n",
    "Once you have graded and provided feedback on assignments with the formgrade application, you can then generate HTML files that contain your feedback and return them to the students using the `nbgrader feedback \"<assignment>\"` command:\n",
    "\n",
    "    nbgrader feedback \"06. Networks\"\n",
    "\n",
    "- Example output:\n",
    "\n",
    "        $ nbgrader feedback \"06. Networks\"\n",
    "        [FeedbackApp | INFO] Copying autograded/jmorgan/06. Networks/timestamp.txt -> feedback/jmorgan/06. Networks/timestamp.txt\n",
    "        [FeedbackApp | INFO] Converting notebook autograded/jmorgan/06. Networks/networks_exercise.ipynb to html\n",
    "        [FeedbackApp | INFO] Writing 210591 bytes to feedback/jmorgan/06. Networks/networks_exercise.html\n",
    "        [FeedbackApp | INFO] Setting destination file permissions to 444\n",
    "\n",
    "After the first time you do this, to overwrite the existing copy of the assignment's feedback in both the course folder and in the exchange:\n",
    "    \n",
    "    nbgrader feedback \"06. Networks\" --force\n",
    "    \n",
    "- Example output:\n",
    "\n",
    "        $ nbgrader feedback \"06. Networks\" --force\n",
    "        [FeedbackApp | WARNING] Removing existing assignment: feedback/jmorgan/06. Networks\n",
    "        [FeedbackApp | INFO] Copying autograded/jmorgan/06. Networks/timestamp.txt -> feedback/jmorgan/06. Networks/timestamp.txt\n",
    "        [FeedbackApp | INFO] Converting notebook autograded/jmorgan/06. Networks/networks_exercise.ipynb to html\n",
    "        [FeedbackApp | INFO] Writing 210591 bytes to feedback/jmorgan/06. Networks/networks_exercise.html\n",
    "        [FeedbackApp | INFO] Setting destination file permissions to 444\n",
    "\n",
    "- Example output if you forget the `--force`:\n",
    "\n",
    "        $ nbgrader feedback \"06. Networks\"\n",
    "        [FeedbackApp | INFO] Skipping existing assignment: feedback/jmorgan/06. Networks\n",
    "i\n",
    "        \n",
    "After running `nbgrader feedback \"<assignment>\"`, all autograded notebooks for the assignment are converted to HTML, including feedback, and the HTML is moved to the `<course_directory>/feedback` folder, which is structured just like the autograded folder, except the assignments have HTML feedback rather than notebooks.\n",
    "\n",
    "Inside, each student has a directory that matches their unix username.  Each student's directory contains a folder for each assignment.  Inside each assignment folder are the HTML rendering of the assignment's notebooks that include grading and feedback, and a timestamp file that holds the date and time the assignment was graded."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 11. Getting feedback back to student\n",
    "\n",
    "- Back to the [Table of Contents](#Table-of-Contents)\n",
    "\n",
    "Run the following code to turn HTML feedback into PDFs and send PDF feedback to each user.\n",
    "\n",
    "Before you run this:\n",
    "\n",
    "- Make sure that the packages `texlive-full` and `pandoc` are installed on your system:\n",
    "\n",
    "         (sudo) apt-get install texlive-full\n",
    "         (sudo) apt-get install pandoc\n",
    "\n",
    "- Make sure to set the following variables in the code below:\n",
    "\n",
    "### nbgrader settings\n",
    "\n",
    "- **`nbgrader_home`** = to the full path to the nbgrader home.\n",
    "- **`assignment_name`** = to the name of the assignment whose feedback you want to generate and email.\n",
    "\n",
    "### SMTP server settings\n",
    "\n",
    "- **`smtp_host`** = string hostname of SMTP server we'll use to send feedback.\n",
    "- **`smtp_port`** = port number of SMTP server (587 is default for TLS).\n",
    "- **`smtp_host_username`** = string username for SMTP server authentication.\n",
    "- **`smtp_host_password`** = string password for SMTP server authentication.\n",
    "\n",
    "### email message settings (optional)\n",
    "\n",
    "- **`message_from`** = email address from which the message is being sent.\n",
    "- **`message_subject`** = String subject of the message - by default, is \"SURV 699Y Fall 2015 Course - Feedback for \" + assignment_name\n",
    "- **`message_body`** = contents of the message you want to accompany the feedback PDFs.\n",
    "- **`custom_message_dict[ '<username>' ]`** = a custom message you want for a given student.\n",
    "- **`message_signature`** = signature string, which will follow message and optional custom message.\n",
    "- **`message_bcc`** = email addresses you want blind-copied on each message."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "#==============================================================================#\n",
    "# imports\n",
    "#==============================================================================#\n",
    "\n",
    "import mimetypes\n",
    "import os\n",
    "import sqlite3\n",
    "import subprocess\n",
    "import sys\n",
    "import traceback\n",
    "\n",
    "# imports - sending email\n",
    "from email.mime.application import MIMEApplication\n",
    "from email.mime.multipart import MIMEMultipart\n",
    "from email.mime.text import MIMEText\n",
    "import smtplib\n",
    "\n",
    "#==============================================================================#\n",
    "# declare variables\n",
    "#==============================================================================#\n",
    "\n",
    "nbgrader_home = \"\"\n",
    "assignment_name = \"\"\n",
    "current_directory_path = \"\"\n",
    "feedback_path = \"\"\n",
    "my_connection = None\n",
    "my_cursor = None\n",
    "sql_string = \"\"\n",
    "result_set = None\n",
    "student_counter = -1\n",
    "current_row = None\n",
    "\n",
    "# declare variables - SMTP server\n",
    "smtp_host = ''\n",
    "smtp_host_username = ''\n",
    "smtp_host_password = ''\n",
    "smtp_port = 587\n",
    "smtp_server = None\n",
    "smtp_send_result = None\n",
    "\n",
    "# declare variables - email message\n",
    "multipart_message = \"\"\n",
    "message_from = \"\"\n",
    "message_subject = \"\"\n",
    "message_body = \"\"\n",
    "custom_message_dict = {}\n",
    "message_signature = \"\"\n",
    "complete_message_body = \"\"\n",
    "message_bcc = \"\"\n",
    "\n",
    "# declare variables - process assignment\n",
    "assignment_result_set = None\n",
    "assignment_count = -1\n",
    "assignment_row = None\n",
    "assignment_id = \"\"\n",
    "notebook_result_set = None\n",
    "notebook_row = None\n",
    "notebook_name = \"\"\n",
    "assignment_notebook_list = []\n",
    "is_assignment_ok = True\n",
    "\n",
    "# declare variables - process student\n",
    "student_username = \"\"\n",
    "student_email = \"\"\n",
    "student_first_name = \"\"\n",
    "student_last_name = \"\"\n",
    "student_feedback_path = \"\"\n",
    "pandoc_command_list = []\n",
    "pandoc_command_result = None\n",
    "#file_list = []\n",
    "file_name = \"\"\n",
    "#file_name_lower = \"\"\n",
    "pdf_file_path = \"\"\n",
    "pdf_file_name_list = []\n",
    "pdf_file_count = -1\n",
    "content_type = \"\"\n",
    "main_type = \"\"\n",
    "sub_type = \"\"\n",
    "encoding = \"\"\n",
    "text_message = None\n",
    "pdf_message = None\n",
    "\n",
    "#==============================================================================#\n",
    "# initialize variables\n",
    "#==============================================================================#\n",
    "\n",
    "#------------------------------------------------------------------------------#\n",
    "# == initialize variables - nbgrader\n",
    "#------------------------------------------------------------------------------#\n",
    "\n",
    "# example: nbgrader_home = \"/home/instructor/nbgrader/courses/2015-fall-big_data\"\n",
    "nbgrader_home = \"\"\n",
    "# example: assignment_name = \"02. Database Basics\"\n",
    "assignment_name = \"\"\n",
    "\n",
    "#------------------------------------------------------------------------------#\n",
    "# == initialize variables - email message\n",
    "#------------------------------------------------------------------------------#\n",
    "\n",
    "message_from = \"\"\n",
    "message_subject = \"Feedback for \" + assignment_name\n",
    "message_body = \"\"\"Hello,\n",
    "\n",
    "Attached is a PDF with feedback on assignment \"\"\"\n",
    "message_body += assignment_name\n",
    "message_body += \"\"\" (it is a copy of your assignment, with “Comments” after each answer).  Please let me know if you have any questions.\n",
    "\"\"\"\n",
    "#custom_message_dict[ \"jmorgan\" ] = \"test custom message!\"\n",
    "message_signature = \"\"\"\n",
    "Thanks!\n",
    "\n",
    "Jon\n",
    "\"\"\"\n",
    "message_bcc = \"\"\n",
    "\n",
    "#------------------------------------------------------------------------------#\n",
    "# == initialize variables - SMTP server\n",
    "#------------------------------------------------------------------------------#\n",
    "\n",
    "smtp_host = ''\n",
    "smtp_host_username = ''\n",
    "smtp_host_password = ''\n",
    "smtp_port = 587\n",
    "# smtp_use_TLS = True - always use TLS.\n",
    "\n",
    "#==============================================================================#\n",
    "# Process feedback\n",
    "#==============================================================================#\n",
    "\n",
    "# get current working directory, use that to make path to feedback directory.\n",
    "current_directory_path = os.getcwd()\n",
    "feedback_path = current_directory_path + \"/feedback\"\n",
    "\n",
    "print( \"Feedback path = \" + feedback_path )\n",
    "\n",
    "# must run this from nbgrader_home.\n",
    "if( nbgrader_home == current_directory_path ):\n",
    "\n",
    "    # must have an assignment.\n",
    "    if ( ( assignment_name is not None ) and ( assignment_name != \"\" ) ):\n",
    "\n",
    "        # use try-->except-->finally to make sure you always close your database\n",
    "        #    connections.\n",
    "        try:\n",
    "\n",
    "            # make a connection - pass the name of the database file.\n",
    "            my_connection = sqlite3.connect( 'gradebook.db' )\n",
    "\n",
    "            # set row_factory that returns values mapped to column names\n",
    "            #   as well as in an ordered list\n",
    "            my_connection.row_factory = sqlite3.Row\n",
    "\n",
    "            # then, make a cursor.\n",
    "            my_cursor = my_connection.cursor()\n",
    "\n",
    "            #------------------------------------------------------------------#\n",
    "            # Assignment processing.\n",
    "            #------------------------------------------------------------------#\n",
    "            \n",
    "            # set assignment OK flag to True to start.\n",
    "            is_assignment_ok = True\n",
    "            \n",
    "            # retrieve assignment record for assignment name.\n",
    "            sql_string = \"SELECT * FROM assignment WHERE name = ?;\"\n",
    "            assignment_result_set = my_cursor.execute( sql_string, ( assignment_name, ) )\n",
    "            \n",
    "            # get assignment ID.\n",
    "            assignment_count = 0\n",
    "            for assignment_row in assignment_result_set:\n",
    "                \n",
    "                # increment count\n",
    "                assignment_count += 1\n",
    "                assignment_id = assignment_row[ \"id\" ]\n",
    "                \n",
    "                print( \"--> Found assignment ID \\\"\" + assignment_id + \"\\\"\" )\n",
    "                \n",
    "            #-- END loop over records to get ID. --#\n",
    "            \n",
    "            # how many?\n",
    "            if ( assignment_count == 1 ):\n",
    "            \n",
    "                # 1 - right answer.\n",
    "                \n",
    "                # Retrieve names of the notebooks that are formally part of the assignment.\n",
    "                sql_string = \"SELECT * FROM notebook WHERE assignment_id = ?;\"\n",
    "                notebook_result_set = my_cursor.execute( sql_string, ( assignment_id, ) )\n",
    "                \n",
    "                # loop over results, storing each name in assignment_notebook_list.\n",
    "                for notebook_row in notebook_result_set:\n",
    "                    \n",
    "                    # retrieve name.\n",
    "                    notebook_name = notebook_row[ \"name\" ]\n",
    "                    \n",
    "                    print( \"----> Found notebook \\\"\" + notebook_name + \"\\\"\" )\n",
    "                    \n",
    "                    # add to list.\n",
    "                    assignment_notebook_list.append( notebook_name )\n",
    "                    \n",
    "                #-- END loop over notebooks. --#\n",
    "                \n",
    "                # got at least 1 notebook name?\n",
    "                if ( len( assignment_notebook_list ) <= 0 ):\n",
    "                    \n",
    "                    # no notebooks.  Nothing to do here.\n",
    "                    print( \"No Notebooks for assignment \" + assignment_name + \" - can not proceed.\" )\n",
    "                    is_assignment_ok = False\n",
    "                    \n",
    "                #-- END check to see if notebooks found. --#\n",
    "\n",
    "            else:\n",
    "                \n",
    "                # not 1.  0?\n",
    "                if ( assignment_count == 0 ):\n",
    "                    \n",
    "                    # 0 - unknown assignment.  Error.\n",
    "                    print( \"Unknown assignemnt: \" + assignment_name + \" - can not proceed.\" )\n",
    "                    is_assignment_ok = False\n",
    "                    \n",
    "                elif ( assignment_count > 1 ):\n",
    "                    \n",
    "                    # more than 1.  Error.\n",
    "                    print( \"Multiple matches for assignment: \" + assignment_name + \" - can not proceed.\" )\n",
    "                    is_assignment_ok = False\n",
    "                \n",
    "                else:\n",
    "                    \n",
    "                    # -1?  Error.\n",
    "                    print( \"Unknown error for assignment: \" + assignment_name + \" - count less than 0.  Can not proceed.\" )\n",
    "                    is_assignment_ok = False\n",
    "                    \n",
    "                #-- END check to see what is going on with result not equal to 1. --#\n",
    "                \n",
    "            #-- END check to see if 1 assignment returned. --#\n",
    "                    \n",
    "            # is assignment OK?\n",
    "            if ( is_assignment_ok == True ):\n",
    "            \n",
    "                # initialize query to count loop over students.\n",
    "                sql_string = \"SELECT * FROM student ORDER BY last_name ASC, first_name ASC;\"\n",
    "                #sql_string = \"SELECT * FROM student WHERE id IN ( 'jmorgan' ) ORDER BY last_name ASC, first_name ASC;\"\n",
    "                #sql_string = \"SELECT * FROM student WHERE id NOT IN ( 'cjones', 'aemad' ) ORDER BY last_name ASC, first_name ASC;\"\n",
    "\n",
    "                # Actually use cursor here...\n",
    "                result_set = my_cursor.execute( sql_string )\n",
    "\n",
    "                # output results\n",
    "                print( \"Student list:\" )\n",
    "\n",
    "                # open SMPT server for sending email.\n",
    "                with smtplib.SMTP( host = smtp_host, port = smtp_port ) as smtp_server:\n",
    "\n",
    "                    # authenticate and start TLS\n",
    "                    smtp_server.starttls()\n",
    "                    smtp_server.login( smtp_host_username, smtp_host_password )\n",
    "\n",
    "                    # loop over students in result set.\n",
    "                    student_counter = 0\n",
    "                    for current_row in result_set:\n",
    "\n",
    "                        # increment student counter\n",
    "                        student_counter += 1\n",
    "\n",
    "                        # initialize variables\n",
    "                        student_username = \"\"\n",
    "                        student_email = \"\"\n",
    "                        student_first_name = \"\"\n",
    "                        student_last_name = \"\"\n",
    "                        pandoc_command_list = []\n",
    "                        pdf_file_name_list = []\n",
    "                        pdf_file_count = -1\n",
    "\n",
    "                        print( \"\\n\\n==> Student \" + str( student_counter ) + \" = \" + current_row[ \"first_name\" ] + \" \" + current_row[ \"last_name\" ] + \" ( \" + current_row[ \"id\" ] + \" ) - \" + current_row[ \"email\" ] )\n",
    "\n",
    "                        # get student information\n",
    "                        student_username = current_row[ \"id\" ]\n",
    "                        student_email = current_row[ \"email\" ]\n",
    "                        student_first_name = current_row[ \"first_name\" ]\n",
    "                        student_last_name = current_row[ \"last_name\" ]\n",
    "\n",
    "                        # generate student feedback path for student and assignment\n",
    "                        student_feedback_path = feedback_path + \"/\" + student_username + \"/\" + assignment_name\n",
    "\n",
    "                        print( \"----> Student feedback path = \" + student_feedback_path )\n",
    "                        \n",
    "                        # use try in case student doesn't have feedback\n",
    "                        try:\n",
    "\n",
    "                            # list contents of directory\n",
    "                            file_list = os.listdir( student_feedback_path )\n",
    "\n",
    "                            # loop over notebook names to convert HTML feedback to PDF.\n",
    "                            for notebook_name in assignment_notebook_list:\n",
    "\n",
    "                                # create pandoc file command to convert from HTML to PDF.\n",
    "                                pandoc_command_list.append( \"pandoc\" )\n",
    "                                pandoc_command_list.append( student_feedback_path + \"/\" + notebook_name + \".html\" )\n",
    "                                pandoc_command_list.append( \"-o\" )\n",
    "\n",
    "                                # make and add pdf_file_path.\n",
    "                                pdf_file_path = student_feedback_path + \"/\" + notebook_name + \".pdf\"\n",
    "                                pandoc_command_list.append( pdf_file_path )\n",
    "\n",
    "                                print( \"----> Pandoc command: \" + str( pandoc_command_list ) )\n",
    "\n",
    "                                # run the command.\n",
    "                                pandoc_command_result = subprocess.call( pandoc_command_list )\n",
    "\n",
    "                                print( \"----> command result = \" + str( pandoc_command_result ) )\n",
    "\n",
    "                                # return code 0 = OK\n",
    "                                if ( pandoc_command_result == 0 ):\n",
    "\n",
    "                                    # add pdf_file_path to pdf_file_list.\n",
    "                                    pdf_file_name_list.append( notebook_name + \".pdf\" )\n",
    "                                \n",
    "                                else:\n",
    "                                    \n",
    "                                    # Not OK.  Error.\n",
    "                                    print( \"--------> ERROR code returned - File not added to list.\" )\n",
    "                                    \n",
    "                                #-- END check to see if pandoc executed OK. --#\n",
    "\n",
    "                            #-- END loop over notbook names. --#\n",
    "\n",
    "                            '''\n",
    "                            # == OLD - used to loop over files, slurp up all PDFs.\n",
    "                            # loop over contents\n",
    "                            for file_name in file_list:\n",
    "\n",
    "                                # get lower-cased file name\n",
    "                                file_name_lower = file_name.lower()\n",
    "\n",
    "                                # eventually will look for .html files, convert to PDF.\n",
    "                                # for now, just look for PDF files.\n",
    "\n",
    "                                # got any PDF files?\n",
    "                                if ( file_name_lower.rfind( \".pdf\" ) == ( len( file_name_lower ) - 4 ) ):\n",
    "\n",
    "                                    print( \"--------> Found PDF file: \" + file_name )\n",
    "\n",
    "                                    # add file path to list.\n",
    "                                    pdf_file_name_list.append( file_name )\n",
    "\n",
    "                                #-- END check to see if PDF. --#\n",
    "\n",
    "                            #-- END loop over files. --#\n",
    "                            '''\n",
    "\n",
    "                            # anything in list of PDF files?\n",
    "                            pdf_file_count = len( pdf_file_name_list )\n",
    "                            if ( pdf_file_count > 0 ):\n",
    "\n",
    "                                # there is at least one.  Create a MIME multipart message.\n",
    "                                multipart_message = MIMEMultipart()\n",
    "\n",
    "                                # set subject\n",
    "                                multipart_message[ 'Subject' ] = message_subject\n",
    "\n",
    "                                # set from\n",
    "                                multipart_message[ 'From' ] = message_from\n",
    "\n",
    "                                # set to\n",
    "                                multipart_message[ 'To' ] = student_email\n",
    "\n",
    "                                # set Bcc\n",
    "                                if ( ( message_bcc is not None ) and ( message_bcc != \"\" ) ):\n",
    "\n",
    "                                    multipart_message[ 'Bcc' ] = message_bcc\n",
    "                                    \n",
    "                                #-- END check to see if message_bcc. --#\n",
    "\n",
    "                                # preamble - will only appear in non-MIME-aware email reader.\n",
    "                                multipart_message.preamble = message_subject\n",
    "\n",
    "                                # add text body of message.\n",
    "                                complete_message_body = message_body\n",
    "\n",
    "                                # see if custom message for this student.\n",
    "                                if ( student_username in custom_message_dict ):\n",
    "\n",
    "                                    # yes.  Add it.\n",
    "                                    complete_message_body += \"\\n\" + custom_message_dict[ student_username ] + \"\\n\"\n",
    "\n",
    "                                #-- END check to see if custom message --#\n",
    "\n",
    "                                # add signature\n",
    "                                complete_message_body += message_signature\n",
    "\n",
    "                                # make text message and append it.\n",
    "                                text_message = MIMEText( complete_message_body )\n",
    "                                multipart_message.attach( text_message )\n",
    "\n",
    "                                # loop over PDF file path list.\n",
    "                                for file_name in pdf_file_name_list:\n",
    "\n",
    "                                    # make it into a full on file path:\n",
    "                                    pdf_file_path = student_feedback_path + \"/\" + file_name\n",
    "\n",
    "                                    print( \"current file path: \" + pdf_file_path )\n",
    "\n",
    "                                    # get mime type and encoding guesses.\n",
    "                                    content_type, encoding = mimetypes.guess_type( pdf_file_path )\n",
    "                                    if content_type is None or encoding is not None:\n",
    "\n",
    "                                        # No guess could be made, or the file is encoded, so\n",
    "                                        # use a generic bag-of-bits type.\n",
    "                                        content_type = 'application/octet-stream'\n",
    "\n",
    "                                    #-- END check to see if we have a content type, encoding --#\n",
    "\n",
    "                                    # get main and sub content types.\n",
    "                                    main_type, sub_type = content_type.split('/', 1)\n",
    "\n",
    "                                    print( \"current content_type: \" + content_type )\n",
    "\n",
    "                                    # open and add PDF file.\n",
    "                                    with open( pdf_file_path, 'rb' ) as fp:\n",
    "\n",
    "                                        pdf_message = MIMEApplication( fp.read(), _subtype = sub_type )\n",
    "\n",
    "                                    #-- END adding PDF to email. --#\n",
    "\n",
    "                                    # Set the filename's Content-Disposition HTTP header.\n",
    "                                    pdf_message.add_header( 'Content-Disposition', 'attachment', filename = file_name )\n",
    "                                    multipart_message.attach( pdf_message )\n",
    "\n",
    "                                #-- END loop over pdf_file_path_list --#\n",
    "\n",
    "                                # send message\n",
    "                                smtp_send_result = smtp_server.send_message( multipart_message )\n",
    "                                print( \"Send result (empty dict is Success!): \" + str( smtp_send_result ) )\n",
    "\n",
    "                            #-- END check to see if PDF file count --#\n",
    "\n",
    "                        except Exception as e:\n",
    "\n",
    "                            exception_type, exception_value, exception_traceback = sys.exc_info()\n",
    "                            print( \"Exception caught trying to read \\\"\" + student_feedback_path + \"\\\":\" )\n",
    "                            print( \"- args = \" + str( e.args ) )\n",
    "                            print( \"- type = \" + str( exception_type ) )\n",
    "                            print( \"- value = \" + str( exception_value ) )\n",
    "                            print( \"- traceback = \" + str( traceback.format_exc() ) )\n",
    "\n",
    "                        #-- END try-except for file IO for a given student. --#\n",
    "\n",
    "                    #-- END loop over results --#\n",
    "\n",
    "                #-- END with smtplib.SMTP( host = smtp_host, port = smtp_port ) as smtp_server: --#\n",
    "            \n",
    "            else:\n",
    "                \n",
    "                # assignment not OK.\n",
    "                print( \"Problem with assignment.  Can't distribute feedback.\" )\n",
    "\n",
    "            #-- END check to see if assignment is OK. --#\n",
    "                \n",
    "        except Exception as e:\n",
    "\n",
    "            exception_type, exception_value, exception_traceback = sys.exc_info()\n",
    "            print( \"Exception caught: \" )\n",
    "            print( \"- args = \" + str( e.args ) )\n",
    "            print( \"- type = \" + str( exception_type ) )\n",
    "            print( \"- value = \" + str( exception_value ) )\n",
    "            print( \"- traceback = \" + str( traceback.format_exc() ) )\n",
    "\n",
    "        finally:\n",
    "\n",
    "            print( \"In the 'finally:', cleaning up our mess.\" )\n",
    "\n",
    "            # close cursor\n",
    "            my_cursor.close()\n",
    "\n",
    "            # close connection\n",
    "            my_connection.close()\n",
    "\n",
    "        #-- END try-->except-->finally around database connection. --#\n",
    "\n",
    "    else:\n",
    "        \n",
    "        print( \"You must set the assignment name.\" )\n",
    "        \n",
    "    #-- END check to make sure we have an assignment name. --#\n",
    "    \n",
    "else:\n",
    "    \n",
    "    print( \"You must only run this from the nbgrader home directory ( \" + nbgrader_home + \" ).\" )\n",
    "\n",
    "#-- END check to make sure we are in nbgrader home. --#"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Troubleshooting\n",
    "\n",
    "- Back to the [Table of Contents](#Table-of-Contents)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `nbgrader formgrade` issues\n",
    "\n",
    "- Back to the [Table of Contents](#Table-of-Contents)\n",
    "\n",
    "### CONFIGPROXY_AUTH_TOKEN mismatch between jupyterhub and formgrade\n",
    "\n",
    "- Back to the [Table of Contents](#Table-of-Contents)\n",
    "\n",
    "Example output for CONFIGPROXY_AUTH_TOKEN mismatch between server jupyterhub and formgrade:\n",
    "\n",
    "    $ nbgrader formgrade\n",
    "    [FormgradeApp | INFO] Proxying /hub/nbgrader/2015-fall-big_data --> http://127.0.0.1:9000\n",
    "    Traceback (most recent call last):\n",
    "      File \"/usr/local/bin/nbgrader\", line 11, in <module>\n",
    "        sys.exit(main())\n",
    "      File \"/usr/local/lib/python3.4/dist-packages/nbgrader/apps/nbgraderapp.py\", line 232, in main\n",
    "        NbGraderApp.launch_instance()\n",
    "      File \"/usr/local/lib/python3.4/dist-packages/jupyter_core/application.py\", line 267, in launch_instance\n",
    "        return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)\n",
    "      File \"/usr/local/lib/python3.4/dist-packages/traitlets/config/application.py\", line 592, in launch_instance\n",
    "        app.start()\n",
    "      File \"/usr/local/lib/python3.4/dist-packages/nbgrader/apps/nbgraderapp.py\", line 225, in start\n",
    "        super(NbGraderApp, self).start()\n",
    "      File \"/usr/local/lib/python3.4/dist-packages/jupyter_core/application.py\", line 256, in start\n",
    "        self.subapp.start()\n",
    "      File \"/usr/local/lib/python3.4/dist-packages/nbgrader/apps/formgradeapp.py\", line 125, in start\n",
    "        parent=self)\n",
    "      File \"/usr/local/lib/python3.4/dist-packages/nbgrader/auth/hubauth.py\", line 102, in __init__\n",
    "        raise Exception('Error while trying to add JupyterHub route. {}: {}'.format(response.status_code, response.text))\n",
    "    Exception: Error while trying to add JupyterHub route. 403: \n",
    "\n",
    "If you see something like this, make sure that you are explicitly setting the CONFIGPROXY_AUTH_TOKEN for the jupyterhub server (preferably in the `c.JupyterHub.proxy_auth_token` variable in `jupyterhub_config.py`), and that you are exporting that same value before starting the formgrade app."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### \"attempt to write a readonly database\"\n",
    "\n",
    "- Back to the [Table of Contents](#Table-of-Contents)\n",
    "\n",
    "If you see an error like this:\n",
    "\n",
    "    sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) attempt to write a readonly database [SQL: 'INSERT INTO api_tokens (hashed, prefix, user_id) VALUES (?, ?, ?)'] [parameters: ('sha512:16384:09072f984a3396f7:7d01c1f9e2e9bef9abdf71342eac9feaa04757c5845e97dbbcc3ed34f2a49dd92bfc741ad7776cdba87a2fc33410b6bc6be1d0d983cb7c05e9debd45455554a0', 'ab39', 6)]\n",
    "    \n",
    "This means your jupyterhub.sqlite database is not able to be written to by the user used to start the formgrade application.  Try adjusting the permissions so that the user who starts formgrade can write to the database."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### \"unable to open database file\" api_tokens\n",
    "\n",
    "- Back to the [Table of Contents](#Table-of-Contents)\n",
    "\n",
    "If you see the following in the output from `nbgrader formgrade`:\n",
    "\n",
    "    sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) unable to open database file [SQL: 'INSERT INTO api_tokens (hashed, prefix, user_id) VALUES (?, ?, ?)'] [parameters: ('sha512:16384:19f732de1639d068:c517288d4f3716af666657325274c91812ad97e8252f5cf218db2c9f62c4c698a3762f62f27d8fe761f781740d2b525ce58a8a33d91308325761950524dd2aba', 'a151', 6)]\n",
    "    \n",
    "This is likely because the folder that contains your `jupyterhub.sqlite` database file is not writeable by the user used to start the formgrade application (sqlite writes temp files in the directory where the database it is using lives).  Try adjusting the permissions on the directory that contains `jupyterhub.sqlite` so that the user who starts formgrade can write to it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### URLs on formgrade pages refer to proxy IP address, not public IP or domain name\n",
    "\n",
    "- Back to the [Table of Contents](#Table-of-Contents)\n",
    "\n",
    "If you get past the above issues and can authenticate, but when the page loads, it is unstyled and all the links point to `localhost:8000` or `127.0.0.1:8000` rather than the actual public address of your server (including javascript and CSS, which won't load, so the page will look unstyled and won't work right), then you need to make sure that the `c.HubAuth.hub_address` property is set to the public IP address or domain name for your server in `nbgrader_config.py` for the instructor user.  Example:\n",
    "\n",
    "    c.HubAuth.hub_address = 'bigdataforsocialscience.com'\n",
    "\n",
    "If you have SSL configured, you'll need to set a different variable, you'll need to instead set the `c.HubAuth.hub_base_url` to the entire URL of the jupyterhub server, including protocol and port number.  This value takes priority over the `c.HubAuth.hub_address`, so no need to set that when this is set.  Example:\n",
    "\n",
    "    #c.HubAuth.hub_base_url = 'https://localhost:8000'\n",
    "    c.HubAuth.hub_base_url = 'https://bigdataforsocialscience.com:8000'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Notebook file not found when you try to grade a given submission\n",
    "\n",
    "- Back to the [Table of Contents](#Table-of-Contents)\n",
    "\n",
    "If when you try to grade a given submission you get a message that a given submission's notebook file could not be found, there is a chance that student in question (which is obfuscated so the grading is blind) might have re-named the notebook.\n",
    "\n",
    "If this is the case, to resolve, you need to:\n",
    "\n",
    "- look in `<nbgrader_home>/submitted`, in each student's folder, in their folder for the assignment in question, to see if you can find the misnamed notebook.\n",
    "- if you find it, fix the name.\n",
    "- in `<nbgrader_home>/autograded`, to get this change reflected in the formgrade application, you'll then need to remove the folder for that assignment from the student's folder, then re-autograde."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## notebook issues\n",
    "\n",
    "- Back to the [Table of Contents](#Table-of-Contents)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### nbgrader solution can not be in a normal cell\n",
    "\n",
    "- Back to the [Table of Contents](#Table-of-Contents)\n",
    "\n",
    "If, when you run `nbgrader assign`, you get the error:\n",
    "\n",
    "    RuntimeError: Solution region detected in a non-solution cell; please make sure all solution regions are within solution cells\n",
    "        \n",
    "It could be because you created your notebook with cells that contain nbgrader (for example, `### BEGIN SOLUTION ###` and `### END SOLUTION ###`), but aren't specified as nbgrader cells using the \"Create Assignment\" cell toolbar.  Make sure that all cells that have solutions are specified correctly as Assignment cells using the toolbar."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### grade cell point value must be greater than 0\n",
    "\n",
    "- Back to the [Table of Contents](#Table-of-Contents)\n",
    "\n",
    "If, when you run `nbgrader assign`, you get the error:\n",
    "\n",
    "    RuntimeError: Point value for grade cell correct_sum_of_squares is invalid:\n",
    "        \n",
    "Make sure that each of your graded Assignment cells has a point value that is greater than 0.  It can be a decimal like 0.5, but 0 is invalid."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### make sure you run code cells that just contain a function definition before calling the function\n",
    "\n",
    "- Back to the [Table of Contents](#Table-of-Contents)\n",
    "\n",
    "If you don't, and then you try to run the function, you'll get a really confusing and amazing explosion of exception messages and stack trace.  Be advised, on the watch for students that do this."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## gradebook database issues\n",
    "\n",
    "- Back to the [Table of Contents](#Table-of-Contents)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### corruption of tables that hold auto-grading\n",
    "\n",
    "- Back to the [Table of Contents](#Table-of-Contents)\n",
    "\n",
    "If your database is corrupted, including information on auto-graded assignments, here is code you can run to re-create the database structure needed to auto-grade.  It doesn't handle orhpaned records and you'll lose any grading recorded in the lost data, but it should at least make it so you can auto-grade over again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "from nbgrader.api import Gradebook, MissingEntry, Grade, SubmittedNotebook, Comment\n",
    "import sqlite3\n",
    "\n",
    "# declare variables\n",
    "student_id_list = []\n",
    "current_student_id = \"\"\n",
    "\n",
    "# make a connection - pass the name of the database file.\n",
    "my_connection = sqlite3.connect( 'gradebook.db' )\n",
    "\n",
    "# set row_factory that returns values mapped to column names\n",
    "#   as well as in an ordered list\n",
    "my_connection.row_factory = sqlite3.Row\n",
    "\n",
    "# then, make a cursor.\n",
    "my_cursor = my_connection.cursor()\n",
    "\n",
    "sql_string = \"SELECT * FROM student ORDER BY last_name ASC, first_name ASC;\"\n",
    "#sql_string = \"SELECT * FROM student WHERE id IN ( 'jmorgan' ) ORDER BY last_name ASC, first_name ASC;\"\n",
    "#sql_string = \"SELECT * FROM student WHERE id NOT IN ( 'cjones', 'aemad' ) ORDER BY last_name ASC, first_name ASC;\"\n",
    "\n",
    "# Actually use cursor here...\n",
    "result_set = my_cursor.execute( sql_string )\n",
    "\n",
    "# output results\n",
    "print( \"Student list:\" )\n",
    "\n",
    "# loop over students in result set.\n",
    "student_counter = 0\n",
    "for current_row in result_set:\n",
    "\n",
    "    # increment student counter\n",
    "    student_counter += 1\n",
    "\n",
    "    # initialize variables\n",
    "    current_student_id = \"\"\n",
    "    student_email = \"\"\n",
    "    student_first_name = \"\"\n",
    "    student_last_name = \"\"\n",
    "\n",
    "    print( \"\\n\\n==> Student \" + str( student_counter ) + \" = \" + current_row[ \"first_name\" ] + \" \" + current_row[ \"last_name\" ] + \" ( \" + current_row[ \"id\" ] + \" ) - \" + current_row[ \"email\" ] )\n",
    "\n",
    "    # get student information\n",
    "    current_student_id = current_row[ \"id\" ]\n",
    "    student_email = current_row[ \"email\" ]\n",
    "    student_first_name = current_row[ \"first_name\" ]\n",
    "    student_last_name = current_row[ \"last_name\" ]\n",
    "    \n",
    "    # add ID to student_id_list\n",
    "    student_id_list.append( current_student_id )\n",
    "\n",
    "#-- END loop over students --#\n",
    "    \n",
    "# close cursor and connection\n",
    "my_cursor.close()\n",
    "my_connection.close()\n",
    "\n",
    "# now, clean up assignments for each student.\n",
    "\n",
    "gb = Gradebook( \"sqlite:///gradebook.db\" )\n",
    "\n",
    "#assignment_name = '02. Database Basics'\n",
    "assignment_name = ''\n",
    "student_id = ''\n",
    "\n",
    "if ( ( assignment_name is not None ) and ( assignment_name != \"\" ) ):\n",
    "\n",
    "    for student_id in student_id_list:\n",
    "\n",
    "        # get submission\n",
    "        submission = gb.update_or_create_submission(assignment_name, student_id)\n",
    "\n",
    "        for notebook in submission.assignment.notebooks:\n",
    "            try:\n",
    "                # query the database for the submitted notebook...\n",
    "                nb = gb.find_submission_notebook(notebook.name, assignment_name, student_id)\n",
    "            except:\n",
    "                # if the submitted notebook doesn't exist, create it\n",
    "                nb = SubmittedNotebook(notebook=notebook, assignment=submission)\n",
    "\n",
    "            for grade_cell in notebook.grade_cells:\n",
    "                try:\n",
    "                    # query the database for the grade...\n",
    "                    gb.find_grade(grade_cell.name, notebook.name, assignment_name, student_id)\n",
    "                except MissingEntry:\n",
    "                    # if the grade doesn't exist, create it\n",
    "                    Grade(cell=grade_cell, notebook=nb)\n",
    "\n",
    "            for solution_cell in notebook.solution_cells:\n",
    "                try:\n",
    "                    # query the database for the comment...\n",
    "                    gb.find_comment(solution_cell.name, notebook.name, assignment_name, student_id)\n",
    "                except MissingEntry:\n",
    "                    # if the comment doesn't exist, create it\n",
    "                    Comment(cell=solution_cell, notebook=nb)\n",
    "\n",
    "        gb.db.add(submission)\n",
    "        gb.db.commit()\n",
    "\n",
    "    #-- END loop over students --#\n",
    "\n",
    "else:\n",
    "    \n",
    "    print( \"No assignment, no work done.\" )\n",
    "\n",
    "#-- END check to see if assignment. --#"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": false
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}