08_CMLReaders_Usage.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Accessing CML data\n",
    "\n",
    "The CML's database of intracranial and scalp EEG comes in a pandas dataframe format. All the pertinent data about each experimental session is recorded in a row of a large dataframe. These databases are organized around experimental protocols. For example, the <b>'r1'</b> database contains all DARPA RAM data, while the **'pyfr'** database includes all free-recall intracranial data collected in the years prior to RAM. \n",
    "\n",
    "Let's load the RAM database to get a better sense of these formats. We're going to use **CMLReaders**, which is a custom library with helper functions to load data for any experiments run by the CML. If you don't already have CMLReaders installed, please follow the instructions here: https://github.com/pennmem/cmlreaders"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 166,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "#First, our import statements. The CMLReader class is your gateway to all experimental data, including electrodes and EEG. The get_data_index function specifically loads experimental databases. \n",
    "from cmlreaders import CMLReader, get_data_index\n",
    "\n",
    "#The \"r1\" database corresponds to all of the RAM subjects. Use \"pyfr\" for the pre-RAM iEEG data.\n",
    "df = get_data_index(\"r1\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Recognition</th>\n",
       "      <th>all_events</th>\n",
       "      <th>contacts</th>\n",
       "      <th>experiment</th>\n",
       "      <th>import_type</th>\n",
       "      <th>localization</th>\n",
       "      <th>math_events</th>\n",
       "      <th>montage</th>\n",
       "      <th>original_experiment</th>\n",
       "      <th>original_session</th>\n",
       "      <th>pairs</th>\n",
       "      <th>ps4_events</th>\n",
       "      <th>session</th>\n",
       "      <th>subject</th>\n",
       "      <th>subject_alias</th>\n",
       "      <th>system_version</th>\n",
       "      <th>task_events</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>NaN</td>\n",
       "      <td>protocols/r1/subjects/R1001P/experiments/FR1/s...</td>\n",
       "      <td>protocols/r1/subjects/R1001P/localizations/0/m...</td>\n",
       "      <td>FR1</td>\n",
       "      <td>build</td>\n",
       "      <td>0</td>\n",
       "      <td>protocols/r1/subjects/R1001P/experiments/FR1/s...</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>protocols/r1/subjects/R1001P/localizations/0/m...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>R1001P</td>\n",
       "      <td>R1001P</td>\n",
       "      <td>NaN</td>\n",
       "      <td>protocols/r1/subjects/R1001P/experiments/FR1/s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>NaN</td>\n",
       "      <td>protocols/r1/subjects/R1001P/experiments/FR1/s...</td>\n",
       "      <td>protocols/r1/subjects/R1001P/localizations/0/m...</td>\n",
       "      <td>FR1</td>\n",
       "      <td>build</td>\n",
       "      <td>0</td>\n",
       "      <td>protocols/r1/subjects/R1001P/experiments/FR1/s...</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>protocols/r1/subjects/R1001P/localizations/0/m...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>R1001P</td>\n",
       "      <td>R1001P</td>\n",
       "      <td>NaN</td>\n",
       "      <td>protocols/r1/subjects/R1001P/experiments/FR1/s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>NaN</td>\n",
       "      <td>protocols/r1/subjects/R1001P/experiments/FR2/s...</td>\n",
       "      <td>protocols/r1/subjects/R1001P/localizations/0/m...</td>\n",
       "      <td>FR2</td>\n",
       "      <td>build</td>\n",
       "      <td>0</td>\n",
       "      <td>protocols/r1/subjects/R1001P/experiments/FR2/s...</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>protocols/r1/subjects/R1001P/localizations/0/m...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>R1001P</td>\n",
       "      <td>R1001P</td>\n",
       "      <td>NaN</td>\n",
       "      <td>protocols/r1/subjects/R1001P/experiments/FR2/s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>NaN</td>\n",
       "      <td>protocols/r1/subjects/R1001P/experiments/FR2/s...</td>\n",
       "      <td>protocols/r1/subjects/R1001P/localizations/0/m...</td>\n",
       "      <td>FR2</td>\n",
       "      <td>build</td>\n",
       "      <td>0</td>\n",
       "      <td>protocols/r1/subjects/R1001P/experiments/FR2/s...</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>protocols/r1/subjects/R1001P/localizations/0/m...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>R1001P</td>\n",
       "      <td>R1001P</td>\n",
       "      <td>NaN</td>\n",
       "      <td>protocols/r1/subjects/R1001P/experiments/FR2/s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>NaN</td>\n",
       "      <td>protocols/r1/subjects/R1001P/experiments/PAL1/...</td>\n",
       "      <td>protocols/r1/subjects/R1001P/localizations/0/m...</td>\n",
       "      <td>PAL1</td>\n",
       "      <td>build</td>\n",
       "      <td>0</td>\n",
       "      <td>protocols/r1/subjects/R1001P/experiments/PAL1/...</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>protocols/r1/subjects/R1001P/localizations/0/m...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>R1001P</td>\n",
       "      <td>R1001P</td>\n",
       "      <td>NaN</td>\n",
       "      <td>protocols/r1/subjects/R1001P/experiments/PAL1/...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>NaN</td>\n",
       "      <td>protocols/r1/subjects/R1001P/experiments/PAL1/...</td>\n",
       "      <td>protocols/r1/subjects/R1001P/localizations/0/m...</td>\n",
       "      <td>PAL1</td>\n",
       "      <td>build</td>\n",
       "      <td>0</td>\n",
       "      <td>protocols/r1/subjects/R1001P/experiments/PAL1/...</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>protocols/r1/subjects/R1001P/localizations/0/m...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>R1001P</td>\n",
       "      <td>R1001P</td>\n",
       "      <td>NaN</td>\n",
       "      <td>protocols/r1/subjects/R1001P/experiments/PAL1/...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>protocols/r1/subjects/R1001P/localizations/0/m...</td>\n",
       "      <td>YC1</td>\n",
       "      <td>conversion</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>protocols/r1/subjects/R1001P/localizations/0/m...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>R1001P</td>\n",
       "      <td>R1001P</td>\n",
       "      <td>NaN</td>\n",
       "      <td>protocols/r1/subjects/R1001P/experiments/YC1/s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>protocols/r1/subjects/R1001P/localizations/0/m...</td>\n",
       "      <td>YC1</td>\n",
       "      <td>conversion</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>protocols/r1/subjects/R1001P/localizations/0/m...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>R1001P</td>\n",
       "      <td>R1001P</td>\n",
       "      <td>NaN</td>\n",
       "      <td>protocols/r1/subjects/R1001P/experiments/YC1/s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>protocols/r1/subjects/R1001P/localizations/0/m...</td>\n",
       "      <td>YC1</td>\n",
       "      <td>conversion</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2</td>\n",
       "      <td>protocols/r1/subjects/R1001P/localizations/0/m...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2</td>\n",
       "      <td>R1001P</td>\n",
       "      <td>R1001P</td>\n",
       "      <td>NaN</td>\n",
       "      <td>protocols/r1/subjects/R1001P/experiments/YC1/s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>NaN</td>\n",
       "      <td>protocols/r1/subjects/R1002P/experiments/FR1/s...</td>\n",
       "      <td>protocols/r1/subjects/R1002P/localizations/0/m...</td>\n",
       "      <td>FR1</td>\n",
       "      <td>build</td>\n",
       "      <td>0</td>\n",
       "      <td>protocols/r1/subjects/R1002P/experiments/FR1/s...</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>protocols/r1/subjects/R1002P/localizations/0/m...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>R1002P</td>\n",
       "      <td>R1002P</td>\n",
       "      <td>NaN</td>\n",
       "      <td>protocols/r1/subjects/R1002P/experiments/FR1/s...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Recognition                                         all_events  \\\n",
       "0         NaN  protocols/r1/subjects/R1001P/experiments/FR1/s...   \n",
       "1         NaN  protocols/r1/subjects/R1001P/experiments/FR1/s...   \n",
       "2         NaN  protocols/r1/subjects/R1001P/experiments/FR2/s...   \n",
       "3         NaN  protocols/r1/subjects/R1001P/experiments/FR2/s...   \n",
       "4         NaN  protocols/r1/subjects/R1001P/experiments/PAL1/...   \n",
       "5         NaN  protocols/r1/subjects/R1001P/experiments/PAL1/...   \n",
       "6         NaN                                                NaN   \n",
       "7         NaN                                                NaN   \n",
       "8         NaN                                                NaN   \n",
       "9         NaN  protocols/r1/subjects/R1002P/experiments/FR1/s...   \n",
       "\n",
       "                                            contacts experiment import_type  \\\n",
       "0  protocols/r1/subjects/R1001P/localizations/0/m...        FR1       build   \n",
       "1  protocols/r1/subjects/R1001P/localizations/0/m...        FR1       build   \n",
       "2  protocols/r1/subjects/R1001P/localizations/0/m...        FR2       build   \n",
       "3  protocols/r1/subjects/R1001P/localizations/0/m...        FR2       build   \n",
       "4  protocols/r1/subjects/R1001P/localizations/0/m...       PAL1       build   \n",
       "5  protocols/r1/subjects/R1001P/localizations/0/m...       PAL1       build   \n",
       "6  protocols/r1/subjects/R1001P/localizations/0/m...        YC1  conversion   \n",
       "7  protocols/r1/subjects/R1001P/localizations/0/m...        YC1  conversion   \n",
       "8  protocols/r1/subjects/R1001P/localizations/0/m...        YC1  conversion   \n",
       "9  protocols/r1/subjects/R1002P/localizations/0/m...        FR1       build   \n",
       "\n",
       "   localization                                        math_events  montage  \\\n",
       "0             0  protocols/r1/subjects/R1001P/experiments/FR1/s...        0   \n",
       "1             0  protocols/r1/subjects/R1001P/experiments/FR1/s...        0   \n",
       "2             0  protocols/r1/subjects/R1001P/experiments/FR2/s...        0   \n",
       "3             0  protocols/r1/subjects/R1001P/experiments/FR2/s...        0   \n",
       "4             0  protocols/r1/subjects/R1001P/experiments/PAL1/...        0   \n",
       "5             0  protocols/r1/subjects/R1001P/experiments/PAL1/...        0   \n",
       "6             0                                                NaN        0   \n",
       "7             0                                                NaN        0   \n",
       "8             0                                                NaN        0   \n",
       "9             0  protocols/r1/subjects/R1002P/experiments/FR1/s...        0   \n",
       "\n",
       "  original_experiment original_session  \\\n",
       "0                 NaN                0   \n",
       "1                 NaN                1   \n",
       "2                 NaN                0   \n",
       "3                 NaN                1   \n",
       "4                 NaN                0   \n",
       "5                 NaN                1   \n",
       "6                 NaN                0   \n",
       "7                 NaN                1   \n",
       "8                 NaN                2   \n",
       "9                 NaN                0   \n",
       "\n",
       "                                               pairs ps4_events  session  \\\n",
       "0  protocols/r1/subjects/R1001P/localizations/0/m...        NaN        0   \n",
       "1  protocols/r1/subjects/R1001P/localizations/0/m...        NaN        1   \n",
       "2  protocols/r1/subjects/R1001P/localizations/0/m...        NaN        0   \n",
       "3  protocols/r1/subjects/R1001P/localizations/0/m...        NaN        1   \n",
       "4  protocols/r1/subjects/R1001P/localizations/0/m...        NaN        0   \n",
       "5  protocols/r1/subjects/R1001P/localizations/0/m...        NaN        1   \n",
       "6  protocols/r1/subjects/R1001P/localizations/0/m...        NaN        0   \n",
       "7  protocols/r1/subjects/R1001P/localizations/0/m...        NaN        1   \n",
       "8  protocols/r1/subjects/R1001P/localizations/0/m...        NaN        2   \n",
       "9  protocols/r1/subjects/R1002P/localizations/0/m...        NaN        0   \n",
       "\n",
       "  subject subject_alias  system_version  \\\n",
       "0  R1001P        R1001P             NaN   \n",
       "1  R1001P        R1001P             NaN   \n",
       "2  R1001P        R1001P             NaN   \n",
       "3  R1001P        R1001P             NaN   \n",
       "4  R1001P        R1001P             NaN   \n",
       "5  R1001P        R1001P             NaN   \n",
       "6  R1001P        R1001P             NaN   \n",
       "7  R1001P        R1001P             NaN   \n",
       "8  R1001P        R1001P             NaN   \n",
       "9  R1002P        R1002P             NaN   \n",
       "\n",
       "                                         task_events  \n",
       "0  protocols/r1/subjects/R1001P/experiments/FR1/s...  \n",
       "1  protocols/r1/subjects/R1001P/experiments/FR1/s...  \n",
       "2  protocols/r1/subjects/R1001P/experiments/FR2/s...  \n",
       "3  protocols/r1/subjects/R1001P/experiments/FR2/s...  \n",
       "4  protocols/r1/subjects/R1001P/experiments/PAL1/...  \n",
       "5  protocols/r1/subjects/R1001P/experiments/PAL1/...  \n",
       "6  protocols/r1/subjects/R1001P/experiments/YC1/s...  \n",
       "7  protocols/r1/subjects/R1001P/experiments/YC1/s...  \n",
       "8  protocols/r1/subjects/R1001P/experiments/YC1/s...  \n",
       "9  protocols/r1/subjects/R1002P/experiments/FR1/s...  "
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#This dataframe contains all the information about every experimental sessions collected in the RAM project\n",
    "df[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load data from an example subject\n",
    "Here, let's go through an example of loading experimental events and EEG from one subject"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "#First, our import statements\n",
    "from cmlreaders import CMLReader, get_data_index\n",
    "\n",
    "#The \"r1\" database corresponds to all of the RAM subjects\n",
    "df = get_data_index(\"r1\")\n",
    "\n",
    "#Specify which subject and experiment we want\n",
    "sub = 'R1001P'\n",
    "exp = 'FR1'\n",
    "\n",
    "#Find out the sessions, localization, and montage for this subject\n",
    "sessions = list(df[(df['subject']==sub) & (df['experiment']==exp)]['session'])\n",
    "mont = int(df[(df['subject']==sub) & (df['experiment']==exp)].iloc()[0]['montage'])      #note that *usually* mont and loc will be 0.\n",
    "loc = int(df[(df['subject']==sub) & (df['experiment']==exp)].iloc()[0]['localization'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "R1001P sessions: [0, 1]\n",
      "R1001P montage: 0\n",
      "R1001P localization: 0\n"
     ]
    }
   ],
   "source": [
    "print(f'{sub} sessions: {sessions}')\n",
    "print(f'{sub} montage: {mont}')\n",
    "print(f'{sub} localization: {loc}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<i>Usually, montage and localization are both zero, meaning a subject had only one surgery and the subset of recorded electrodes did not change. But not always!</i>\n",
    "\n",
    "<b>Montage:</b> Refers to set of a subject's electrodes that were recorded in a given experimental session.\n",
    "\n",
    "<b>Localization:</b> A subject will get a new localization if they were reimplanted after another surgery. Therefore, electrodes may be in different places altogether. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This subject completed two sessions of FR1, and only had one montage/localization. Let's load data from the first session. First, we'll need to instantiate an instance of the 'CMLReader' object, which is the object class for accessing any CML data. Think of it as a \"finder\" for any kind of experimental data. At a minimum, you'll need to give it a subject and experiment for it to find anything.\n",
    "\n",
    "Once you have constructed a reader, the data loading shows the same behavior as the CMLLoad reader used elsewhere in this workshop. For differences between available arguments, please use the '?' operator on loader functions (such as reader.load_eeg?) to find details."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['voxel_coordinates', 'jacksheet', 'classifier_excluded_leads', 'good_leads', 'leads', 'area', 'electrode_coordinates', 'prior_stim_results', 'target_selection_table', 'experiment_log', 'all_events', 'events', 'math_events', 'ps4_events', 'task_events', 'used_classifier', 'baseline_classifier', 'sources', 'eeg', 'matlab_contacts', 'matlab_pairs', 'pairs', 'contacts', 'localization', 'electrode_categories', 'classifier_summary', 'session_summary', 'math_summary'])"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#For first session...\n",
    "reader = CMLReader(sub, exp, sessions[0], montage=mont, localization=loc)   #reader for loading CML data\n",
    "\n",
    "#What kind of data can the reader get for us?\n",
    "reader.reader_names.keys()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}