Aligning Data across Modalities

With many forms of data, it can be highly useful to align multiple recording modalities together to identify how they relate to one-another or to interpret brain response activity. Modalities in most of files consist of behavioral data, like running speed and eye tracking recordings, as well as recordings from the brain. In this example, we use an Ophys file with the 2-Photon dF/F recording. This notebook shows these modalities, as well as the types of stimulus presented, aligned in time to provide insight.

Environment Setup

⚠️Note: If running on a new environment, run this cell once and then restart the kernel⚠️

try:
    from databook_utils.dandi_utils import dandi_download_open
except:
    !git clone https://github.com/AllenInstitute/openscope_databook.git
    %cd openscope_databook
    %pip install -e .

c:\Users\carter.peene\AppData\Local\Programs\Python\Python39\lib\site-packages\numpy\_distributor_init.py:30: UserWarning: loaded more than 1 DLL from .libs:
c:\Users\carter.peene\AppData\Local\Programs\Python\Python39\lib\site-packages\numpy\.libs\libopenblas.EL2C6PLE4ZYW3ECEVIV3OXXGRN2NRFM2.gfortran-win_amd64.dll
c:\Users\carter.peene\AppData\Local\Programs\Python\Python39\lib\site-packages\numpy\.libs\libopenblas.XWYDX2IKJW2NMTWSFYNGFUWKQU3LYTCZ.gfortran-win_amd64.dll
  warnings.warn("loaded more than 1 DLL from .libs:"

import os

import matplotlib as mpl
import matplotlib.pyplot as plt
import numpy as np

from scipy import interpolate

%matplotlib inline

Downloading NWB File

Change the values below to download the file you’re interested in. In this example, we the Units table of an Ecephys file from The Allen Institute’s Dendritic Coupling dataset, which is currently embargoed, so you’ll have to choose one with the same kind of data. Set dandiset_id and dandi_filepath to correspond to the dandiset id and filepath of the file you want. If you’re accessing an embargoed dataset, set dandi_api_key to your DANDI API key.

dandiset_id = "000336"
dandi_filepath = "sub-655042/sub-655042_ses-1241145430-acq-1241919401_ophys.nwb"
download_loc = "."
dandi_api_key = os.environ["DANDI_API_KEY"]

# This can sometimes take a while depending on the size of the file
io = dandi_download_open(dandiset_id, dandi_filepath, download_loc, dandi_api_key=dandi_api_key)
nwb = io.read()

A newer version (0.56.0) of dandi/dandi-cli is available. You are using 0.55.1

dir before download
c:\Users\carter.peene\Desktop\Projects\openscope_databook\docs\embargoed
['cell_matching.ipynb', 'modality_alignment.ipynb', 'sub-655042_ses-1241145430-acq-1241919401_ophys.nwb', 'test_2p_responses_embargoed.ipynb', 'visualize_behavior.ipynb']
File already exists
dir after download
c:\Users\carter.peene\Desktop\Projects\openscope_databook\docs\embargoed
['cell_matching.ipynb', 'modality_alignment.ipynb', 'sub-655042_ses-1241145430-acq-1241919401_ophys.nwb', 'test_2p_responses_embargoed.ipynb', 'visualize_behavior.ipynb']
Opening file

c:\Users\carter.peene\AppData\Local\Programs\Python\Python39\lib\site-packages\hdmf\spec\namespace.py:531: UserWarning: Ignoring cached namespace 'hdmf-common' version 1.5.0 because version 1.5.1 is already loaded.
  warn("Ignoring cached namespace '%s' version %s because version %s is already loaded."
c:\Users\carter.peene\AppData\Local\Programs\Python\Python39\lib\site-packages\hdmf\spec\namespace.py:531: UserWarning: Ignoring cached namespace 'core' version 2.3.0 because version 2.5.0 is already loaded.
  warn("Ignoring cached namespace '%s' version %s because version %s is already loaded."
c:\Users\carter.peene\AppData\Local\Programs\Python\Python39\lib\site-packages\hdmf\spec\namespace.py:531: UserWarning: Ignoring cached namespace 'hdmf-experimental' version 0.1.0 because version 0.2.0 is already loaded.
  warn("Ignoring cached namespace '%s' version %s because version %s is already loaded."

# shows properties of this NWB file's imaging plane
nwb.lab_meta_data

{'metadata': metadata abc.OphysMetadata at 0x2427754511952
 Fields:
   experiment_container_id: 0
   field_of_view_height: 512
   field_of_view_width: 512
   imaging_depth: 32
   imaging_plane_group: 0
   imaging_plane_group_count: 4
   ophys_experiment_id: 1241919401
   ophys_session_id: 1241145430}

Getting Running Speed, Eye Tracking, and Fluorescence Data

Since this notebook is meant for Ophys data, it will include the fluorescence recordings from ROIs. But it will also include the behavioral measurements. The important arrays to extract here are running_speed_trace, eye_tracking_area, blink_times, and dff_trace. Their associated timestamps are also needed so these arrays can each be interpolated to the same aligned timescale.

running_speed = nwb.processing["running"]["speed"]
running_speed_trace = running_speed.data
running_speed_timestamps = running_speed.timestamps

eye_tracking = nwb.acquisition["EyeTracking"].eye_tracking
# eye_tracking = nwb.acquisition["EyeTracking"].corneal_reflection_tracking
# eye_tracking = nwb.acquisition["EyeTracking"].pupil_tracking
eye_tracking_area = eye_tracking.area
eye_tracking_timestamps = eye_tracking.timestamps

blink_times = nwb.acquisition["EyeTracking"].likely_blink

dff = nwb.processing["ophys"]["dff"].roi_response_series["traces"]
dff_trace = dff.data
dff_timestamps = dff.timestamps

Getting Stimulus Epochs

In addition to the data mentioned above, the stimulus presentations can also be shown alongside the time-aligned modalities. This could help to identify what responses are evoked by what stimuli.

Here, epochs are extracted from all the stimulus tables in the Intervals section. In this case, an epoch is a continuous period of time during a session where a particular type of stimulus is shown. The output here is a list of epochs, where an epoch is a tuple of four values; the stimulus name, the stimulus block, the starting time and the ending time. Since stimulus information can vary significantly between experiments and NWB files, you may need to tailor the code below to extract epochs for the file you’re interested in. For more information on the stimulus tables, see Visualizing 2P Responses to Stimulus.

### extract epoch times from stim table where stimulus rows have a different 'block' than following row
### returns list of epochs, where an epoch is of the form (stimulus name, stimulus block, start time, stop time)
def extract_epochs(stim_name, stim_table, epochs):
    
    # specify a current epoch stop and start time
    epoch_start = stim_table.start_time[0]
    epoch_stop = stim_table.stop_time[0]

    # for each row, try to extend current epoch stop_time
    for i in range(len(stim_table)):
        this_block = stim_table.stimulus_block[i]
        # if end of table, end the current epoch
        if i+1 >= len(stim_table):
            epochs.append((stim_name, this_block, epoch_start, epoch_stop))
            break
            
        next_block = stim_table.stimulus_block[i+1]
        # if next row is the same stim block, push back epoch_stop time
        if next_block == this_block:
            epoch_stop = stim_table.stop_time[i+1]
        # otherwise, end the current epoch, start new epoch
        else:
            epochs.append((stim_name, this_block, epoch_start, epoch_stop))
            epoch_start = stim_table.start_time[i+1]
            epoch_stop = stim_table.stop_time[i+1]
    
    return epochs

# extract epochs from all valid stimulus tables
epochs = []
for stim_name in nwb.intervals.keys():
    stim_table = nwb.intervals[stim_name]
    try:
        epochs = extract_epochs(stim_name, stim_table, epochs)
    except:
        continue

# epochs take the form (stimulus name, stimulus block, start time, stop time)
print(len(epochs))
# sort epochs based on start time
epochs.sort(key=lambda x: x[2])
for epoch in epochs:
    print(epoch)

185
('movie_touch_of_evil_fwd_presentations', 0.0, 30.024899999999995, 39.03239859745236)
('movie_flower_fwd_presentations', 1.0, 40.03321, 49.04067859745236)
('movie_flower_fwd_presentations', 2.0, 50.04149000000001, 59.04900859745236)
('movie_touch_of_evil_fwd_presentations', 3.0, 60.049800000000005, 69.05729859745236)
('movie_worms_fwd_presentations', 4.0, 70.05809, 79.06561859745236)
('movie_touch_of_evil_fwd_presentations', 5.0, 80.06641, 89.07392859745237)
('movie_flower_fwd_presentations', 6.0, 90.0747, 99.08226859745236)
('movie_worms_fwd_presentations', 7.0, 100.08303, 109.09053859745237)
('movie_flower_fwd_presentations', 8.0, 110.09136, 119.09887859745236)
('movie_worms_fwd_presentations', 9.0, 120.09967000000002, 129.10717859745236)
('movie_worms_fwd_presentations', 10.0, 130.10798000000003, 139.11550859745236)
('movie_touch_of_evil_fwd_presentations', 11.0, 140.11628000000002, 149.12377859745237)
('movie_touch_of_evil_fwd_presentations', 12.0, 150.12462000000002, 159.13211859745238)
('movie_flower_fwd_presentations', 13.0, 160.13288000000003, 169.14041859745237)
('movie_touch_of_evil_fwd_presentations', 14.0, 170.14120000000003, 179.14870859745238)
('movie_flower_fwd_presentations', 15.0, 180.14951, 189.1570285974524)
('movie_worms_fwd_presentations', 16.0, 190.15781, 199.1653085974524)
('movie_touch_of_evil_fwd_presentations', 17.0, 200.16611000000003, 209.1735785974524)
('movie_flower_fwd_presentations', 18.0, 210.17439, 219.1818985974524)
('movie_flower_fwd_presentations', 19.0, 220.1827, 229.1901985974524)
('movie_worms_fwd_presentations', 20.0, 230.19098, 239.1984585974524)
('movie_worms_fwd_presentations', 21.0, 240.19927, 249.2067685974524)
('movie_touch_of_evil_fwd_presentations', 22.0, 250.20754, 259.2150385974524)
('movie_worms_fwd_presentations', 23.0, 260.21586, 269.2233285974524)
('movie_touch_of_evil_fwd_presentations', 24.0, 270.22414000000003, 279.2316085974524)
('movie_flower_fwd_presentations', 25.0, 280.23242, 289.2398985974524)
('movie_flower_fwd_presentations', 26.0, 290.24071000000004, 299.2481685974524)
('movie_flower_fwd_presentations', 27.0, 300.249, 309.2564785974524)
('movie_worms_fwd_presentations', 28.0, 310.25728000000004, 319.2647585974524)
('movie_worms_fwd_presentations', 29.0, 320.26557, 329.2730285974524)
('movie_flower_fwd_presentations', 30.0, 330.27385000000004, 339.2813285974524)
('movie_touch_of_evil_fwd_presentations', 31.0, 340.28213, 349.2896085974524)
('movie_touch_of_evil_fwd_presentations', 32.0, 350.29044, 359.2978885974524)
('movie_touch_of_evil_fwd_presentations', 33.0, 360.29871, 369.3061785974524)
('movie_worms_fwd_presentations', 34.0, 370.30701, 379.3144685974524)
('movie_worms_fwd_presentations', 35.0, 380.31528, 389.3227685974524)
('movie_worms_fwd_presentations', 36.0, 390.32357, 399.3310485974524)
('movie_worms_fwd_presentations', 37.0, 400.33186, 409.3393485974524)
('movie_flower_fwd_presentations', 38.0, 410.34016, 419.3476185974524)
('movie_touch_of_evil_fwd_presentations', 39.0, 420.34844, 429.3559185974524)
('movie_touch_of_evil_fwd_presentations', 40.0, 430.35673, 439.3642085974524)
('movie_flower_fwd_presentations', 41.0, 440.365, 449.3724785974524)
('movie_touch_of_evil_fwd_presentations', 42.0, 450.37332, 459.3807785974524)
('movie_worms_fwd_presentations', 43.0, 460.38158, 469.3890585974524)
('movie_flower_fwd_presentations', 44.0, 470.38988, 479.3973385974524)
('movie_worms_fwd_presentations', 45.0, 480.39815, 489.4056185974524)
('movie_touch_of_evil_fwd_presentations', 46.0, 490.4064800000001, 499.4139185974524)
('movie_flower_fwd_presentations', 47.0, 500.41474000000005, 509.4221885974524)
('movie_touch_of_evil_fwd_presentations', 48.0, 510.4230400000001, 519.4305285974524)
('movie_touch_of_evil_fwd_presentations', 49.0, 520.43131, 529.4387885974525)
('movie_flower_fwd_presentations', 50.0, 530.4395800000001, 539.4470485974524)
('movie_worms_fwd_presentations', 51.0, 540.4479100000001, 549.4553785974524)
('movie_flower_fwd_presentations', 52.0, 550.45617, 559.4636385974524)
('movie_flower_fwd_presentations', 53.0, 560.4644700000001, 569.4719185974524)
('movie_touch_of_evil_fwd_presentations', 54.0, 570.4727800000001, 579.4802385974524)
('movie_worms_fwd_presentations', 55.0, 580.4810500000001, 589.4885085974524)
('movie_worms_fwd_presentations', 56.0, 590.4893300000001, 599.4967985974524)
('movie_touch_of_evil_fwd_presentations', 57.0, 600.4976100000001, 609.5050685974524)
('movie_worms_fwd_presentations', 58.0, 610.5059100000001, 619.5133685974524)
('movie_flower_fwd_presentations', 59.0, 620.5142000000001, 629.5216585974524)
('movie_touch_of_evil_fwd_presentations', 60.0, 630.5225, 639.5299485974524)
('movie_worms_fwd_presentations', 61.0, 640.5307700000001, 649.5382385974524)
('movie_flower_fwd_presentations', 62.0, 650.5390600000001, 659.5465285974524)
('movie_worms_fwd_presentations', 63.0, 660.54735, 669.5548085974524)
('movie_touch_of_evil_fwd_presentations', 64.0, 670.55564, 679.5630985974524)
('movie_flower_fwd_presentations', 65.0, 680.5639400000001, 689.5714085974524)
('movie_touch_of_evil_fwd_presentations', 66.0, 690.5722300000001, 699.5796685974524)
('movie_touch_of_evil_fwd_presentations', 67.0, 700.58049, 709.5879685974523)
('movie_flower_fwd_presentations', 68.0, 710.58878, 719.5962385974524)
('movie_worms_fwd_presentations', 69.0, 720.5970800000001, 729.6045285974524)
('movie_flower_fwd_presentations', 70.0, 730.60535, 739.6128485974524)
('movie_worms_fwd_presentations', 71.0, 740.6136600000001, 749.6211285974524)
('movie_touch_of_evil_fwd_presentations', 72.0, 750.62193, 759.6293985974525)
('movie_worms_fwd_presentations', 73.0, 760.63021, 769.6376885974524)
('movie_flower_fwd_presentations', 74.0, 770.6385, 779.6459885974524)
('movie_worms_fwd_presentations', 75.0, 780.6468100000001, 789.6542585974524)
('movie_touch_of_evil_fwd_presentations', 76.0, 790.6550900000001, 799.6625685974524)
('movie_flower_fwd_presentations', 77.0, 800.6633700000001, 809.6708285974524)
('movie_flower_fwd_presentations', 78.0, 810.6716900000001, 819.6791485974524)
('movie_worms_fwd_presentations', 79.0, 820.67996, 829.6874085974524)
('movie_flower_fwd_presentations', 80.0, 830.6882600000001, 839.6956985974524)
('movie_worms_fwd_presentations', 81.0, 840.69653, 849.7039985974524)
('movie_touch_of_evil_fwd_presentations', 82.0, 850.7048100000001, 859.7122885974524)
('movie_touch_of_evil_fwd_presentations', 83.0, 860.7130900000001, 869.7205785974523)
('movie_touch_of_evil_fwd_presentations', 84.0, 870.7214000000001, 879.7288585974524)
('movie_touch_of_evil_fwd_presentations', 85.0, 880.72968, 889.7371285974524)
('movie_flower_fwd_presentations', 86.0, 890.73797, 899.7454385974524)
('movie_touch_of_evil_fwd_presentations', 87.0, 900.74627, 909.7537385974524)
('movie_flower_fwd_presentations', 88.0, 910.75455, 919.7620285974524)
('movie_flower_fwd_presentations', 89.0, 920.76283, 929.7702885974523)
('movie_worms_fwd_presentations', 90.0, 930.77114, 939.7786085974524)
('movie_worms_fwd_presentations', 91.0, 940.7794, 949.7868785974524)
('movie_touch_of_evil_fwd_presentations', 92.0, 950.78771, 959.7951785974524)
('movie_flower_fwd_presentations', 93.0, 960.79601, 969.8034485974524)
('movie_worms_fwd_presentations', 94.0, 970.80431, 979.8117485974524)
('movie_worms_fwd_presentations', 95.0, 980.81257, 989.8200385974524)
('movie_flower_fwd_presentations', 96.0, 990.82085, 999.8282985974524)
('movie_flower_fwd_presentations', 97.0, 1000.82916, 1009.8366085974524)
('movie_touch_of_evil_fwd_presentations', 98.0, 1010.83744, 1019.8448985974524)
('movie_worms_fwd_presentations', 99.0, 1020.84571, 1029.8531785974526)
('movie_worms_fwd_presentations', 100.0, 1030.85403, 1039.8614785974526)
('movie_touch_of_evil_fwd_presentations', 101.0, 1040.8623, 1049.8697585974526)
('movie_touch_of_evil_fwd_presentations', 102.0, 1050.87059, 1059.8780485974526)
('movie_worms_fwd_presentations', 103.0, 1060.87888, 1069.8863585974527)
('movie_flower_fwd_presentations', 104.0, 1070.88716, 1079.8946285974523)
('movie_worms_fwd_presentations', 105.0, 1080.89544, 1089.9029085974526)
('movie_touch_of_evil_fwd_presentations', 106.0, 1090.90374, 1099.9111985974523)
('movie_flower_fwd_presentations', 107.0, 1100.9120400000002, 1109.9194985974527)
('movie_touch_of_evil_fwd_presentations', 108.0, 1110.9203200000002, 1119.927808597453)
('movie_flower_fwd_presentations', 109.0, 1120.9286100000002, 1129.9360785974527)
('movie_worms_fwd_presentations', 110.0, 1130.9369000000002, 1139.9443585974527)
('movie_flower_fwd_presentations', 111.0, 1140.94519, 1149.952658597453)
('movie_flower_fwd_presentations', 112.0, 1150.9534700000002, 1159.9609285974527)
('movie_touch_of_evil_fwd_presentations', 113.0, 1160.96179, 1169.9692385974529)
('movie_touch_of_evil_fwd_presentations', 114.0, 1170.97006, 1179.9774885974523)
('movie_worms_fwd_presentations', 115.0, 1180.97836, 1189.9858185974526)
('movie_touch_of_evil_fwd_presentations', 116.0, 1190.98666, 1199.9940985974526)
('movie_worms_fwd_presentations', 117.0, 1200.99491, 1210.0024085974526)
('movie_worms_fwd_presentations', 118.0, 1211.00321, 1220.0106985974526)
('movie_flower_fwd_presentations', 119.0, 1221.01148, 1230.018948597453)
('movie_touch_of_evil_fwd_presentations', 120.0, 1231.0198, 1240.0272485974526)
('movie_worms_fwd_presentations', 121.0, 1241.02809, 1250.0355285974526)
('movie_touch_of_evil_fwd_presentations', 122.0, 1251.03637, 1260.0438385974526)
('movie_flower_fwd_presentations', 123.0, 1261.04465, 1270.0520985974526)
('movie_flower_fwd_presentations', 124.0, 1271.0529700000002, 1280.0604085974526)
('movie_flower_fwd_presentations', 125.0, 1281.06123, 1290.0686785974526)
('movie_touch_of_evil_fwd_presentations', 126.0, 1291.06952, 1300.0769785974526)
('movie_worms_fwd_presentations', 127.0, 1301.07782, 1310.0852785974523)
('movie_worms_fwd_presentations', 128.0, 1311.0861, 1320.0935385974526)
('movie_worms_fwd_presentations', 129.0, 1321.0944000000002, 1330.1018285974526)
('movie_touch_of_evil_fwd_presentations', 130.0, 1331.10265, 1340.1101185974526)
('movie_flower_fwd_presentations', 131.0, 1341.11099, 1350.1184085974526)
('movie_worms_fwd_presentations', 132.0, 1351.11927, 1360.1266985974526)
('movie_worms_fwd_presentations', 133.0, 1361.1275500000002, 1370.1349985974523)
('movie_touch_of_evil_fwd_presentations', 134.0, 1371.1358300000002, 1380.1432885974527)
('movie_touch_of_evil_fwd_presentations', 135.0, 1381.14413, 1390.1515685974523)
('movie_flower_fwd_presentations', 136.0, 1391.15243, 1400.1598785974527)
('movie_flower_fwd_presentations', 137.0, 1401.1606900000002, 1410.1681585974527)
('movie_touch_of_evil_fwd_presentations', 138.0, 1411.16899, 1420.1764385974527)
('movie_flower_fwd_presentations', 139.0, 1421.17729, 1430.1847185974527)
('movie_worms_fwd_presentations', 140.0, 1431.18558, 1440.193028597453)
('movie_flower_fwd_presentations', 141.0, 1441.19386, 1450.2013185974529)
('movie_worms_fwd_presentations', 142.0, 1451.20214, 1460.2095785974527)
('movie_touch_of_evil_fwd_presentations', 143.0, 1461.21045, 1470.2178985974526)
('movie_touch_of_evil_fwd_presentations', 144.0, 1471.21871, 1480.226168597453)
('movie_touch_of_evil_fwd_presentations', 145.0, 1481.22702, 1490.2344785974526)
('movie_flower_fwd_presentations', 146.0, 1491.23528, 1500.2427685974526)
('movie_worms_fwd_presentations', 147.0, 1501.24359, 1510.2510485974526)
('movie_touch_of_evil_fwd_presentations', 148.0, 1511.25187, 1520.2593285974526)
('movie_flower_fwd_presentations', 149.0, 1521.26016, 1530.2676185974526)
('movie_worms_fwd_presentations', 150.0, 1531.26847, 1540.2759085974526)
('movie_flower_fwd_presentations', 151.0, 1541.27675, 1550.2842185974523)
('movie_worms_fwd_presentations', 152.0, 1551.28502, 1560.2924885974526)
('movie_flower_fwd_presentations', 153.0, 1561.29332, 1570.3007685974526)
('movie_touch_of_evil_fwd_presentations', 154.0, 1571.3016300000002, 1580.3090485974526)
('movie_worms_fwd_presentations', 155.0, 1581.3099, 1590.3173385974526)
('movie_worms_fwd_presentations', 156.0, 1591.31821, 1600.3256285974526)
('movie_flower_fwd_presentations', 157.0, 1601.32649, 1610.3339285974523)
('movie_flower_fwd_presentations', 158.0, 1611.3347700000002, 1620.3422185974523)
('movie_touch_of_evil_fwd_presentations', 159.0, 1621.3430500000002, 1630.3505085974523)
('movie_worms_fwd_presentations', 160.0, 1631.35133, 1640.3587785974526)
('movie_touch_of_evil_fwd_presentations', 161.0, 1641.35964, 1650.3670585974526)
('movie_flower_fwd_presentations', 162.0, 1651.36792, 1660.375388597453)
('movie_flower_fwd_presentations', 163.0, 1661.37623, 1670.3836685974527)
('movie_touch_of_evil_fwd_presentations', 164.0, 1671.38451, 1680.391958597453)
('movie_worms_fwd_presentations', 165.0, 1681.3928, 1690.4002385974527)
('movie_touch_of_evil_fwd_presentations', 166.0, 1691.40108, 1700.4085285974527)
('movie_worms_fwd_presentations', 167.0, 1701.40939, 1710.4168085974527)
('movie_touch_of_evil_fwd_presentations', 168.0, 1711.41765, 1720.425108597453)
('movie_worms_fwd_presentations', 169.0, 1721.42593, 1730.4333685974527)
('movie_worms_fwd_presentations', 170.0, 1731.43424, 1740.4416985974526)
('movie_flower_fwd_presentations', 171.0, 1741.44254, 1750.4499785974526)
('movie_touch_of_evil_fwd_presentations', 172.0, 1751.4508, 1760.4582685974526)
('movie_worms_fwd_presentations', 173.0, 1761.45908, 1770.466528597453)
('movie_flower_fwd_presentations', 174.0, 1771.46739, 1780.4748685974523)
('movie_flower_fwd_presentations', 175.0, 1781.47568, 1790.4831285974526)
('movie_touch_of_evil_fwd_presentations', 176.0, 1791.48397, 1800.4914085974526)
('movie_worms_fwd_presentations', 177.0, 1801.49224, 1810.4996985974528)
('movie_flower_fwd_presentations', 178.0, 1811.50056, 1820.5079985974528)
('movie_touch_of_evil_fwd_presentations', 179.0, 1821.50882, 1830.5162785974528)
('rotate_gabors_presentations', 180.0, 1861.54199, 2311.9150700000005)
('rotate_gabors_presentations', 181.0, 2311.9150700000005, 2762.2882200000004)
('fixed_gabors_presentations', 182.0, 2792.31309, 3242.68618)
('fixed_gabors_presentations', 183.0, 3242.68618, 3693.0593)
('gratings_presentations', 184.0, 3723.08417, 3910.439360497877)

Interpolation

Finally before plotting, in order to align these modalities, they must be interpolated to the same hertz and timescale. Set start_time and end_time to the bounds of the period you’re interested in seeing. That section of the data will be aligned and displayed together. interp_hz can also be adjusted if necessary to change the Hertz that the data are interpolated to.

start_time = 1000
end_time = 1100
interp_hz = 100

final_time = max(running_speed_timestamps[-1], eye_tracking_timestamps[-1], dff_timestamps[-1])
time_axis = np.arange(0, final_time, step=1/interp_hz)

f1 = interpolate.interp1d(running_speed_timestamps, running_speed_trace, axis=0, kind="nearest", fill_value="extrapolate")
interp_running_speed = f1(time_axis)

f2 = interpolate.interp1d(eye_tracking_timestamps, eye_tracking_area, axis=0, kind="nearest", fill_value="extrapolate")
interp_eye_area = f2(time_axis)

f3 = interpolate.interp1d(dff_timestamps, dff_trace, axis=0, kind="nearest", fill_value="extrapolate")
interp_dff = f3(time_axis) * 100 # to yield dF/F %
interp_dff = np.array(interp_dff).transpose() # transpose to make time axis secondary

start_idx = int(start_time * interp_hz)
end_idx = start_idx + int((end_time-start_time) * interp_hz)

time_axis_slice = time_axis[start_idx:end_idx]
interp_running_speed_slice = interp_running_speed[start_idx:end_idx]
interp_eye_area_slice = interp_eye_area[start_idx:end_idx]
interp_dff_slice = interp_dff[:,start_idx:end_idx]

# get blink times that occur between start_time and end_time
blink_start_idx, blink_end_idx = np.searchsorted(blink_times.timestamps, [start_time, end_time])
blink_idxs = np.where(blink_times.data[blink_start_idx:blink_end_idx])[0] # where blink time value is 'True'
interval_blink_times = blink_times.timestamps[blink_start_idx:blink_end_idx][blink_idxs]

Displaying Aligned Modalities

Below we defined the method show_epochs which displays the blocks of time that the epochs occupy. Lastly, each modality is displayed on their own subplots and all share the x-axis.

### on given axis, take each epoch and draw it as a uniquely colored rectangle
### returns dict of colors mapped to epochs
def show_epochs(ax, epochs):
    # make unique color for each stim name
    stim_names = list({epoch[0] for epoch in epochs})
    colors = plt.cm.rainbow(np.linspace(0,1,len(stim_names)))
    stim_color_map = {stim_names[i]:colors[i] for i in range(len(stim_names))}

    epoch_key = {}
    # draw colored rectangles for each epoch
    for epoch in epochs:
        stim_name, stim_block, epoch_start, epoch_end = epoch
        if epoch_end < time_axis[0] or epoch_start > time_axis[-1]:
            continue

        color = stim_color_map[stim_name]
        rec = ax.add_patch(mpl.patches.Rectangle((epoch_start, 0), epoch_end-epoch_start, 1, alpha=0.2, facecolor=color))
        epoch_key[stim_name] = rec

    ax.set_yticks([])
    return epoch_key

gridspec_dict = {"height_ratios":[0.5,2,2,4]}
fig, axes = plt.subplots(4,1, figsize=(10,10), sharex=True, gridspec_kw=gridspec_dict)

# epochs plot with legend
epoch_key = show_epochs(axes[0], epochs)
axes[0].set_xlim(time_axis_slice[0], time_axis_slice[-1])
axes[0].set_title("Stimulus Epochs")
legend_offset = 1
axes[0].legend(epoch_key.values(), epoch_key.keys(), ncols=3, fontsize=8, loc="upper left", bbox_to_anchor=(0,4))

# running speed modality
axes[1].plot(time_axis_slice, interp_running_speed_slice)
axes[1].set_ylabel("m/s")
axes[1].set_title("Running Speed")

# eye area modality with blink times marked
axes[2].plot(time_axis_slice, interp_eye_area_slice)
axes[2].set_ylabel("pixels²\nOrange markers are blink times")
ymin, ymax = axes[2].get_ylim()
axes[2].vlines(x=interval_blink_times, ymin=ymin, ymax=ymax, color="orange", lw=0.5, alpha=0.25, zorder=0)
axes[2].set_title("Eye Area")

# fluorescence modality
im = axes[3].imshow(interp_dff_slice, extent=[start_time, end_time, 0, interp_dff.shape[0]], aspect="auto", vmin=0, vmax=20) # change vmin and vmax to suit your plot
axes[3].set_ylabel("ROI #")
axes[3].set_xlabel("Time (s)")
axes[3].set_title("Fluorescence")
fig.colorbar(im, label="$\Delta$F/F (%)", orientation="horizontal", shrink=0.5)

fig.suptitle("Time-aligned modalities", fontsize=20)
plt.tight_layout()
plt.show()

Stimulus-Aligned Modality Windows

Similarly to in other notebooks, such as in Visualizing LFP Responses to Stimulus and Statistically Testing 2P Responses to Stimulus we can get response windows, time slices of the data that occur around the onset of stimulus, and then generate an average response window as one way to examine how these different components of the mouse’s activity act in response to the stimulus. Below, we perform this averaging, and then display the average response windows aligned together time. First, this will require selecting stimulus times. Below is shown the list of stimulus tables for this file and then the beginning of one such table. You can modify the stim_select code below or write your own code to select a list of stim_times from the table(s). The output should be a list of times that the stimulus you’re interested in was presented. In the code below, stim times are selected where the 40th frame of the natural worms movie is shown.

nwb.intervals.keys()

dict_keys(['fixed_gabors_presentations', 'gratings_presentations', 'movie_flower_fwd_presentations', 'movie_touch_of_evil_fwd_presentations', 'movie_worms_fwd_presentations', 'rotate_gabors_presentations', 'spontaneous_presentations'])

stim_table = nwb.intervals["movie_worms_fwd_presentations"]
print(stim_table.colnames)
stim_table[:10]

('start_time', 'stop_time', 'stimulus_name', 'stimulus_block', 'frame', 'color', 'contrast', 'opacity', 'ori', 'size', 'units', 'stimulus_index', 'tags', 'timeseries')

	start_time	stop_time	stimulus_name	stimulus_block	frame	color	contrast	opacity	ori	size	units	stimulus_index	tags	timeseries
id
0	70.058090	70.091453	movie_worms_fwd	4.0	0.0	[1.0, 1.0, 1.0]	1.0	1.0	0.0	[1920.0, 1080.0]	pix	2.0	[stimulus_time_interval]	[(2606, 2, timestamps pynwb.base.TimeSeries at...
1	70.091453	70.124816	movie_worms_fwd	4.0	1.0	[1.0, 1.0, 1.0]	1.0	1.0	0.0	[1920.0, 1080.0]	pix	2.0	[stimulus_time_interval]	[(2608, 2, timestamps pynwb.base.TimeSeries at...
2	70.124816	70.158179	movie_worms_fwd	4.0	2.0	[1.0, 1.0, 1.0]	1.0	1.0	0.0	[1920.0, 1080.0]	pix	2.0	[stimulus_time_interval]	[(2610, 2, timestamps pynwb.base.TimeSeries at...
3	70.158179	70.191542	movie_worms_fwd	4.0	3.0	[1.0, 1.0, 1.0]	1.0	1.0	0.0	[1920.0, 1080.0]	pix	2.0	[stimulus_time_interval]	[(2612, 2, timestamps pynwb.base.TimeSeries at...
4	70.191542	70.224905	movie_worms_fwd	4.0	4.0	[1.0, 1.0, 1.0]	1.0	1.0	0.0	[1920.0, 1080.0]	pix	2.0	[stimulus_time_interval]	[(2614, 2, timestamps pynwb.base.TimeSeries at...
5	70.224905	70.258268	movie_worms_fwd	4.0	5.0	[1.0, 1.0, 1.0]	1.0	1.0	0.0	[1920.0, 1080.0]	pix	2.0	[stimulus_time_interval]	[(2616, 2, timestamps pynwb.base.TimeSeries at...
6	70.258268	70.291631	movie_worms_fwd	4.0	6.0	[1.0, 1.0, 1.0]	1.0	1.0	0.0	[1920.0, 1080.0]	pix	2.0	[stimulus_time_interval]	[(2618, 2, timestamps pynwb.base.TimeSeries at...
7	70.291631	70.324994	movie_worms_fwd	4.0	7.0	[1.0, 1.0, 1.0]	1.0	1.0	0.0	[1920.0, 1080.0]	pix	2.0	[stimulus_time_interval]	[(2620, 2, timestamps pynwb.base.TimeSeries at...
8	70.324994	70.358357	movie_worms_fwd	4.0	8.0	[1.0, 1.0, 1.0]	1.0	1.0	0.0	[1920.0, 1080.0]	pix	2.0	[stimulus_time_interval]	[(2622, 2, timestamps pynwb.base.TimeSeries at...
9	70.358357	70.391720	movie_worms_fwd	4.0	9.0	[1.0, 1.0, 1.0]	1.0	1.0	0.0	[1920.0, 1080.0]	pix	2.0	[stimulus_time_interval]	[(2624, 2, timestamps pynwb.base.TimeSeries at...

stim_select = lambda row: float(row.frame) == 40
selected_stim_times = [float(stim_table[i].start_time) for i in range(len(stim_table)) if stim_select(stim_table[i])]
len(selected_stim_times)

60

Getting the Windows

Below, set window_start_time and window_end_time to be the time bounds, in seconds, of the response windows you want to extract. The function get_stim_windows takes these as input, along with the input data, where the last dimension is the time axis, and the array of timestamps and selected stimulus timestamps to extract the set of response windows. get_stim_windows is used to get the response windows for the mouse’s running speed, its eye area, and the dff recording from its brain.

window_start_time = -2
window_end_time = 3

# Takes data array whose last dimension is time, along with the data timestamps and all the stimulus timestamps. 
# Uses relative window start time and end time, returns align stimulus windows of the data
def get_stim_windows(data, data_timestamps, selected_stim_times, window_start_time, window_end_time):
    # get event windows
    windows = []
    window_length = int((window_end_time-window_start_time) * interp_hz)

    count = 0
    for stim_ts in selected_stim_times:
        # convert time to index
        start_idx = int( (stim_ts + window_start_time - data_timestamps[0]) * interp_hz )
        end_idx = start_idx + window_length
    
        # bounds checking
        if start_idx < 0 or end_idx > data.shape[-1]:
            count += 1
            continue
            
        windows.append(data[..., start_idx:end_idx])
        
    if len(windows) == 0:
        raise ValueError("There are no windows for these timestamps")

    return np.array(windows)

# validate window bounds
if window_start_time > 0:
    raise ValueError("start time must be non-positive number")
if window_end_time <= 0:
    raise ValueError("end time must be positive number")

### running speed, eye area, dff
speed_windows = get_stim_windows(interp_running_speed, running_speed_timestamps, selected_stim_times, window_start_time, window_end_time)
speed_time_axis = np.linspace(window_start_time, window_end_time, speed_windows.shape[1])
print(speed_windows.shape)

eye_area_windows = get_stim_windows(interp_eye_area, eye_tracking_timestamps, selected_stim_times, window_start_time, window_end_time)
eye_time_axis = np.linspace(window_start_time, window_end_time, eye_area_windows.shape[1])
print(eye_area_windows.shape)

dff_windows = get_stim_windows(interp_dff, dff_timestamps, selected_stim_times, window_start_time, window_end_time)
dff_windows = dff_windows
print(dff_windows.shape)

(60, 500)
(60, 500)
(60, 14, 500)

Displaying the Aligned Modalities Average Response Windows

With the input windows extracted, they can be averaged and displayed alongside each other aligned in time.

avg_speed_response = np.nanmean(speed_windows, axis=0)
avg_eye_response = np.nanmean(eye_area_windows, axis=0)
avg_dff_response = np.nanmean(dff_windows, axis=0)

print(avg_speed_response.shape)
print(avg_eye_response.shape)
print(avg_dff_response.shape)

(500,)
(500,)
(14, 500)

gridspec_dict = {"height_ratios":[2,2,4]}
fig, axes = plt.subplots(3,1, figsize=(10,10), sharex=True, gridspec_kw=gridspec_dict)

# running speed modality
axes[0].plot(speed_time_axis, avg_speed_response)
axes[0].set_ylabel("m/s")
axes[0].set_title("Running Speed")

# eye area modality with blink times marked
axes[1].plot(eye_time_axis, avg_eye_response)
axes[1].set_ylabel("pixels²")
axes[1].set_title("Eye Area")

# fluorescence modality
im = axes[2].imshow(avg_dff_response, extent=[window_start_time, window_end_time, 0, avg_dff_response.shape[0]], aspect="auto", vmin=0, vmax=20) # change vmin and vmax to suit your plot
axes[2].set_ylabel("ROI #")
axes[2].set_xlabel("Time (s)")
axes[2].set_title("Fluorescence")
fig.colorbar(im, label="$\Delta$F/F (%)", orientation="horizontal", shrink=0.5)

fig.suptitle("Time-aligned modalities", fontsize=20)
plt.tight_layout()
plt.show()

# get standard deviation across trials of these traces
running_std = np.nanstd(speed_windows, axis=0)
eye_std = np.nanstd(eye_area_windows, axis=0)

# sort ROIs in avg_dff_response by peak time
peak_times = np.argmax(avg_dff_response, axis=1)
sorted_idxs = np.argsort(peak_times)
sorted_avg_dff_response = avg_dff_response[sorted_idxs]

gridspec_dict = {"height_ratios":[4,4,4]}
fig, axes = plt.subplots(3,1, figsize=(10,10), sharex=True, gridspec_kw=gridspec_dict)

# running speed modality
axes[0].plot(speed_time_axis, avg_speed_response)
axes[0].fill_between(speed_time_axis, avg_speed_response+running_std, avg_speed_response-running_std, color="lightgrey")
axes[0].set_ylabel("m/s")
axes[0].set_title("Running Speed")

# eye area modality with blink times marked
axes[1].plot(eye_time_axis, avg_eye_response)
axes[1].fill_between(eye_time_axis, avg_eye_response+eye_std, avg_eye_response-eye_std, color="lightgrey")
axes[1].set_ylabel("pixels²")
axes[1].set_title("Eye Area")

# fluorescence modality
im = axes[2].imshow(sorted_avg_dff_response, extent=[window_start_time, window_end_time, 0, avg_dff_response.shape[0]], aspect="auto", vmin=0, vmax=30)
axes[2].set_ylabel("ROI #")
axes[2].set_xlabel("Time (s)")
axes[2].set_title("Fluorescence")
fig.colorbar(im, label="$\Delta$F/F (%)", orientation="horizontal", shrink=0.5, pad=0.2)

fig.suptitle(f"Time-aligned modalities\n(ROIs sorted by peak time) N = {len(selected_stim_times)}", fontsize=20)
plt.tight_layout()
plt.show()