{ "cells": [ { "cell_type": "markdown", "id": "6811cb23", "metadata": {}, "source": [ "# External Resources for Large-Scale Network Modeling" ] }, { "cell_type": "markdown", "id": "6a6332d1-d4a2-4461-857c-64f65fb3b666", "metadata": {}, "source": [ "Building, simulating, and analyzing large-scale network models can often be a resource intensive process. Although BMTK can often run simple-to-moderate network models using commercially available hardware - many scientists will find themselves quickly requiring access to High-Performance Computing (HPC) resources when the computational, memory, and/or storage requirements become greater than even the best modern laptops and desktops. When this happens you will have to rely on the resources of you institution's computing facilities or even on the Cloud Computing Platforms.\n", "\n", "Beyond access to more powerful computing and memory resources, there are other benefits to using Cloud and HPC resources in your workflow! Using a shared environment can improve collaboration and replication, reducing the headaches and pitfalls when every one has to install the software requirements on their own machines.\n", "\n", "In this tutorial we provide a list of tools and resources that neuroscientists can take advantage of.\n", "\n", "**Note** - scripts and files for running this tutorial can be found in the directory [06_opt_hpc_resources/](https://github.com/AllenInstitute/bmtk/tree/develop/docs/tutorial/06_opt_hpc_resources)" ] }, { "cell_type": "markdown", "id": "0bdf3e5e-4f4f-4a7e-bd2a-fdfed55612f6", "metadata": {}, "source": [ "## 1. Neuroscience Gateway (NSG) " ] }, { "cell_type": "markdown", "id": "1e8b4601-23fc-4283-b661-703d4acb45f5", "metadata": {}, "source": [ "[The Neuroscience Gateway (NSG)](http://www.nsgportal.org/) is an NSF funded project that provides HPC resouces for neuroscienists. It provides access to extensive CPU, GPU, and Memory resources that may not always be available locally. But just as important, NSG includes a wide variety of pre-installed computational neuroscience software that can be used for everything from modeling, data analysis, and even AI and ML applications. For modeling and simulation not only does NSG include BMTK, but also other popular tools like NEURON, NEST, Brian, PyNN, NetPyNE, among others. See [https://nsgprod.sdsc.edu:8443/portal2/tools.action](https://nsgprod.sdsc.edu:8443/portal2/tools.action) for a list of available software.\n", "\n", "The Neuroscience Gateway can be accessed in two different ways, either through the Web Portal interface or through the REST API. The focus in this tutorial will be on using the Web Portal." ] }, { "cell_type": "markdown", "id": "a00fbcc0-f7ab-4b03-9568-e89fa328e9e3", "metadata": {}, "source": [ "### 1.1. Example: Using NSG to run L4 Model simulations \n", "\n", "As an example of how one can utilize NSG running BMTK we will take the [existing Mouse V1 Layer 4 Model](https://github.com/AllenInstitute/bmtk-workshop/blob/main/Ch6_l4model/6.%20More%20Realistic%20Model%20and%20Advanced%20Features.ipynb) network and use it to run simulations on the Expanse computing cluster. " ] }, { "cell_type": "markdown", "id": "6abab8e1-7d63-47f9-9790-2b3e07dbcc5e", "metadata": {}, "source": [ "#### Step 1: (Register and) Log-in\n", "\n", "If you are planning to use NSG for your research you will first need to register an account at [https://www.nsgportal.org/gest/reg.php](https://www.nsgportal.org/gest/reg.php). It is freely available to most researchers, educators, and students working in the neuroscience field. \n", "\n", "After you have registered go ahead and log in to the [NSG Portal](https://nsgprod.sdsc.edu:8443/portal2)\n" ] }, { "cell_type": "markdown", "id": "d1a49cda-d9e1-48ed-8fa5-56b3c895ad2d", "metadata": {}, "source": [ "#### Step 2: Packaging the environment files\n", "\n", "We will need to package all necessary files into a .zip folder to be uploaded to NSG. This will include the run_pointnet.py python script to execute, plus any necessary SONATA and model files.\n", "\n", "**Remember** to download the files from the [Mouse V1 Layer 4 Model tutorial](https://github.com/AllenInstitute/bmtk-workshop/blob/main/Ch6_l4model/6.%20More%20Realistic%20Model%20and%20Advanced%20Features.ipynb)." ] }, { "cell_type": "markdown", "id": "3913c8f3-0913-466c-9a8d-ce7143f619ec", "metadata": {}, "source": [ "``` bash\n", " $ cd ../ && zip -rq l4model.zip Ch6_l4model/ \n", "```" ] }, { "cell_type": "markdown", "id": "896ab6e9-921c-418d-9673-476d9b3dfc4b", "metadata": {}, "source": [ "#### Step 3: Setting Up an NSG Folder\n", "\n", "1. When you login you will be prompted to ``Create New Folder`` that can be used for running BMTK applications. Click on that button, we will give it a \"Label\" of **bmtk_simulations** and an appropiate description:\n", "\n", "
\n", " \n", "
\n", "
\n", "\n", "2. Navagate to **bmtk_simulations** > **Data** and select the ``Upload Data`` button. Provide an appropiate label (we'll use Label **pointnet_l4model**) and click the ``Choose File`` button to upload the previously created zip file. Click ``Save`` to upload the file.\n", "\n", "
\n", " \n", "
\n", "
" ] }, { "cell_type": "markdown", "id": "b0a5f004-efec-418b-8fbd-4decf57188f9", "metadata": {}, "source": [ "#### Step 4: Setup and Run Task\n", "\n", "1. Navigate to the **bmtk_simulations** > **Tasks** folder and select ``Create New Task`` button.\n", "\n", "2. First give a ``Description`` for the task - we'll call it **pointnet_l4model**\n", "\n", "
\n", " \n", "
\n", "
\n", "\n", "\n", "3. Under the ``Select Data`` panel choose the uploaded zip file for with the **pointent_l4model** Label and click ``Select Data``\n", "\n", "
\n", " \n", "
\n", "
\n", "\n", "\n", "4. Under the ``Select Tool`` panel we select the **Python on Expanse** option\n", "\n", "
\n", " \n", "
\n", "
\n", "\n", "5. Finally Select the ``Set Parameters`` Panel to give the run-time parameters. Enter the parameters as specified below and click the ``Save Parameters`` button.\n", "\n", "
\n", " \n", "
\n", "
\n", "\n", "* Set **Max Hours to Run: 1.0** - Depending on the size of the network, the **tstop** and **dt**, and the number of Cores we may have to increase/decrease this value.\n", "* Unselect **Do you require nrnivmodl compilation** - flag only applies to NEURON (eg. BioNet) applications.\n", "* Set **Enter Main Input Python Filename: run_pointnet.py** - Entry python script to run.\n", "* Set **Enter sub-directory name: Ch6_l4model** - This is required since the way we zipped up our data everything is under the *Ch6_l4model/* folder.\n", " * To speed up the simulation we will run it parallelized using MPI with 8 processes. \n", " * Set **Enter Number of Nodes: 1** - Since we use significantly less than the available cores/memory per node we can benefit from keeping all MPI Tasks on the same node.\n", " * Set **Enter Number of MPI Tasks per Node: 8** and **Enter Number of Cores per Node: 8**\n", " * Set **Enter Number of GB Memory per Node: 16**\n", "\n", "**Notes: Building the Network and LGN Inputs**\n", "\n", "Running the **run_pointnet.py** script requires that the SONATA network files (*network/*) and lgn inputs (*inputs/spikes.gratings.90deg_4Hz.h5*) already exists. If they don't, or you want to re-generate them, you can replace the **Enter Main Input Python Filename:** option with the *build_network.py* or *run_filternet.py* scripts.\n", "\n", "\n", "6. Now that you have selected the data, the tool, and the parameters click the ``Save and Run Task`` button to submit the task to the HPC." ] }, { "cell_type": "markdown", "id": "e3bf8ed6-3d5f-41ce-8a4a-629308ca045d", "metadata": {}, "source": [ "#### Fetching the Results\n", "\n", "After clicking on `Save and Run Task` the task will be submitted to the NSQ queue. Depending on current HPC usage and the requested resources there may be a delay between the time when a job is submitted and the time the task starts running. You can select the `View Status` button to see the **Status** of the current task.\n", "\n", "
\n", " \n", "
\n", "
\n", "\n" ] }, { "cell_type": "markdown", "id": "5f027d47-1960-488a-a926-8c0a373d4ccf", "metadata": {}, "source": [ "### 1.2. Example: Using NSG for BioNet (eg. NEURON) Simulations \n", "\n", "The above tutorial will work when running *PointNet*, *FilterNet*, *Builder*, or *Analysis* components of BMTK. However running *BioNet* simulations require a slight changes to intialize a task properly since it requires using the NEURON simulation underneath. In this example we will show a how to run a multi-core BioNet simualation using the network generated in [bmtk-workshop Chapter 3](https://github.com/AllenInstitute/bmtk-workshop/blob/main/Ch3_multicells/3.%20Multipopulation%20biophysical.ipynb) taking note to highlight difference from the example above." ] }, { "cell_type": "markdown", "id": "1531052f-e284-462f-9428-5930688fb19f", "metadata": {}, "source": [ "#### Step 1: Packaging Data\n", "\n", "We first want to package the *Ch3_mutlicells/* folder into a single zip file so it will contain the python scripts and all necessary network and model files to run the full simulation. **HOWEVER** The modfiles stored in the *components/mechanisms/modfiles/* folder need to be moved to the root directory. NSG will need to compile these files before running the simulation and if they are stored under the *components/* sub-directory they will not be found." ] }, { "cell_type": "markdown", "id": "dcca7f4e-9945-48b9-b232-e2f683d630cb", "metadata": {}, "source": [ "``` bash\n", " $ cp -r ../Ch3_multicells .\n", " $ mv Ch3_multicells/components/mechanisms/modfiles/* Ch3_multicells/\n", " $ zip -rq bionet_multicells.zip Ch3_multicells/\n", "```" ] }, { "cell_type": "markdown", "id": "0cdee10f-defa-4ee8-8588-c9ef762860c5", "metadata": {}, "source": [ "#### Step 2: Uploading the Data\n", "1. Log into NSGPortal and either select ``Create New Folder``, or select the existing **bmtk_simulations** folder if it exists.\n", "\n", "2. Under **bmtk_simulations** > **Data** folder select `Upload Data` to upload the created *bionet_multicells.zip* folder with an descriptive Label\n", "\n", "
\n", " \n", "
\n", "
\n", " " ] }, { "cell_type": "markdown", "id": "6b7f06a5-4555-4302-89a9-8c6d51df9f82", "metadata": {}, "source": [ "#### Step 3: Creating and Submitting a Task\n", "\n", "Under **bmtk_simulations** > **Tasks** select the `Create New Task` button\n", "\n", "1. In the `Task Summary` tab give the task an appropiate description (we'll call it **bionet_multicells**)\n", "\n", "\n", "2. In the `Select Data` tab make sure to select the **bionet_multicell** data we uploaded in the previous step and click `Select Data`.\n", "\n", "\n", "3. In the `Select Tool` tab choose option **NEURON on Expanse** \n", "\n", "
\n", " \n", "
\n", "
\n", "\n", "4. In the `Set Parameters` tab we can use the following setup, the click `Save Parameters` button.\n", "\n", "
\n", " \n", "
\n", "
\n", "\n", "The following parameters are important\n", "\n", "* Make sure the select **Do you require nrnivmodl compilation** in order to compile the .mod file necessary to run the models\n", "* Set **Enter Main Input Filename: run_bionet.py** as this is the script to execute the BMTK simulation\n", "- Make sure to select **Please click here if your code is in python and not HOC**\n", "* Due to the structure of the zip file we need to set **Enter sub-directory name: Ch3_mutlicells**\n", "\n", "Other settings that can be adjusted:\n", "\n", "* For a modest network **Maximum Hours to Run: 0.5** is plenty enough. But for longer simulations or larger networks you may need to increase this value otherwise the job will stop before BMTK has finished.\n", "* We will set **Number of MPI Tasks per Node** and **Number of Cores per Node** to **8**, and **Number of Nodes** to **1** to parallelize the simulation on 8 cores.\n", "* For current network **Number of GB Memory per Node: 32** will suffice.\n", "\n", "\n", "5. Finally select `Save and Run Task` to submit the Task to the queue." ] }, { "cell_type": "code", "execution_count": null, "id": "5d65754c-315e-4aad-a2d7-a4d0be89d7f3", "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.16" } }, "nbformat": 4, "nbformat_minor": 5 }