#################################### Network Builder Features and Options #################################### NetworkBuilder Methods ====================== Iterating and Filtering Nodes from a Network using the ``nodes()`` Method ------------------------------------------------------------------------- You can use the The :py:meth:`NetworkBuilder.nodes() ` method of the ``NetworkBuilder`` class to iterate over the nodes stored in a network. The ``nodes()`` method returns an iterable list which contains dictionaries of the properties of each individual cell that will be created by each call to ``add_nodes()``. .. code:: python import numpy as np from bmtk.builder.networks import NetworkBuilder net = NetworkBuilder('my_model') net.add_nodes( N=10, model_type='biophysical', cell_order=range(10), coord=np.random.uniform(-100.0, 100.0, 10) ) assert(len(net.nodes()) == 10) for node_dict in net.nodes(): print(node_dict['node_id'], ':', node_dict) The above will show each 10 nodes/cells created by the ``add_nodes()`` call: .. code:: bash 0 : {'model_type': 'biophysical', 'node_type_id': 100, 'cell_order': 0, 'coord': -74.13859238611045, 'node_id': 0} 1 : {'model_type': 'biophysical', 'node_type_id': 100, 'cell_order': 1, 'coord': -22.88965188502479, 'node_id': 1} 2 : {'model_type': 'biophysical', 'node_type_id': 100, 'cell_order': 2, 'coord': 54.31170697608857, 'node_id': 2} ... You can also filter by property values by passing in a condition as a parameter. .. code:: python from bmtk.builder.networks import NetworkBuilder from pprint import pprint net = NetworkBuilder('my_model') net.add_nodes( N=5, model_type='biophysical', ei='exc', cell_order=range(5), ) net.add_nodes( N=5, model_type='biophysical', ei='inh', cell_order=range(5), ) net.add_nodes( N=5, model_type='point', ei='exc', cell_order=range(5), ) net.add_nodes( N=5, model_type='point', cell_order=range(5), ) pprint(list(net.nodes(ei='exc'))) The above will return only the 10 nodes from the two calls with ``ei=='exc'``. If you want to filter by multiple properties in the format ``prop1==A && prop2==B && prop3==C`` you can just pass in multiple key-value pairs into the ``nodes()`` parameters. For example if we want to find only those cells that are **excitatory point** neurons from the example above: .. code:: python pprint(list(net.nodes(ei='exc', model_type='point'))) Iterating and Filtering Edges using the edges() Method ------------------------------------------------------ Similar to the ``nodes()`` method, the :py:meth:`NetworkBuilder.edges() ` method allows you to iterate and filter all the edges/synapses generated by each call to ``add_edges()``. When called with no parameters it will return a list of all edges/synapses in a dictionary like format. .. code:: python from bmtk.builder.networks import NetworkBuilder net = NetworkBuilder('my_model') net.add_nodes( N=5, model_type='biophysical', ei='exc' ) net.add_nodes( N=5, model_type='biophysical', ei='inh', ) net.add_nodes( N=5, model_type='point', ei='exc', ) net.add_edges( source=net.nodes(ei='inh'), target=net.nodes(ei='exc'), connection_rule=lambda *_: np.random.randint(0, 10), syn_function='inh2exc', syn_weight=-20.0 ) net.add_edges( source=net.nodes(ei='exc', model_type='point'), target=net.nodes(model_type='biophysical'), connection_rule=1, syn_function='point_exc', syn_weight=10.0 ) edge0 = net.edges()[0] print(edge0['source_node_id']) print(edge0['target_node_id']) print(edge0['syn_function']) print(edge0['syn_weight']) for edge in net.edges(): print(edge) BMTK will display the edge parameters from an example small network as follows: (**note that the order of edges may be different**) .. code:: 5 1 inh2exc -20.0 5 --> 1 {syn_function: inh2exc, syn_weight: -20.0, edge_type_id: 100, source_query: ei=='inh', target_query: ei=='exc', source_node_id: 5, target_node_id: 1, nsyns: 4} 5 --> 2 {syn_function: inh2exc, syn_weight: -20.0, edge_type_id: 100, source_query: ei=='inh', target_query: ei=='exc', source_node_id: 5, target_node_id: 2, nsyns: 5} 5 --> 4 {syn_function: inh2exc, syn_weight: -20.0, edge_type_id: 100, source_query: ei=='inh', target_query: ei=='exc', source_node_id: 5, target_node_id: 4, nsyns: 7} .. important:: Before the ``edges()`` method is called the `connectivity matrix needs to be created `__ which is equivalent to calling the ``build()`` method. For very large networks this can take a long amount of time and memory. Also note that you will need to rebuild the network if you call ``add_edges()`` or ``add_nodes()`` afterwards. As such it is highly recommend to not call the ``edge()`` iterator until after all calls to ``add_nodes()`` and ``add_edges()`` are completed. The easiest way to filter only certain edges by query is, like with the ``nodes()`` method, add key-value parameters to the method. If more than one parameter is called then it will be treated as a logical "and". .. code:: python # We should expect 5x(5+5) = 50 edges in total edges = net.edges(syn_function='point_exc', syn_weight=10.0) print(len(edges)) for edge in edges: # print(edge) assert(edge['nsyns'] == 1) assert(edge['syn_weight'] > 0) # Due to the "syn_weight" conditional we expect to find no (0) edges. edges = net.edges(syn_function='point_exc', syn_weight=-20.0) print(len(edges)) You can also filter edges by either source and/or target properties using the **source_nodes** and **target_nodes** parameters, respectively. Use the ``nodes()`` iterator to create filters for nodes (this will work even if a source/target set of nodes comes from a different population - as long as the edge still exists). .. code:: python # We should expect <= 50 inh -> exc edges. edges = net.edges( source_nodes=net.nodes(ei='inh'), target_nodes=net.nodes(ei='exc') ) assert(len(edges) > 0) for edge in edges: assert(edge['syn_function'] == 'inh2exc') assert(edge['syn_weight'] < 0.0) edges = net.edges( source_nodes=net.nodes(ei='inh'), target_nodes=net.nodes(ei='exc') ) Importing Existing Nodes ------------------------ If you have an existing SONATA network (.h5 and .csv file) you can import the nodes using the ``import_nodes()`` method. .. code:: python from bmtk.builder.networks import NetworkBuilder net = NetworkBuilder('my_model') # expect 0 nodes print(len(net.nodes())) net.import_nodes( nodes_file_name='network/v1_nodes.h5', node_types_file_name='network/v1_node_types.csv' ) # Should have more than 0 nodes print(len(net.nodes())) Connector Functions =================== Options for Setting Synapse Location ------------------------------------ When creating a network with morphologically realistic cells, the location of the afferent synapses along a given cell dendrites/soma is often very important. However trying to manually place each synapse quickly becomes an intractable problem even for small networks. Also with simulators like NEURON it is very difficult to know the name/identifier of each dendritic branch and segment prior to simulation. BMTK Network Builder along with BioNet offers some solutions to make it easier to manage placing synaptic locations when using the ``add_edges()`` calls. "target_sections" and "distance_range" properties +++++++++++++++++++++++++++++++++++++++++++++++++ The easiest and most efficient solution is to add the properties **target_section** and **distance_range** when calling the ``add_edges()`` method. **target_section** is a list of of one or more cell section names as defined in the SWC format (*axon*, *somatic*, *basal*, *apical*, *dend*, *other*). while **distance_range** is a vector of two values - the start and end distance (both inclusive) in arc-length a segment has to be away from the hilt of the soma. For example, we want to create edges that only connect to *apical* or *basal* dendritic sections that are between 50-200 microns away from the soma. (If you want connections at the end of the dendrite you can use a large number like ``[50.0, 1.0e20]``). .. code:: python net.add_edges( source=net.nodes(ei='exc'), target=net.nodes(model_type='biophysical', morphology='pyramidal.swc'), ... distance_range=[50.0, 200.0], target_sections=['basal', 'apical'], ... ) At the beginning of each simulation, *BioNet* will see the **target_section** and **distance_range** parameters and use them to find every morphology segment that matches, then choose one at random and create a synapse. There are two potential issues when using this method for assigning synapse locations: 1. **target_section** and **distance_range** are not SONATA reserved keywords, meaning other tools will likely not be able to use these parameters to regenerate synaptic locations. 2. Since the synapses are generated by *BioNet* dynamically before each simulation, it could lead to issues with reproducibility of network results. Explicitly setting afferent synapse section and position ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ The more reproducible alternative is to set the parameters **afferent_section_id** and **afferent_section_pos**, which are reserve SONATA keywords recognized by other tools, for every synapse. To help with this you can use the :py:meth:`NetworkBuilder.nodes() ` method. Point to this method in the ``add_properties()`` call and choose the **target_section** and **distance_range**, like above. When build() is called it will choose appropriate synapse location and store them in the SONATA file. .. code:: python from bmtk.builder.bionet import rand_syn_locations cm = net.add_edges( source=net.nodes(ei='exc'), target=net.nodes(model_type='biophysical', morphology='pyramidal.swc'), ... distance_range=[50.0, 200.0], target_sections=['basal', 'apical'], ... ) cm.add_properties( ['afferent_section_id', 'afferent_section_pos'], rule=rand_syn_locations, rule_params={ 'section_names': ['somatic', 'basal'], 'distance_range': [50.0, 200.0], 'morphology_dir': './components/morphologies' }, dtypes=[int, float] ) When ``build()`` is called on the previous code it will load the 'pyramidal.swc' file with NEURON to determine how the simulator will generate and name all the branches and segments. If you prefer you can use the ``SWCReader`` class inside your script to access information about available sections and segments (including x, y, z coordinates): .. code:: python from bmtk.builder.bionet.swc_reader import SWCReader morph = SWCReader('./components/morphologies/pyramidal.swc') morph.set_segment_dl(dL) morph.move_and_rotate( soma_coords=[x, y, z], rotation_angles=[rotation_angle_xaxis, rotation_angle_yaxis, rotation_angle_zaxis], ) Built-in Connection Rules ------------------------- The BMTK Network Builder includes a number of built-in functions that you can use for the **connection_rule** option when using ``add_edges()`` method. :py:meth:`bmtk.builder.auxi.edge_connector.distance_connector() ` - Creates connections between source and target cells based on distance, cells that are closer together will more likely to have more connections between them (will rule-out self-connections). :py:meth:`bmtk.builder.auxi.edge_connector.connect_random() ` - Will choose at random if two cells share a connection or not. Options for Saving a Network to SONATA format ============================================= Network File Name and Path -------------------------- By default, when you when have a network with name "NETNAME" and call ``NetworkBuilder.save()`` or ``NetworkBuilder.save_nodes()``, the Builder will save it to file names *NETNAME_nodes.h5* and *NETNAME_nodes.csv*. Similarly, edges from populations "NETSRC" to "NETTRG" will be saved by default to file names *NETSRC_to_NETTRG_edges.h5* and *NETSRC_to_NETTRG_edge_types.csv*. To change the file names, use the **node[_type]s_file_name** and **edge[_types]_file_name** parameters. You can use absolute file paths, and when using relative file paths, it will place it under the **output_dir** option: .. code:: python # will save the network nodes and edges files into "network/custom-*" file-name. net.save( output_dir='./network/', nodes_file_name='custom-nodes.h5' node_types_file_name='custom-node-types.csv', edges_file_name='custom-edges.h5', edge_types_file_name='custom-edge-types.csv', ) Append Mode ----------- When the ``NetworkBuilder.save()`` method is called it will by default overwrite any existing SONATA .h5 and .csv files if they already exist (you can change this behavior by setting ``force_overwrite=False``). The SONATA format does allow you to save multiple nodes and edges populations in the same files - although each population must have a unique name. With BMTK you must set the **mode** parameter to ``a`` or ``append``. .. code:: python # Assumes network/shared-nodes.h5 already exists and user wants to append another node population into it. net.save( output_dir='./network/', mode='a', nodes_file_name='shared-nodes.h5' ) Sorting of Edges ---------------- By default, the edges are sorted by "target_node_id" when the NetworkBuilder saves the SONATA edges file (ie. all the edges with post-synapse node 0 comes before edges with post-synapse node 1). The **sort_by** parameter in the ``save()`` method allows you to choose between sorting by: * "target_node_id" - Group and sort by post-synaptic node-id * "source_node_id" - Group and sort by pre-synaptic node-id * "edge_type_id" - Will group edges that share the same **edge-type** properties and id. .. code:: python net.save( output_dir='./network/', sort_by='source_node_id', ) Edges Indexing -------------- SONATA edges formats allows the addition of indices, which can be used to quickly search for edges based on either the target_node_id, source_node_id, or edge_type_id. By default, the BMTK Network Builder will create indices for both target_node_id and source_node_id. This can be changed using the **index_by** parameter. To build an edges file with **No indexes**: .. code:: python net.save( index_by=None, ... ) To build an edges file with only **target_node_id indices**: .. code:: python net.save( index_by=('target_node_id') ... ) Or to build an edges file with **all indexing options**: .. code:: python net.save( index_by=('target_node_id', 'source_node_id', 'edge_type_id') ... ) File Compression ---------------- the `HDF5 Format allows for data-sets to be compressed inside the file `__. To save an hdf5 file with compression, use the **compression** option (**note** that the options for compression may vary depending on machine or hdf5/h5py libraries installed). To save with GZIP compression: .. code:: python net.save( compression='gzip' ... ) To ensure no compression use the ``None`` option: .. code:: python net.save( compression=None ... ) Options for Optimization of Network Building ============================================ Iterators ---------- Building with Multiple Cores ---------------------------- When running on a computer or cluster with `MPI `__ installed, the BMTK Network Builder can automatically parallelize building the network. Just make sure to call the script with the appropriate command for your system (eg. ``mpirun``, ``mpiexec``, ``srun``, etc). .. code:: bash # Split building of script across 10 cores $ mpirun -np 10 python build_network.py The resulting SONATA network file will be similar to the file created with a single core. There is no need for the user to change their code. The one **exception** is when passing is parameters to either the ``add_nodes()`` or ``add_edges()`` methods that rely on randomized values. For example .. code:: python net.add_nodes( N=100, x=np.random.normal(0.0, 100.0, 100), y=np.random.normal(0.0, 100.0, 100), z=np.random.uniform(-100.0, 100.0, 100), ... ) In such cases, when not taking extra precautions, each rank/core will generate different values for each cell's (x, y, z) coordinates, and BMTK will not know which values to save into the final file. The Network Builder will by default check if values are different across ranks and throw an error if needed. The simplest solution is to make sure to hard-code the random seed. This will better ensure that the same random numbers are generated across all ranks. .. code:: python np.random.seed(100) net.add_nodes( N=100, x=np.random.normal(0.0, 100.0, 100), y=np.random.normal(0.0, 100.0, 100), z=np.random.uniform(-100.0, 100.0, 100), ... ) Another option is to use the **mpi4py** ``Bcast`` function to generate values on one rank then send them to all the others.