Network Builder Features and Options#

NetworkBuilder Methods#

Iterating and Filtering Nodes from a Network using the nodes() Method#

You can use the The NetworkBuilder.nodes() method of the NetworkBuilder class to iterate over the nodes stored in a network. The nodes() method returns an iterable list which contains dictionaries of the properties of each individual cell that will be created by each call to add_nodes().

import numpy as np
from bmtk.builder.networks import NetworkBuilder

net = NetworkBuilder('my_model')
net.add_nodes(
    N=10,
    model_type='biophysical',
    cell_order=range(10),
    coord=np.random.uniform(-100.0, 100.0, 10)
)

assert(len(net.nodes()) == 10)
for node_dict in net.nodes():
    print(node_dict['node_id'], ':', node_dict)

The above will show each 10 nodes/cells created by the add_nodes() call:

0 :  {'model_type': 'biophysical', 'node_type_id': 100, 'cell_order': 0, 'coord': -74.13859238611045, 'node_id': 0}
1 :  {'model_type': 'biophysical', 'node_type_id': 100, 'cell_order': 1, 'coord': -22.88965188502479, 'node_id': 1}
2 :  {'model_type': 'biophysical', 'node_type_id': 100, 'cell_order': 2, 'coord': 54.31170697608857, 'node_id': 2}
...

You can also filter by property values by passing in a condition as a parameter.

from bmtk.builder.networks import NetworkBuilder
from pprint import pprint

net = NetworkBuilder('my_model')
net.add_nodes(
    N=5,
    model_type='biophysical',
    ei='exc',
    cell_order=range(5),
)
net.add_nodes(
    N=5,
    model_type='biophysical',
    ei='inh',
    cell_order=range(5),
)
net.add_nodes(
    N=5,
    model_type='point',
    ei='exc',
    cell_order=range(5),
)

net.add_nodes(
    N=5,
    model_type='point',
    cell_order=range(5),
)

pprint(list(net.nodes(ei='exc')))

The above will return only the 10 nodes from the two calls with ei=='exc'. If you want to filter by multiple properties in the format prop1==A && prop2==B && prop3==C you can just pass in multiple key-value pairs into the nodes() parameters. For example if we want to find only those cells that are excitatory point neurons from the example above:

pprint(list(net.nodes(ei='exc', model_type='point')))

Iterating and Filtering Edges using the edges() Method#

Similar to the nodes() method, the NetworkBuilder.edges() method allows you to iterate and filter all the edges/synapses generated by each call to add_edges(). When called with no parameters it will return a list of all edges/synapses in a dictionary like format.

from bmtk.builder.networks import NetworkBuilder

net = NetworkBuilder('my_model')
net.add_nodes(
    N=5,
    model_type='biophysical',
    ei='exc'
)
net.add_nodes(
    N=5,
    model_type='biophysical',
    ei='inh',
)
net.add_nodes(
    N=5,
    model_type='point',
    ei='exc',
)

net.add_edges(
    source=net.nodes(ei='inh'),
    target=net.nodes(ei='exc'),
    connection_rule=lambda *_: np.random.randint(0, 10),
    syn_function='inh2exc',
    syn_weight=-20.0
)

net.add_edges(
    source=net.nodes(ei='exc', model_type='point'),
    target=net.nodes(model_type='biophysical'),
    connection_rule=1,
    syn_function='point_exc',
    syn_weight=10.0
)

edge0 = net.edges()[0]
print(edge0['source_node_id'])
print(edge0['target_node_id'])
print(edge0['syn_function'])
print(edge0['syn_weight'])

for edge in net.edges():
    print(edge)

BMTK will display the edge parameters from an example small network as follows: (note that the order of edges may be different)

5
1
inh2exc
-20.0
5 --> 1 {syn_function: inh2exc, syn_weight: -20.0, edge_type_id: 100, source_query: ei=='inh', target_query: ei=='exc', source_node_id: 5, target_node_id: 1, nsyns: 4}
5 --> 2 {syn_function: inh2exc, syn_weight: -20.0, edge_type_id: 100, source_query: ei=='inh', target_query: ei=='exc', source_node_id: 5, target_node_id: 2, nsyns: 5}
5 --> 4 {syn_function: inh2exc, syn_weight: -20.0, edge_type_id: 100, source_query: ei=='inh', target_query: ei=='exc', source_node_id: 5, target_node_id: 4, nsyns: 7}

Important

Before the edges() method is called the connectivity matrix needs to be created which is equivalent to calling the build() method. For very large networks this can take a long amount of time and memory. Also note that you will need to rebuild the network if you call add_edges() or add_nodes() afterwards.

As such it is highly recommend to not call the edge() iterator until after all calls to add_nodes() and add_edges() are completed.

The easiest way to filter only certain edges by query is, like with the nodes() method, add key-value parameters to the method. If more than one parameter is called then it will be treated as a logical “and”.

# We should expect 5x(5+5) = 50 edges in total
edges = net.edges(syn_function='point_exc', syn_weight=10.0)
print(len(edges))
for edge in edges:
    # print(edge)
    assert(edge['nsyns'] == 1)
    assert(edge['syn_weight'] > 0)

# Due to the "syn_weight" conditional we expect to find no (0) edges.
edges = net.edges(syn_function='point_exc', syn_weight=-20.0)
print(len(edges))

You can also filter edges by either source and/or target properties using the source_nodes and target_nodes parameters, respectively. Use the nodes() iterator to create filters for nodes (this will work even if a source/target set of nodes comes from a different population - as long as the edge still exists).

# We should expect <= 50 inh -> exc edges.
edges = net.edges(
    source_nodes=net.nodes(ei='inh'),
    target_nodes=net.nodes(ei='exc')

)
assert(len(edges) > 0)
for edge in edges:
    assert(edge['syn_function'] == 'inh2exc')
    assert(edge['syn_weight'] < 0.0)

edges = net.edges(
    source_nodes=net.nodes(ei='inh'),
    target_nodes=net.nodes(ei='exc')

)

Importing Existing Nodes#

If you have an existing SONATA network (.h5 and .csv file) you can import the nodes using the import_nodes() method.

from bmtk.builder.networks import NetworkBuilder

net = NetworkBuilder('my_model')

# expect 0 nodes
print(len(net.nodes()))

net.import_nodes(
    nodes_file_name='network/v1_nodes.h5',
    node_types_file_name='network/v1_node_types.csv'
)

# Should have more than 0 nodes
print(len(net.nodes()))

Connector Functions#

Options for Setting Synapse Location#

When creating a network with morphologically realistic cells, the location of the afferent synapses along a given cell dendrites/soma is often very important. However trying to manually place each synapse quickly becomes an intractable problem even for small networks. Also with simulators like NEURON it is very difficult to know the name/identifier of each dendritic branch and segment prior to simulation.

BMTK Network Builder along with BioNet offers some solutions to make it easier to manage placing synaptic locations when using the add_edges() calls.

“target_sections” and “distance_range” properties#

The easiest and most efficient solution is to add the properties target_section and distance_range when calling the add_edges() method. target_section is a list of of one or more cell section names as defined in the SWC format (axon, somatic, basal, apical, dend, other). while distance_range is a vector of two values - the start and end distance (both inclusive) in arc-length a segment has to be away from the hilt of the soma.

For example, we want to create edges that only connect to apical or basal dendritic sections that are between 50-200 microns away from the soma. (If you want connections at the end of the dendrite you can use a large number like [50.0, 1.0e20]).

net.add_edges(
    source=net.nodes(ei='exc'),
    target=net.nodes(model_type='biophysical', morphology='pyramidal.swc'),
    ...
    distance_range=[50.0, 200.0],
    target_sections=['basal', 'apical'],
    ...
)

At the beginning of each simulation, BioNet will see the target_section and distance_range parameters and use them to find every morphology segment that matches, then choose one at random and create a synapse.

There are two potential issues when using this method for assigning synapse locations:

1. target_section and distance_range are not SONATA reserved keywords, meaning other tools will likely not be able to use these parameters to regenerate synaptic locations. 2. Since the synapses are generated by BioNet dynamically before each simulation, it could lead to issues with reproducibility of network results.

Explicitly setting afferent synapse section and position#

The more reproducible alternative is to set the parameters afferent_section_id and afferent_section_pos, which are reserve SONATA keywords recognized by other tools, for every synapse. To help with this you can use the NetworkBuilder.nodes() method. Point to this method in the add_properties() call and choose the target_section and distance_range, like above. When build() is called it will choose appropriate synapse location and store them in the SONATA file.

from bmtk.builder.bionet import rand_syn_locations

cm = net.add_edges(
    source=net.nodes(ei='exc'),
    target=net.nodes(model_type='biophysical', morphology='pyramidal.swc'),
    ...
    distance_range=[50.0, 200.0],
    target_sections=['basal', 'apical'],
    ...
)

cm.add_properties(
    ['afferent_section_id', 'afferent_section_pos'],
    rule=rand_syn_locations,
    rule_params={
        'section_names': ['somatic', 'basal'],
        'distance_range': [50.0, 200.0],
        'morphology_dir': './components/morphologies'
    },
    dtypes=[int, float]
)

When build() is called on the previous code it will load the ‘pyramidal.swc’ file with NEURON to determine how the simulator will generate and name all the branches and segments. If you prefer you can use the SWCReader class inside your script to access information about available sections and segments (including x, y, z coordinates):

from bmtk.builder.bionet.swc_reader import SWCReader

morph = SWCReader('./components/morphologies/pyramidal.swc')
morph.set_segment_dl(dL)
morph.move_and_rotate(
    soma_coords=[x, y, z],
    rotation_angles=[rotation_angle_xaxis, rotation_angle_yaxis, rotation_angle_zaxis],
)

Built-in Connection Rules#

The BMTK Network Builder includes a number of built-in functions that you can use for the connection_rule option when using add_edges() method.

bmtk.builder.auxi.edge_connector.distance_connector() - Creates connections between source and target cells based on distance, cells that are closer together will more likely to have more connections between them (will rule-out self-connections).

bmtk.builder.auxi.edge_connector.connect_random() - Will choose at random if two cells share a connection or not.

Options for Saving a Network to SONATA format#

Network File Name and Path#

By default, when you when have a network with name “NETNAME” and call NetworkBuilder.save() or NetworkBuilder.save_nodes(), the Builder will save it to file names NETNAME_nodes.h5 and NETNAME_nodes.csv. Similarly, edges from populations “NETSRC” to “NETTRG” will be saved by default to file names NETSRC_to_NETTRG_edges.h5 and NETSRC_to_NETTRG_edge_types.csv.

To change the file names, use the node[_type]s_file_name and edge[_types]_file_name parameters. You can use absolute file paths, and when using relative file paths, it will place it under the output_dir option:

# will save the network nodes and edges files into "network/custom-*" file-name.
net.save(
    output_dir='./network/',
    nodes_file_name='custom-nodes.h5'
    node_types_file_name='custom-node-types.csv',
    edges_file_name='custom-edges.h5',
    edge_types_file_name='custom-edge-types.csv',
)

Append Mode#

When the NetworkBuilder.save() method is called it will by default overwrite any existing SONATA .h5 and .csv files if they already exist (you can change this behavior by setting force_overwrite=False). The SONATA format does allow you to save multiple nodes and edges populations in the same files - although each population must have a unique name. With BMTK you must set the mode parameter to a or append.

# Assumes network/shared-nodes.h5 already exists and user wants to append another node population into it.
net.save(
    output_dir='./network/',
    mode='a',
    nodes_file_name='shared-nodes.h5'
)

Sorting of Edges#

By default, the edges are sorted by “target_node_id” when the NetworkBuilder saves the SONATA edges file (ie. all the edges with post-synapse node 0 comes before edges with post-synapse node 1). The sort_by parameter in the save() method allows you to choose between sorting by:

  • “target_node_id” - Group and sort by post-synaptic node-id

  • “source_node_id” - Group and sort by pre-synaptic node-id

  • “edge_type_id” - Will group edges that share the same edge-type properties and id.

net.save(
    output_dir='./network/',
    sort_by='source_node_id',
)

Edges Indexing#

SONATA edges formats allows the addition of indices, which can be used to quickly search for edges based on either the target_node_id, source_node_id, or edge_type_id. By default, the BMTK Network Builder will create indices for both target_node_id and source_node_id. This can be changed using the index_by parameter.

To build an edges file with No indexes:

net.save(
    index_by=None,
    ...
)

To build an edges file with only target_node_id indices:

net.save(
    index_by=('target_node_id')
    ...
)

Or to build an edges file with all indexing options:

net.save(
    index_by=('target_node_id', 'source_node_id', 'edge_type_id')
    ...
)

File Compression#

the HDF5 Format allows for data-sets to be compressed inside the file. To save an hdf5 file with compression, use the compression option (note that the options for compression may vary depending on machine or hdf5/h5py libraries installed).

To save with GZIP compression:

net.save(
    compression='gzip'
    ...
)

To ensure no compression use the None option:

net.save(
    compression=None
    ...
)

Options for Optimization of Network Building#

Iterators#

Building with Multiple Cores#

When running on a computer or cluster with MPI installed, the BMTK Network Builder can automatically parallelize building the network. Just make sure to call the script with the appropriate command for your system (eg. mpirun, mpiexec, srun, etc).

# Split building of script across 10 cores
$ mpirun -np 10 python build_network.py

The resulting SONATA network file will be similar to the file created with a single core. There is no need for the user to change their code.

The one exception is when passing is parameters to either the add_nodes() or add_edges() methods that rely on randomized values. For example

net.add_nodes(
    N=100,
    x=np.random.normal(0.0, 100.0, 100),
    y=np.random.normal(0.0, 100.0, 100),
    z=np.random.uniform(-100.0, 100.0, 100),
    ...
)

In such cases, when not taking extra precautions, each rank/core will generate different values for each cell’s (x, y, z) coordinates, and BMTK will not know which values to save into the final file. The Network Builder will by default check if values are different across ranks and throw an error if needed.

The simplest solution is to make sure to hard-code the random seed. This will better ensure that the same random numbers are generated across all ranks.

np.random.seed(100)

net.add_nodes(
    N=100,
    x=np.random.normal(0.0, 100.0, 100),
    y=np.random.normal(0.0, 100.0, 100),
    z=np.random.uniform(-100.0, 100.0, 100),
    ...
)

Another option is to use the mpi4py Bcast function to generate values on one rank then send them to all the others.