Network Builder Features and Options#

NetworkBuilder methods#

Iterating and Filter Nodes from a Network using nodes() method#

You can use the The NetworkBuilder.nodes() method of the NetworkBuilder class to iterate over the nodes stored in a network. The nodes() method returns an iterable list which contains dictionaries of the properties of each individual cell that will be created by each call to add_nodes()

import numpy as np
from bmtk.builder.networks import NetworkBuilder

net = NetworkBuilder('my_model')
net.add_nodes(
    N=10,
    model_type='biophysical',
    cell_order=range(10),
    coord=np.random.uniform(-100.0, 100.0, 10)
)

assert(len(net.nodes()) == 10)
for node_dict in net.nodes():
    print(node_dict['node_id'], ':', node_dict)

The above will show each 10 nodes/cells created by the add_nodes() call:

0 :  {'model_type': 'biophysical', 'node_type_id': 100, 'cell_order': 0, 'coord': -74.13859238611045, 'node_id': 0}
1 :  {'model_type': 'biophysical', 'node_type_id': 100, 'cell_order': 1, 'coord': -22.88965188502479, 'node_id': 1}
2 :  {'model_type': 'biophysical', 'node_type_id': 100, 'cell_order': 2, 'coord': 54.31170697608857, 'node_id': 2}
...

You can also filter by property values by passing in value into parameter.

from bmtk.builder.networks import NetworkBuilder
from pprint import pprint

net = NetworkBuilder('my_model')
net.add_nodes(
    N=5,
    model_type='biophysical',
    ei='exc',
    cell_order=range(5),
)
net.add_nodes(
    N=5,
    model_type='biophysical',
    ei='inh',
    cell_order=range(5),
)
net.add_nodes(
    N=5,
    model_type='point',
    ei='exc',
    cell_order=range(5),
)

net.add_nodes(
    N=5,
    model_type='point',
    cell_order=range(5),
)

pprint(list(net.nodes(ei='exc')))

The above will return only the 10 nodes from the two calls with ei=='exc'. If you want to filter by multiple properties in the format prop1==A && prop2==B && prop3==C you can just pass in multiple key-value pairs into the nodes() parameters. For example if we want to find only those cells that are excitatory point neuron cells from the example above

pprint(list(net.nodes(ei='exc', model_type='point')))

Iterating and Filtering Edges using the edges() method#

Simular to the nodes() method, the NetworkBuilder.edges() method allows you to iterate and filter all the edges/synapses generated by each call to add_edges(). When called with no parameters it will return a list of all edges/synapses in a dictionary like format.

from bmtk.builder.networks import NetworkBuilder

net = NetworkBuilder('my_model')
net.add_nodes(
    N=5,
    model_type='biophysical',
    ei='exc'
)
net.add_nodes(
    N=5,
    model_type='biophysical',
    ei='inh',
)
net.add_nodes(
    N=5,
    model_type='point',
    ei='exc',
)

net.add_edges(
    source=net.nodes(ei='inh'),
    target=net.nodes(ei='exc'),
    connection_rule=lambda *_: np.random.randint(0, 10),
    syn_function='inh2exc',
    syn_weight=-20.0
)

net.add_edges(
    source=net.nodes(ei='exc', model_type='point'),
    target=net.nodes(model_type='biophysical'),
    connection_rule=1,
    syn_function='point_exc',
    syn_weight=10.0
)

edge0 = net.edges()[0]
print(edge0['source_node_id'])
print(edge0['target_node_id'])
print(edge0['syn_function'])
print(edge0['syn_weight'])

for edge in net.edges():
    print(edge)

The following small network will display the following like the following (note that the order of edges may be different)

5
1
inh2exc
-20.0
5 --> 1 {syn_function: inh2exc, syn_weight: -20.0, edge_type_id: 100, source_query: ei=='inh', target_query: ei=='exc', source_node_id: 5, target_node_id: 1, nsyns: 4}
5 --> 2 {syn_function: inh2exc, syn_weight: -20.0, edge_type_id: 100, source_query: ei=='inh', target_query: ei=='exc', source_node_id: 5, target_node_id: 2, nsyns: 5}
5 --> 4 {syn_function: inh2exc, syn_weight: -20.0, edge_type_id: 100, source_query: ei=='inh', target_query: ei=='exc', source_node_id: 5, target_node_id: 4, nsyns: 7}

Important

Before the edges() method is called the connectivity matrix needs to be created which is equivalent to calling the build() method. For very large networks this can take a long amount of time and memory. And if you call add_edges() or add_nodes() after it will need to rebuild the network.

As such it is highly recommend to not call edge() iterator until after all calls to add_nodes() and add_edges() are completed.

The easiest way to filter only certain edges by query is, like with the nodes() method, add key-value parameters to the method. If more than one parameter is called then it will be treated as a logical “and”.

# We should expect 5x(5+5) = 50 edges in total
edges = net.edges(syn_function='point_exc', syn_weight=10.0)
print(len(edges))
for edge in edges:
    # print(edge)
    assert(edge['nsyns'] == 1)
    assert(edge['syn_weight'] > 0)

# Due to the "syn_weight" conditional we expect to find no (0) edges.
edges = net.edges(syn_function='point_exc', syn_weight=-20.0)
print(len(edges))

You can also filter edges by either source and/or target properties using the source_nodes and target_nodes parameters, respectively. Use the nodes() iterator to create filters for nodes (this will work even if a source/target set of nodes comes from a differnt population - as long as the edges still exists).

# We should expect <= 50 inh -> exc edges.
edges = net.edges(
    source_nodes=net.nodes(ei='inh'),
    target_nodes=net.nodes(ei='exc')

)
assert(len(edges) > 0)
for edge in edges:
    assert(edge['syn_function'] == 'inh2exc')
    assert(edge['syn_weight'] < 0.0)

edges = net.edges(
    source_nodes=net.nodes(ei='inh'),
    target_nodes=net.nodes(ei='exc')

)

Importing Existing Nodes#

If you have an existing SONATA network (.h5 and .csv file) you can import the nodes using the import_nodes() method.

from bmtk.builder.networks import NetworkBuilder

net = NetworkBuilder('my_model')

# expect 0 nodes
print(len(net.nodes()))

net.import_nodes(
    nodes_file_name='network/v1_nodes.h5',
    node_types_file_name='network/v1_node_types.csv'
)

# Should have more than 0 nodes
print(len(net.nodes()))

Connector Functions#

Options for setting synapse location#

When creating a network with morphologically realistic cells, the location of the afferent synapses along a given cell dendrites/soma is often very important. However trying to manually place each synapse quickly becomes an intractable problem even for small networks. Also with simulators like NEURON it is very difficult to know the name/identifier of each dendritic branch and segment prior to simulation.

BMTK Network Builder along with BioNet offers some solutions to make it easier to manage placing synaptic locations when using the add_edges() calls.

“target_sections” and “distance_range” properties#

The easiest and most efficient solution is add properties target_section and distance_range when calling the add_edges() method. target_section is a list of of one or more cell section names as defined in the SWC format (axon, somatic, basal, apical, dend, other). while distance_range is a vector of two values - the start and end distance (both inclusive) in arc-length a segment has to be away from the hilt of the soma.

For example we want to create edges that on connect to apical or basal dendritic sections that are between 50-200 microns away from the soma. (If you want to have it go to the end of the dendrite you can use a large number like [50.0, 1.0e20].

net.add_edges(
    source=net.nodes(ei='exc'),
    target=net.nodes(model_type='biophysical', morphology='pyramidal.swc'),
    ...
    distance_range=[50.0, 200.0],
    target_sections=['basal', 'apical'],
    ...
)

At the beggining of each simulation BioNet will see the target_section and distance_range parameters and use them to find every morphology segment that matches, choose one at random, and create a synapse.

There are two potential issues when using this method for assigning synapse locations

1. target_section and distance_range are not SONATA reserved keywords, meaning other tools will likely not be able to use these parameters to regenerate synaptic locations. 2. Since the synapses are generated by BioNet dynamically before each simulation, it could lead to issues with reproducability of network results.

Explicity setting afferent synapse section and position#

The more reproducable alternative is to set the parameters afferent_section_id and afferent_section_pos, which are reserve SONATA keywords recognized by other tools, for every synapse. To help with this you can use the NetworkBuilder.nodes() method. Point to this method in the add_properties() call and choose the target_section and distance_range, like above. When build() is called it will choose appropiate synapse location and store them in the SONATA file.

from bmtk.builder.bionet import rand_syn_locations

cm = net.add_edges(
    source=net.nodes(ei='exc'),
    target=net.nodes(model_type='biophysical', morphology='pyramidal.swc'),
    ...
    distance_range=[50.0, 200.0],
    target_sections=['basal', 'apical'],
    ...
)

cm.add_properties(
    ['afferent_section_id', 'afferent_section_pos'],
    rule=rand_syn_locations,
    rule_params={
        'section_names': ['somatic', 'basal'],
        'distance_range': [50.0, 200.0],
        'morphology_dir': './components/morphologies'
    },
    dtypes=[int, float]
)

When build() is called on the previous code it will load the ‘pyramidal.swc’ file with NEURON to determine how the simulator will generate and name all the branches and segments. If you prefer you can use the SWCReader class inside your script to access information about availabe sections and segments (including x, y, z coordinates):

from bmtk.builder.bionet.swc_reader import SWCReader

morph = SWCReader('./components/morphologies/pyramidal.swc')
morph.set_segment_dl(dL)
morph.move_and_rotate(
    soma_coords=[x, y, z],
    rotation_angles=[rotation_angle_xaxis, rotation_angle_yaxis, rotation_angle_zaxis],
)

Built-in connection rules#

The BMTK Network Builder includes a number of built-in functions that you can use for the connection_rule option when using add_edges() method.

bmtk.builder.auxi.edge_connector.distance_connector() - Creates connections between source and target cells based on distance, cells that are closer together will more likely to have more connections between them (will rule-out self-connections).

bmtk.builder.auxi.edge_connector.connect_random() - Will choose at random if two cells share a connection or not.

Options for Saving Network to SONATA#

network file name and path#

By default when you when have a population network with name “NETNAME” and call NetworkBuilder.save() or NetworkBuilder.save_nodes() the Builder will by default save it to file names NETNAME_nodes.h5 and NETNAME_nodes.csv. Similarly for edges from populations “NETSRC” to “NETTRG” will be saved by default under file names NETSRC_to_NETTRG_edges.h5 and NETSRC_to_NETTRG_edge_types.csv.

To change the file names use the node[_type]s_file_name and edge[_types]_file_name parameters. You can use absolute file paths, and when using relative file paths it will place it under the output_dir option

# will save the network nodes and edges files into "network/custom-*" file-name.
net.save(
    output_dir='./network/',
    nodes_file_name='custom-nodes.h5'
    node_types_file_name='custom-node-types.csv',
    edges_file_name='custom-edges.h5',
    edge_types_file_name='custom-edge-types.csv',
)

append mode#

When the NetworkBuilder.save() method is called it will by default overwrite any existing SONATA h5 and csv files if they already exist (you can change this behavior by setting force_overwrite=False). The SONATA format does allow you to save multiple nodes and edges populations in the save files - although each population must have a unique name., With BMTK you must set the mode parameter to a or append.

# Assumes network/shared-nodes.h5 already exists and user wants to append another node population into it.
net.save(
    output_dir='./network/',
    mode='a',
    nodes_file_name='shared-nodes.h5'
)

Sorting of Edges#

By default when the NetworkBuilder saves the SONATA edges file the edges are sorted by “target_node_id” (ie. all the edges with post-synapse node 0 comes before edges with post-synapse node 1). The sort_by parameter in the save() method allows you to choose between sorting by

  • “target_node_id” - Group and sort by post-synaptic node-id

  • “source_node_id” - Group and sort by pre-synaptic node-id

  • “edge_type_id” - Will group edges that share the same edge-type properties and id.

net.save(
    output_dir='./network/',
    sort_by='source_node_id',
)

Edges Indexing#

SONATA edges formats allows the addition of indices which can be used to quickly search for edges based on either the target_node_id, source_node_id, or edge_type_id. By default BMTK Network Builder will create indices for both target_node_id and source_node_id. This can be changed using the index_by parameter.

To build an edges file with No indexes

net.save(
    index_by=None,
    ...
)

To build an edges file with only target_node_id indices

net.save(
    index_by=('target_node_id')
    ...
)

Or to build an edges file with all indexing options

net.save(
    index_by=('target_node_id', 'source_node_id', 'edge_type_id')
    ...
)

File Compression#

the HDF5 Format allows for data-sets to be compressed inside the file. To save an hdf5 file with a compression use the compression option (note that the options for compression may vary depending on machine on hdf5/h5py libraries installed).

To save with GZIP compression

net.save(
    compression='gzip'
    ...
)

To ensure no compression use the None option.

net.save(
    compression=None
    ...
)

Options for Optimization of Network Building#

iterators#

Building with multiple cores#

When running on a computer or cluster with MPI installed the BMTK Network Builder can automatically parallelize the bulding of the network. Just make sure to call the script with the approiate command for your system (eg. mpirun, mpiexec, srun, etc).

# Split building of script across 10 cores
$ mpirun -np 10 python build_network.py

The resulting SONATA network file will be similar to as if run in single core python. There is no need for the user to change their code.

The one exception is when passing is parameters to either the add_nodes() or add_edges() methods that rely on randomized values. For example

net.add_nodes(
    N=100,
    x=np.random.normal(0.0, 100.0, 100),
    y=np.random.normal(0.0, 100.0, 100),
    z=np.random.uniform(-100.0, 100.0, 100),
    ...
)

In such cases, when not taking extra precautions, each rank/core will generate different values for each cell’s (x, y, z) coordinates, and BMTK will not know which values to save into the final file. The Network Builder will by default check if values are different across ranks and throw an error if needed.

The simplest solution is to make sure to hard-code the random seed. This will better ensure that each take a random numbers are generated they produce the same values across all ranks.

np.random.seed(100)

net.add_nodes(
    N=100,
    x=np.random.normal(0.0, 100.0, 100),
    y=np.random.normal(0.0, 100.0, 100),
    z=np.random.uniform(-100.0, 100.0, 100),
    ...
)

Another option is to use the mpi4py Bcast function to generate values to generate inputs on one rank then send them to all the others.