TEM Architecture
TEM Architecture

TEM Architecture

Welcome to the documentation for the Allen Institute next generation TEM data acquisition system. This is an event driven system with various components running in docker containers, along with other services running on other machines. Below, is a diagram showing a general overview of this system.

%3 cluster_AWS AWS cluster_Docker Compose Docker Compose cluster_Platform9 Platform9 cluster_Lab Server Lab Server d1bd99b9c20241eba67e1aa5b1229d22 Database f0ae9d2d68884519b038cd56409b8036 S3 Bucket 0937e979909948d2855d1cd56ebd4367 AC/QC f0ae9d2d68884519b038cd56409b8036->0937e979909948d2855d1cd56ebd4367 fd813ba53a384ab18f4a8034056c2be2 Aloha f0ae9d2d68884519b038cd56409b8036->fd813ba53a384ab18f4a8034056c2be2 0f822716c4a745c38d6fea40cc50de80 TEM DB 0f822716c4a745c38d6fea40cc50de80->d1bd99b9c20241eba67e1aa5b1229d22 0f822716c4a745c38d6fea40cc50de80->0937e979909948d2855d1cd56ebd4367 52806287e1d048f9aecf5cb5b1278bbc pyTEM 0f822716c4a745c38d6fea40cc50de80->52806287e1d048f9aecf5cb5b1278bbc fd813ba53a384ab18f4a8034056c2be2->0f822716c4a745c38d6fea40cc50de80 0efd7dfa1bce436a8e30b45a917b8190 AC/QC user 0efd7dfa1bce436a8e30b45a917b8190->0937e979909948d2855d1cd56ebd4367 8f27ffb7b42a4d8d9be6e6fe385f9a79 Microscope Operator 4f66c07042f04ef5b62df7e8eb7a88bb UI Server 8f27ffb7b42a4d8d9be6e6fe385f9a79->4f66c07042f04ef5b62df7e8eb7a88bb fc873b38577240bc89e25496efa7f70b ActiveMQ 3f3de44340494497be4d79eb7b072d80 Microscope Service 52806287e1d048f9aecf5cb5b1278bbc->3f3de44340494497be4d79eb7b072d80 8cd03a291cfa49bdbcb629018da7e644 Camera Service 52806287e1d048f9aecf5cb5b1278bbc->8cd03a291cfa49bdbcb629018da7e644 6e60a832c3a34c55939968da4dd988c6 Stage Service 52806287e1d048f9aecf5cb5b1278bbc->6e60a832c3a34c55939968da4dd988c6 e8ef28d035ed4361ad05c20f0dfdcf02 Image Processing Pipeline 52806287e1d048f9aecf5cb5b1278bbc->e8ef28d035ed4361ad05c20f0dfdcf02 52806287e1d048f9aecf5cb5b1278bbc->4f66c07042f04ef5b62df7e8eb7a88bb 5818e95d20894353a263dac6f5a8631f Buffer Service e8ef28d035ed4361ad05c20f0dfdcf02->5818e95d20894353a263dac6f5a8631f e8ef28d035ed4361ad05c20f0dfdcf02->4f66c07042f04ef5b62df7e8eb7a88bb df49aa25e1d84d3bafbdaec3b84cdbb8 QC Service e8ef28d035ed4361ad05c20f0dfdcf02->df49aa25e1d84d3bafbdaec3b84cdbb8 5818e95d20894353a263dac6f5a8631f->fd813ba53a384ab18f4a8034056c2be2 df49aa25e1d84d3bafbdaec3b84cdbb8->52806287e1d048f9aecf5cb5b1278bbc 2e72dd9314ff460fbae89ffe652348fd Microscope 2e72dd9314ff460fbae89ffe652348fd->3f3de44340494497be4d79eb7b072d80 19accdc183a147f29db11b032fee4b99 Stage 19accdc183a147f29db11b032fee4b99->6e60a832c3a34c55939968da4dd988c6 4b1b6653d7dd4604992c027224998e8f Camera 4b1b6653d7dd4604992c027224998e8f->8cd03a291cfa49bdbcb629018da7e644 a05bc0eba23a4b8dbf6fbfdd2d461a56 Log Server 4c705378887f4e4d9d8ae1ce91d0ec98 Container Registry f73ccaaabf2e4b02aa649ff67f9d90ab GitHub Actions Runner

Most of the nodes in this diagram are hyperlinks to further documentation of this system.

This system primarily sends messages between services using the STOMP protocul with the Pigeon library. The configuration for this system as a whole is stored on GitHub at AllenInstitute/TEM_config. This repository includes both a Docker compose file for starting all the services that are part of the system, and the configuration files for the services.