Build and Deployment Protocol

SEA-AD Multiregion Gene Expression Viewer

Protocol overview

This protocol describes the construction and deployment of the SEA-AD Multiregion Gene Expression Viewer from regional single-nucleus RNA-sequencing count matrices and associated ABC Atlas metadata. The procedure generates library-by-Supertype pseudobulk data, prepares a gene-partitioned AWS back end, deploys an R Shiny application, and publishes its documentation.

Complete the sections in order. Run the named scripts from the project repository; select the inputs and outputs specified below. Directory trees describe file organization, not executable commands.

Go to the GitHub source · Return to the user guide

Starting materials

A shinyapps.io account is created during the protocol. Donor-level continuous pseudoprogression scores (CPS) are downloaded as a separate published metadata resource; they are not calculated from the RNA count matrices.

Products

1. Establish the project configuration

  1. Fork the application repository into the GitHub account that will maintain the new deployment. Record the source commit identifier. Keep the Shiny deployment workflow disabled until the new application has been created and its deployment target configured.
  2. Create a project configuration record in the repository. Record the ABC Atlas manifest, included regions, source commit, Capsule identifier, S3 bucket, AWS region, S3 prefix, shinyapps.io account name, application name, and application ID. Populate the hosting fields when they are assigned.
  3. Use releases/20260711/manifest.json for the ABC Atlas release. Include AnG, DFC, FI, HIP, ITG, LEC, MEC, MTG, STG, and V1C from SEA-AD-Multiregion-10X, with annotations from SEA-AD-Multiregion-taxonomy. Keep expression and taxonomy inputs tied to this release.
  4. Use SEAAD_multiregion/v1 as the deployment prefix in the new S3 bucket and multiregion_gene_exploration as the new Shiny application name. Use the operator’s own bucket and hosting account throughout the protocol.
Critical step. The S3 prefix is the directory containing counts_by_gene/, metadata/, and statistics/. Use the same prefix in the upload destination and application configuration. Do not reuse an application ID or account name from another deployment.

2. Create the Code Ocean environment

  1. Connect the GitHub account to Code Ocean using its Git-provider integration. Select New Capsule → Clone from Git and enter the HTTPS address of the fork. Confirm that code/, environment/, and metadata/ are present.
  2. Build the Capsule environment using environment/Dockerfile, environment/postInstall, and requirements.txt. Launch a Cloud Workstation with RStudio and a terminal. Retain the environment definitions in version control.
  3. Verify the Python dependencies abc_atlas_access, anndata, h5py, numpy, pandas, scipy, pyarrow, and pybiomart. Install missing dependencies in the Capsule environment and rebuild it.
  4. Restore the R environment from code/shiny/renv.lock. Verify Shiny, bslib, Arrow with S3 support, dplyr, ggplot2, scales, shinycssloaders, cachem, DT, and jsonlite. Install rsconnect for publishing. Install Node.js and the kiwi-schema package for ABC Atlas URL generation.
  5. Allocate writable storage for the ABC Atlas cache and regional intermediates. Set the cache directory to /scratch/abc_atlas_cache, the intermediate directory to /scratch/seaad_pseudobulk, and the durable output directory to /results. Select compute resources appropriate for the regional input matrices and monitor storage and memory during processing.
  6. Keep credentials in account integrations or secret storage. Exclude AWS credentials, shinyapps.io tokens, saved R workspaces, raw matrices, and local package libraries from Git commits and application bundles.
Checkpoint. Start a clean R session and Python process. Confirm that the required packages load, the scratch directories are writable, and Arrow can initialize an S3 filesystem.

3. Prepare the source data

  1. Place the ABC Atlas Access files in the cache directory, retaining the manifest-defined directory structure and filenames. Configure ABC Atlas Access to use that cache and the selected manifest.
  2. Confirm that every selected region has a raw-count H5AD matrix. Retain the nucleus-level metadata, library metadata, donor metadata, and taxonomy membership and annotation tables required by code/create_seaad_pseudobulk.py. Do not substitute log-normalized expression matrices.
  3. Download global_cps_multiregion.csv and retain it as a local processing input. Record its source and checksum. Verify the columns donor_id, CPS_Global, CPS_Global_ABeta, and CPS_Global_pTau.
  4. Check that CPS donor identifiers are nonmissing and unique and that score columns are numeric. Compare the identifiers with donor metadata as strings. Preserve missing scores as missing values.
Critical step. CPS is an independent study-derived measurement. Do not replace missing scores with zero or estimate them from the expression matrices. Obtain the published CPS resource before proceeding with the complete trajectory-enabled application.

4. Generate the pseudobulk H5AD

  1. Run code/create_seaad_pseudobulk.py with the selected manifest, all ten regions, the cache and scratch directories defined above, and output /results/SEA_AD_Multiregion_library_supertype_pseudobulk.h5ad. Use a matrix chunk size of 1,024 observations and a metadata chunk size of 250,000 rows.
  2. Process each regional matrix independently. Join nucleus identifiers to library and donor metadata and to the matching Class, Subclass, and Supertype annotations. Normalize identifier types before joining. Review the processing log for unmatched or excluded nuclei.
  3. Aggregate assigned nuclei by library_label × Supertype. For each gene, sum raw counts and count nuclei with raw expression greater than zero. Record the total number of contributing nuclei for each pseudobulk observation.
  4. Combine the regional pseudobulk objects after confirming a consistent gene axis. Reopen the final H5AD and verify the matrix and metadata contract below.
  5. Capture the final H5AD and processing log as a versioned Code Ocean Data Asset. Record the source manifest, source commit, region list, and checksum. Attach the asset to the Capsule at /data/pseudobulk for subsequent steps.

Required H5AD structure

Checkpoint. Counts are nonnegative integers. Nucleus totals are positive. Expressing-nucleus counts lie between zero and the corresponding nucleus total. Observation identifiers are unique, and the gene order is identical in the count and detection matrices.

5. Generate gene-partitioned Parquet files

  1. Run code/create_seaad_parquet.py. Set the input to /data/pseudobulk/SEA_AD_Multiregion_library_supertype_pseudobulk.h5ad, the output to /results/SEAAD_parquet_by_gene, and the CPS source to the local CSV downloaded in Step 13. Use a gene block size of 64 and Zstandard compression.
  2. Write metadata/obs_metadata.parquet with one unique sample_id per pseudobulk observation. Preserve the observation order and its library_label. Write metadata/gene_map.parquet with the mapping from gene_symbol to gene_column.
  3. Calculate each pseudobulk row’s total raw counts across all genes. Write metadata/CPM_scaling.csv with library_label and CPM_scaling_factor, retaining exactly the observation-metadata row order.
  4. Write one counts_by_gene/gene=<gene_column>/data.parquet file per gene. Retain sample_id, summed_counts, and number_of_nuclei_expressing, including zero-valued observations. Use the partition keys generated by the converter.
  5. Confirm that the converter copied the CPS CSV to metadata/global_cps_multiregion.csv and wrote its data manifest. Validate representative gene files against the H5AD before proceeding.

Expression and detection calculations

CPM_scaling_factor = 1,000,000 / total raw counts in the pseudobulk row
CP10K = summed_counts × CPM_scaling_factor / 100
Displayed log expression = ln(CP10K + 1)
Fraction expressing = number_of_nuclei_expressing / Number of nuclei
Critical step. Expression normalization uses the total raw counts, not the number of nuclei. Detection fractions use nucleus totals. The converter generates CPM scaling as part of this step; do not calculate a second scaling table from selected genes.
Checkpoint. Every gene-map key resolves to a partition. Gene files have the expected sample identifiers. The scaling table and observation metadata have identical row counts and library_label sequences.

6. Generate statistics and categorical metadata

  1. Run code/create_seaad_statistics.py with the mounted pseudobulk H5AD as input and /results/SEAAD_parquet_by_gene/statistics/seaad_gene_summary_statistics.csv as output. Use the same count and detection matrices used for Parquet conversion.
  2. Calculate CPM-based expression summaries for the complete dataset, Dementia versus No dementia groups, Brain Region, and Subclass. Calculate detection fractions from pooled expressing-nucleus counts divided by pooled nucleus totals. Retain the script’s fold-change and fraction-difference definitions.
  3. Annotate gene biotypes using the script’s human BioMart lookup. Retain the gene-to-biotype mapping with the processing record and verify that each annotation is aligned to its gene symbol. Validate the required output columns below.
  4. Run code/create_seaad_value_sets.py with the same pseudobulk H5AD and output /results/SEAAD_parquet_by_gene/metadata/value_sets.csv. Use the columns label, field, color_hex_triplet, and order.
  5. Join the Class, Subclass, and Supertype labels in the value-set table to the matching ABC Atlas taxonomy annotation tables. Populate their taxonomy colors and order from that release. Preserve the script-generated clinical category order and colors for other fields. Exclude continuous variables from categorical value sets.
  6. Validate the CPS table against obs_metadata.parquet. Join donor_id to Donor ID without changing observation order or duplicating observations. Retain unmatched scores as missing values and record their frequency.

Required statistics columns

gene_symbol
ln(mean_CPM+1)
frac_expr
dementia_log2fc
dementia_frac_diff
max_ROI
log2fc_ROI
max_frac_ROI
frac_diff_ROI
max_subclass
log2fc_subclass
max_frac_subclass
frac_diff_subclass
gene_type
Critical step. Keep ln(mean_CPM+1) in the source CSV. The application converts this summary to CP10K for display. Keep fraction differences as proportions, without multiplying them by 100.
Checkpoint. The statistics file has the required schema and gene coverage. Value-set field/label pairs are unique and match observation metadata. The three CPS columns remain numeric.

7. Publish the data to AWS

  1. Create a new S3 bucket in us-west-2 with a globally unique name. Dedicate it to the public application data. Record the bucket name and SEAAD_multiregion/v1 prefix in the project configuration.
  2. Create an upload identity with permissions limited to the application bucket and required upload operations. Configure authenticated AWS access in Code Ocean using secret storage. Do not embed credentials in the application.
  3. Assemble the complete directory shown below. Validate filenames, partition coverage, sample alignment, detection-count bounds, CPS joins, and statistics before upload. Capture this directory as a versioned Code Ocean Data Asset.
  4. Upload the contents of /results/SEAAD_parquet_by_gene directly beneath SEAAD_multiregion/v1. Preserve the directory hierarchy. Compare local and remote object counts and sizes and read back representative files.
  5. Configure a bucket policy granting anonymous s3:GetObject access only to objects under the application prefix. Grant s3:ListBucket for the application-prefix listings required by Arrow dataset discovery. Do not grant anonymous write or delete permissions. Adjust Block Public Access settings only as required for this public-data policy.
  6. Test access without AWS credentials. Read each metadata file and the statistics CSV, then list and open a representative gene partition using Arrow. Keep private data outside this bucket.

Local output and S3 prefix contents

SEAAD_parquet_by_gene/
  counts_by_gene/
    gene=<gene_column>/
      data.parquet
    ...
  metadata/
    obs_metadata.parquet
    gene_map.parquet
    CPM_scaling.csv
    value_sets.csv
    global_cps_multiregion.csv
  statistics/
    seaad_gene_summary_statistics.csv
  manifest.json
Critical step. Public read access exposes the selected objects to anyone on the internet. Publish only the study data and metadata approved for public distribution. The deployed app uses anonymous S3 reads; upload credentials are not part of its runtime bundle.
Checkpoint. All startup resources and representative gene partitions can be read anonymously from the new bucket and prefix.
  1. Open ABC Atlas and select the SEA-AD multiregion dataset. Select PTPRG as the displayed gene and configure the intended starting view. Copy its complete share URL and record that URL together with the starting gene PTPRG.
  2. Load code/shiny/abc_atlas_gene_url_function.R in the preparation environment. Supply abc_atlas_schema_kiwi.txt, the recorded starting URL, the starting gene, and the gene-symbol list from gene_map.parquet. Generate a URL for each gene using the helper’s gene-substitution procedure.
  3. Write a gzip-compressed CSV to code/shiny/abc_atlas_gene_urls.csv.gz. Put the gene symbol in the first column and its URL in the second column. Include a header and omit row names or index columns.
  4. Check coverage against the gene map and remove duplicate symbol entries. Open links for PTPRG, APOE, and another gene and confirm that each opens the intended gene and SEA-AD view. Retain the seed URL, starting gene, schema, and generation settings with the release record.
Checkpoint. The compressed lookup has two columns, nonempty URLs, and entries for the app’s gene symbols. Node.js is used for preparation; the deployed app reads the CSV lookup.

9. Configure and test the application

  1. Open code/shiny/app.R. Set its S3 bucket, AWS region, and data prefix to the values recorded in Step 31. Confirm that its metadata, scaling, CPS, statistics, and gene-partition paths resolve beneath that prefix.
  2. Assemble the application directory shown below. Keep table_column_definitions.csv aligned to the statistics schema, with columns column_names and column_definitions. Include the generated ABC Atlas lookup and the static assets.
  3. Start a clean R session in code/shiny and launch the application. Verify startup without a saved workspace. Confirm that no scientific-data path points to a local scratch directory or mounted Data Asset.
  4. Load PTPRG and APOE and compare selected values with the generated Parquet files. Verify CP10K, log expression, detection fractions, and zero-valued observations. Test gene-table sorting, filtering, and selection.
  5. Test trajectory, heatmap, dot, distribution, and two-gene correlation plots. Confirm Subclass and Brain Region defaults for heatmap and dot plots, pooled detection fractions for dot sizes, and continuous numeric CPS axes for trajectories. Verify that missing CPS values are excluded from the selected trajectory axis rather than placed at zero.
  6. Test combined metadata filters, facets, plot sizing, distribution pagination, expression limits, comparison genes, and external links. Create bookmarks for a filtered trajectory, a faceted distribution, and a two-gene plot; reopen each in a fresh session and verify restoration.

Application directory

code/shiny/
  app.R
  table_column_definitions.csv
  abc_atlas_gene_urls.csv.gz
  renv.lock
  manifest.json
  www/
    app_logo.png
    allen_institute_logo.svg
    tokens.csv
Checkpoint. The app starts from a clean session, reads the new S3 back end anonymously, and passes the plot and bookmark checks. Record the tested source commit and package environment.

10. Create and deploy the hosted application

  1. Create a shinyapps.io account using the GitHub sign-in option. Choose an available account name and record it. Select a hosting plan and application resources appropriate for the tested app.
  2. Open the account’s token settings and obtain a publishing token and secret. Configure rsconnect in the Code Ocean R environment using that account name and credentials. Store credentials outside the repository.
  3. Generate code/shiny/manifest.json from the tested R environment. Include app.R, table definitions, the compressed ABC Atlas lookup, and required www/ files. Verify package versions and public package sources. Exclude local libraries, credentials, raw data, saved workspaces, and development-only activation files.
  4. Deploy code/shiny with rsconnect to the new account using application name multiregion_gene_exploration. Create a new application for this first deployment; do not supply an application ID belonging to another account.
  5. Record the assigned application ID and hosted URL. Review the build and runtime logs. Open the hosted app in a fresh browser session and repeat the startup, gene retrieval, plot, external-link, and bookmark checks.
  6. Commit the tested application files, renv.lock, and deployment manifest. Keep the data-generation manifest in the data output directory separate from the Shiny deployment manifest.
Checkpoint. The hosted application reads the new S3 location and reproduces the validated local views. The release record contains its account name, app name, app ID, URL, and source commit.

11. Configure automated deployment

  1. Open .github/workflows/deploy.yml in the fork. Set the publishing account, application name, and application ID to the values assigned in Section 10. Retain code/shiny as the application directory.
  2. Add SHINYAPPS_TOKEN and SHINYAPPS_SECRET as GitHub Actions repository secrets. Use credentials authorized to publish to the new account.
  3. Configure the workflow to publish using the committed deployment manifest. Validate its Shiny application mode, package records, and file inventory. Refresh bundled-file checksums while preserving the tested package metadata. Exclude the local .Rprofile from the runtime bundle.
  4. Validate YAML and embedded R syntax. Enable manual dispatch and deployment on pushes to main affecting code/shiny/** or the deployment workflow. Enable the workflow and run a manual deployment.
  5. Review the Actions log and verify that the existing new application was updated rather than duplicated. Repeat the hosted startup and bookmark checks. Commit future package changes together with an updated lockfile and regenerated deployment manifest.
Checkpoint. A manual GitHub Actions run successfully updates the application created in Section 10. No production account identifiers or credentials from another deployment remain in the workflow.

12. Publish the documentation and archive the release

  1. Place index.html and deployment_documentation.html in the repository root. Keep relative links between the two pages. Retain the image paths code/shiny/www/app_logo.png and code/shiny/www/allen_institute_logo.svg.
  2. Update repository links to the fork and application launch links to the new hosted URL. In each example bookmark, replace only the application base address; preserve the complete encoded viewer_state query string.
  3. In GitHub repository settings, open Pages. Select Deploy from a branch, choose main and the repository root, and save. Open the published site and test navigation, images, application links, and every Getting started example.
  4. Archive the source commit, environment definitions, lockfile, deployment manifest, ABC Atlas manifest, input checksums, CPS resource, URL-generation seed, Data Asset versions, S3 location, and validation record. Record the hosted app URL and documentation URL.
Checkpoint. The public user guide and protocol are accessible, all example links target the new app, and the release record identifies every input and deployment component.