Troubleshoot initial-state input generation#

This page applies when an initial state is generated by a preparatory input rollout (from_input). It is not a general fallback for failed training updates or invalid model states.

What the notices mean#

The existing retry policy can recover from certain input-generation interruptions. An INFO notice reports a successful retry. If retries cannot generate a new state and a previous state exists, a WARNING reports reuse:

[WARNING] Initial-state input generation was interrupted; continuing with the previous initial state. Details: /output/dpointnet-diagnostics-<unique-id>.log.

Continuing does not mean a new initial state was generated. Check whether reusing the previous state is acceptable for your experiment. It can invalidate a comparison that requires fresh initial states. If no previous state exists, the original exception is raised. Unexpected errors remain fatal.

Find the diagnostic#

Use this JSON fragment in a complete config to retain a file log:

{
  "output": {
    "output_dir": "output",
    "log_file": "log.txt",
    "log_level": "INFO"
  }
}

Configure it before constructing/running the model:

from bmtk.simulator import dpointnet

config = dpointnet.Config.from_json("config.json")
config.build_env()
rnn = dpointnet.RNN.from_config(config)

Recovery diagnostics are written beside the configured log file, or to the system temporary directory if no file log is configured. The notice gives the absolute path. The diagnostic contains the Python traceback, TensorFlow exception text, attempt count and batch size. Diagnostic-write failures are reported rather than hidden.

What to do next#

  1. Open the named diagnostic and identify the failing input generator and first exception.

  2. Check its input/component files, requested batch/shape, TensorFlow/GPU compatibility and memory. For LGN inputs, consult Visual input from an LGN model and, where relevant, Pascal GPU compatibility.

  3. Distinguish a recovered retry from previous-state reuse. Record reuse and reassess results whose interpretation depends on fresh initialization.

  4. Correct the cause before restarting the affected experiment. Do not treat a fallback notice as proof that its initialization protocol passed.

Interrupted partial LGN batches are discarded; no substitute spikes are generated. Initial-state LGN warmup uses host-reference generation. TensorFlow native notices may still appear on stderr: the diagnostic captures the handled exception and traceback, not all process-wide native output.