Dry Lab / Computational: out of memory (oom)

1. Ranked causes

   1. Requested memory allocation too low for the input sizeExit code 137 / OOM-killer entry in dmesg or scheduler log.
   2. Input dataset much larger than expected (wrong or unfiltered file set)File size / read count far above the protocol's expected footprint.
   3. Memory leak or unbounded in-memory accumulation in the scriptMemory usage climbs steadily over the run instead of plateauing.
   4. Batch size too large for available memory per workerOOM occurs at a predictable point tied to batch index.
   5. Too many parallel threads/workers each holding a full copy of the datasetMemory use scales roughly linearly with the thread count.

2. Recovery steps

   1. Increase the Slurm memory flag (e.g. --mem=32G → --mem=64G)The single fastest fix when the job is simply under-provisioned.
   2. Reduce batch size and re-runLowers peak memory per iteration at the cost of some throughput.
   3. Verify input file sizes against the protocol's expected datasetCatches a wrong or unfiltered material input before more memory is requested.
   4. Reduce thread/worker count so fewer dataset copies are held in memory simultaneouslyTrades wall-clock time for memory headroom.

3. Controls first

   • Did a smoke-test / small control input complete without OOM on the same allocation?If yes: Input size or batching — not a broken scheduler allocation for tiny jobs.
   • Did the job exit with code 137 or an explicit OOM-killer log entry?If yes: Confirmed memory kill — increase allocation or reduce batch/thread footprint.

Diagnose your run

Opens Diagnostics capture with this first-party playbook behind it. Sign in if the workspace asks.