Compute-node safety conventions (L1–L7)
reproduce.sh runs on a fresh compute node inside the SLURM allocation, not on the login node where the agent generated it. The following conventions (L1–L7) ensure the script can actually complete there.
The PaperBench replicator agent is prompted with these conventions via ari-skill-paper-re/src/prompts/replicator.md (look for the COMPUTE-NODE EXECUTION CONVENTIONS block). They are reproduced here so you can audit a generated reproduce.sh by hand.
L1 — Shared filesystem
All paths in reproduce.sh must resolve on every allocated node.
- ✅
$HOME,/work/...,/scratch/...,/lustre/...,/nfs/... - ❌
/tmp,/var/tmp,/local, container-local mount-only paths
ARI warns when the checkpoint dir is on a node-local FS, but it does not refuse to run — your job will silently fail on multi-node when ranks 1+ cannot see ranks 0's files.
L2 — MPI invocation: prefer srun over mpirun
# Preferred (uses SLURM's PMI/PMIx integration; works without OpenMPI
# being separately installed):
srun -n $SLURM_NTASKS ./my_program
# Acceptable fallback (only when OpenMPI/MPICH is loaded as a module):
mpirun -np $SLURM_NTASKS ./my_program
# Last-resort Python fallback (when neither srun nor mpirun is on PATH):
pip install --user mpi4py
python -c "from mpi4py import MPI; ..."Test with which srun mpirun first. The agent prompt instructs the replicator to emit this check.
L3 — GRES probe
When the rubric's execution_profile.gpu_type is set, ARI checks sinfo -o '%G' before adding --gres=gpu:<type>:N to sbatch. If GRES is unconfigured ("(null)"), the flag is dropped and a warning is logged — --gpus-per-task survives.
You can verify the probe interactively:
from ari_skill_paper_re.server import _slurm_has_gres
_slurm_has_gres() # True / FalseL4 — Conda / virtualenv activation
#!/usr/bin/env bash does not activate any Python environment. If your reproduce.sh needs a specific env, prepend:
# Option A: source ~/.bashrc (cluster default Python env)
source ~/.bashrc
# Option B: explicit conda activate
source ~/miniconda3/etc/profile.d/conda.sh
conda activate ari-repro
# Option C: rely on /usr/bin/python3 + user-site installs
pip install --user numpy matplotlib mpi4pyThe agent prompt nudges the replicator to pick one of these.
L5 — Module loads
Site-specific modules (CUDA, OpenMPI, compilers, mathlibs) must be loaded inside reproduce.sh, not just on the login node. Use the rubric's execution_profile.module_loads to specify them:
"module_loads": ["cuda/12.4", "openmpi/4.1", "gcc/11.3"]The agent then emits:
module load cuda/12.4 openmpi/4.1 gcc/11.3at the top of reproduce.sh. ARI does not validate the module names — your cluster's module avail is authoritative.
L6 — Multi-node fan-out
reproduce.sh starts as one rank on the first allocated node. To use every allocated node, the script must fan out:
srun -N $SLURM_JOB_NUM_NODES -n $SLURM_NTASKS ./my_programWithout this, your script uses 1 node regardless of allocation size, even if sbatch --nodes=4 was successful. The agent prompt includes a dedicated "MULTI-NODE FAN-OUT" section to remind the replicator.
L7 — Timeout wrapping
SLURM --time enforces a hard wallclock and SIGTERMs jobs at the limit. For partial-result safety, wrap long stages in timeout:
timeout 1800 python long_step.py # 30-min per-step ceiling
timeout 600 ./bench # 10-min benchmark capThis ensures one slow step does not eat the entire allocation. The agent prompt encourages but does not enforce this.
Verifying a generated reproduce.sh
A quick checklist before launching:
# 1. No node-local paths
grep -E '/(tmp|var/tmp|local)/' repro_sandbox/reproduce.sh && echo BAD
# 2. srun OR mpirun (not bare ./program)
grep -E 'srun|mpirun' repro_sandbox/reproduce.sh || echo MISSING_FANOUT
# 3. Module loads when execution_profile demanded them
grep -E '^module load' repro_sandbox/reproduce.sh
# 4. Timeout wrappers around long steps
grep -E 'timeout' repro_sandbox/reproduce.sh