This guide focuses on:

Setting up the runtime environment
Submitting benchmark jobs on Slurm

Prerequisites

Linux login node with Slurm access
Access to H200 partition (hpcai in current scripts)
Model and dataset paths available on shared storage
Workspace locates at /home/pengzhi/deepseek-r1-0528

Workspace Layout (Required Scripts)

The workflow uses these scripts in repository root:

env.sh (runtime paths and CUDA/NCCL settings)
submit.sh (single-node TP jobs)
submit-pp.sh (dual-node PP jobs)
submit-2ntp.sh (dual-node TP16 jobs)
bench.sh, bench-pp.sh, bench-2ntp.sh (Slurm execution entrypoints)

Optional helper:

bench-hw-diag.sh (hardware diagnostics)

Environment Setup

Install / Verify `sglang` in Runtime Python

The benchmark scripts call $PYTHON -m sglang.bench_offline_throughput, so sglang must exist in the same interpreter configured in env.sh.

cd /home/pengzhi/deepseek-r1-0528
source ./env.sh

$PYTHON -m pip show sglang || $PYTHON -m pip install -U sglang
$PYTHON - <<'PY'
import sglang
print("sglang:", sglang.__version__)
PY
$PYTHON -m sglang.bench_offline_throughput --help >/dev/null && echo "bench entrypoint ok"

If your cluster already provides a prebuilt runtime Python at $PYTHON, this step is only a verification step.

Update `env.sh`

Edit env.sh and verify these fields match your cluster:

P=/scratch/public
PYTHON=$P/home/pengzhi/python313/bin/python
MODEL=$P/models/deepseek-ai/DeepSeek-R1-0528
DATASET=$P/datasets/ShareGPT_V3_unfiltered_cleaned_split.json

export CUDA_HOME=$P/nvidia/cuda/cuda-13.2
export CUDACXX=$CUDA_HOME/bin/nvcc
export LD_PRELOAD=$P/nvidia/nccl/libnccl-2.29.7-1+cuda13.2.x86_64/usr/lib64/libnccl.so

Validate Environment

cd /home/pengzhi/deepseek-r1-0528
source ./env.sh

echo "$PYTHON"
test -x "$PYTHON" && echo "python ok"
test -f "$MODEL/config.json" && echo "model ok"
test -f "$DATASET" && echo "dataset ok"

nvidia-smi

(Optional) Run Hardware Diagnostic Job

sbatch bench-hw-diag.sh

Log path:

/home/pengzhi/deepseek-r1-0528/logs/hw_diag_<jobid>.out

Submit Your First TP8 Job

Use submit.sh for single-node TP jobs.

TP8 Smoke Run

cd /home/pengzhi/deepseek-r1-0528

NUM_PROMPTS=100 \
TAG=smoke-tp8 \
TIME_LIMIT=00:30:00 \
EXTRA_ARGS="--load-format dummy --mem-fraction-static 0.87" \
./submit.sh

TP8 Champion-Like Run

NUM_PROMPTS=7000 \
TAG=tp8-champ \
TIME_LIMIT=00:30:00 \
EXTRA_ARGS="--load-format dummy \
  --mem-fraction-static 0.87 \
  --schedule-policy lof \
  --moe-runner-backend cutlass \
  --enable-piecewise-cuda-graph \
  --decode-attention-backend flashinfer \
  --enable-flashinfer-allreduce-fusion \
  --chunked-prefill-size 32768 \
  --cuda-graph-max-bs 2048 \
  --disable-radix-cache" \
./submit.sh

Submit PP2 Dual-Node Job

Use submit-pp.sh for PP jobs (PP=2 by default).

NUM_PROMPTS=7000 \
TAG=pp2-champ \
TIME_LIMIT=00:30:00 \
PP=2 \
PP_LAYER_PARTITION=32-29 \
EXTRA_ARGS="--load-format dummy \
  --mem-fraction-static 0.83 \
  --disable-radix-cache \
  --chunked-prefill-size 32768" \
./submit-pp.sh

Notes:

submit-pp.sh forwards PP_LAYER_PARTITION to bench-pp.sh.
bench-pp.sh converts 32-29 to SGLANG_PP_LAYER_PARTITION=32,29.

Submit TP16 Dual-Node Job

Use submit-2ntp.sh for pure dual-node TP jobs.

NUM_PROMPTS=8000 \
TAG=tp16-ref \
TIME_LIMIT=00:30:00 \
TP_SIZE=16 \
EXTRA_ARGS="--load-format dummy \
  --mem-fraction-static 0.87 \
  --schedule-policy lof \
  --moe-runner-backend cutlass \
  --decode-attention-backend flashinfer \
  --chunked-prefill-size 32768" \
./submit-2ntp.sh

Monitor Jobs and Check Results

Queue Monitoring

squeue -u pengzhi

Log Tail

submit*.sh prints a tail command. Log naming pattern:

TP8: logs/bench_tp*_n*_<tag>_<jobid>.out
PP2: logs/pp_pp*_tp*_n*_<tag>_<jobid>.out
TP16: logs/tp16_tp*_<tag>_<jobid>.out

Example:

tail -f /home/pengzhi/deepseek-r1-0528/logs/bench_tp8_n7000_tp8-champ_<jobid>.out

Result JSON

Each benchmark writes a JSON file under logs/, for example:

/home/pengzhi/deepseek-r1-0528/logs/tp8_n7000_tp8-champ_<jobid>.json

Quick Troubleshooting

Symptom	Likely Cause	Action

Symptom	Likely Cause	Action
OOM	`mem-fraction-static` too aggressive or stack too heavy	Lower `mf`, reduce stack complexity first
Job killed by Slurm	Time limit too short	Increase `TIME_LIMIT` in submission
Assertion failure	Unsupported backend combination	Roll back to validated stack (`cutlass`, `flashinfer`)
No JSON output	Runtime aborted before benchmark write	Inspect `.out` log, retry with smoke configuration
Unstable behavior across runs	Environment drift	Re-source `env.sh`, avoid custom conflicting exports

Important: submit*.sh resets conflicting CUDA/NCCL env variables before submitting:

unset LD_PRELOAD LD_LIBRARY_PATH CUDA_HOME CUDACXX TRITON_CACHE_DIR

Keep this behavior unless you are intentionally testing an environment override.

Optional: NCCL Variant Submission

For communication tuning experiments, use NCCL wrappers:

submit-nccl.sh
submit-nccl-pp.sh
submit-nccl-2ntp.sh

They pass NCCL_EXTRA to the corresponding bench-nccl*.sh.