/
DeepSeek R1 Inference Optimization Best Practice

DeepSeek R1 Inference Optimization Best Practice

This guide summarizes the current optimization strategy for DeepSeek R1-0528 (671B) inference on H200 clusters, based on the latest benchmark results.

Overall Summary

Metric

Value

Configuration Anchor

Metric

Value

Configuration Anchor

TP8 single-node peak

22,854 tok/s

TP8 + champion stack at N=7000

PP2 dual-node peak

29,426 tok/s

pp-size=2, partition 32:29, mf=0.83, N=7000

TP16 dual-node peak

21,416 tok/s

TP16 + LOF + FlashInfer at N=8000

TP8 efficiency

2,857 tok/s/GPU

Best per-GPU throughput among topologies

PP2 efficiency

1,839 tok/s/GPU

Highest total throughput, lower per-GPU efficiency

TP16 efficiency

1,338 tok/s/GPU

Communication-bound on cross-node AllReduce

image-20260414-094146.png
Throughput scaling + per-GPU efficiency, total vs per-GPU bars

Practical Conclusions

  1. mem-fraction-static is the first-order control variable. On TP8, 0.87 is the stable sweet spot.

  2. PP2 is the throughput-first topology for dual-node deployment.

  3. TP8 remains the efficiency-first topology for single-node and cost-sensitive operation.

  4. Parameter effects are topology-dependent and load-dependent. Re-validate when topology or N changes.

Benchmark Setup and Scope

Platform

  • GPU: NVIDIA H200 SXM 140 GB

  • Intra-node interconnect: NVLink/NVSwitch

  • Inter-node interconnect: 400 Gb/s NDR InfiniBand

  • Runtime: SGLang offline throughput benchmark mode

Workloads

  • Dataset profile: ShareGPT V3 style workload

  • Standard comparison load: N=5000

  • Peak validation load: N=7000 (TP8/PP2), N=8000 (TP16)

  • Weight mode for benchmark profiling: dummy weights

  • Reproducibility seed: 2025

Important Boundary

All throughput values in this guide are throughput values under benchmark conditions. Keep the same protocol when comparing configurations.

For production service sizing, run a final validation pass with your deployment weight mode and traffic profile.

Topology Decision Guide

Goal

Recommended Topology

Why

Goal

Recommended Topology

Why

Max total throughput

PP2

Highest proven peak and stronger high-load behavior

Max per-GPU efficiency

TP8

Best tok/s/GPU and simpler operational profile

TP scaling reference on dual-node

TP16

Only for boundary reference

Communication Reality Behind TP16

TP16 improves per-GPU weight/KV balance but pays heavy cross-node synchronization cost. For DeepSeek R1's 61-layer structure, TP16 can hit 122 cross-node AllReduce operations per forward path (2 per layer x 61), which limits net scaling efficiency.

Key Interaction Signals

These signals are repeatedly useful when tuning beyond baseline:

Signal

Observed Effect

Practical Meaning

Signal

Observed Effect

Practical Meaning

LOF + FlashInfer

Super-additive behavior (+7.5% vs +2.9% linear expectation in TP8 studies)

Evaluate them as a pair, not in isolation

CGBS uplift

cuda-graph-max-bs=2048 can unlock the largest single-step TP8 gains in high-load runs

Keep CGBS in the core TP8 stack

NRC load dependency

Can be weak/negative at low N, then strongly positive at high N (for example up to +20.6% at very high load slices)

Validate disable-radix-cache at production-like load

EP behavior

Structural negative region in current tested setup

Treat EP as an explicit exception path, not a default optimization

image-20260414-093915.png
LOF catalyst chain, stepwise stacking shows stronger-than-expected LOF+FI interaction

Optimization Playbook A: TP8 Single Node

Key Findings

  • mf=0.87 is the stable TP8 sweet spot.

  • The TP8 champion is a stacked path, not a single flag.

  • CGBS is a major unlock: raising cuda-graph-max-bs to 2048 is a top gain source.

  • disable-radix-cache becomes valuable at high load (especially from N >= 7000).

Recommended TP8 Champion Command

python -m sglang.bench_offline_throughput \ --model-path $MODEL \ --tp 8 \ --mem-fraction-static 0.87 \ --schedule-policy lof \ --moe-runner-backend cutlass \ --enable-piecewise-cuda-graph \ --decode-attention-backend flashinfer \ --enable-flashinfer-allreduce-fusion \ --chunked-prefill-size 32768 \ --cuda-graph-max-bs 2048 \ --disable-radix-cache \ --num-prompts 7000

Expected result: ~22,854 tok/s at N=7000.

image-20260414-093432.png
From baseline 12,021 to peak 22,854 tok/s, late-stage CGBS+NRC contribute +15.8%

TP8 Stability Notes

  • If load is low (N=1000), many parameters look flat or misleading.

  • disable-radix-cache is not a universal default for every load. It is most useful at higher request scales in offline throughput scenarios.

Optimization Playbook B: PP2 Dual Node

Key Findings

  • PP2 provides the best total throughput with lower tuning risk.

  • Layer partition matters: 32:29 is the current best-known split.

  • mf=0.83 is the practical sweet spot for the PP2 peak profile.

  • Keep micro-batch tuning conservative; default/auto behavior is often best.

Recommended PP2 Champion Command

SGLANG_PP_LAYER_PARTITION=32,29 \ python -m sglang.bench_offline_throughput \ --model-path $MODEL \ --tp 8 \ --pp-size 2 \ --nnodes 2 \ --mem-fraction-static 0.83 \ --disable-radix-cache \ --chunked-prefill-size 32768 \ --num-prompts 7000

Expected result: ~29,426 tok/s at N=7000.

image-20260414-093547.png
Partition 32:29 (+0.9%); micro-batch manual = 10× disaster; chunk spread < 2.4%

PP2 Operational Notes

  • PP2 is more tolerant than TP8 to small parameter perturbations.

  • Partition tuning should be done first, then cache/chunk refinement.

Optimization Playbook C: TP16 Dual Node (Reference Path)

TP16 is useful as a dual-node tensor-parallel reference, but it is not the throughput champion under the current tested conditions.

python -m sglang.bench_offline_throughput \ --model-path $MODEL \ --tp 16 \ --nnodes 2 \ --mem-fraction-static 0.87 \ --schedule-policy lof \ --moe-runner-backend cutlass \ --decode-attention-backend flashinfer \ --chunked-prefill-size 32768 \ --num-prompts 8000

Expected result: ~21,416 tok/s at N=8000.

image-20260414-093644.png
PP2 vs TP16 at 16× H200, PP2 wins 36–43% at every load; 1 cross-node sync vs 122 AllReduces

What Not To Do (High-Risk Anti-Patterns)

Anti-pattern

Typical Impact

Why It Fails

Anti-pattern

Typical Impact

Why It Fails

FP4 KV cache path

Up to ~-78%

Quant/dequant overhead dominates in throughput-bound runs

Speculative decoding in saturated batch mode

Up to ~-69%

Adds overhead when GPU pipeline is already full

FlashInfer path on PP2 without validated context

Negative cases observed (for example ~-18.5%)

Kernel behavior conflicts with PP2 communication/scheduling dynamics

EP on current single-node FP8 workflow

Structural negative zone (commonly ~-34% to -40% in tested EP runs)

All-to-all communication overhead dominates

Manual small PP micro-batch override

Severe collapse (up to 10x drop)

Pipeline bubbles and underfilled compute

Deciding from low-load-only tests (N=1000)

False ranking and direction flips

Load reversal appears at higher N

Methodology and Reproducibility Checklist

Use this checklist before claiming result parity:

  1. Keep topology fixed (TP8 vs PP2 vs TP16 must not be mixed in comparison).

  2. Keep load fixed (N=5000 for comparison, N=7000/8000 for peak checks).

  3. Keep the default model loading path fixed across comparisons.

  4. Keep mem-fraction-static aligned with topology (0.87 for TP8, 0.83 for PP2 peak profile).

  5. Use the same scheduling/backend stack when reproducing champion numbers.

  6. Record command line and environment for every run.

Deployment Decision Matrix

Deployment Intent

Recommended Stack

Deployment Intent

Recommended Stack

Single-node, best efficiency

TP8 champion stack

Dual-node, highest throughput

PP2 champion stack (32:29, mf=0.83)

Dual-node tensor-parallel reference

TP16 reference stack