TL;DR

Maximize batched inference throughput of a dense 70B model using llama.cpp on your cluster. Your perplexity must stay within 10% of the unoptimized baseline.

Scored Model

Llama-3.1-Nemotron-70B-Instruct-HF-Q4_K_M.gguf — a dense transformer, 70B parameters, Q4_K_M quantization (~42.5 GB).

• The model weight will be provided to teams in flash drives at the competition.

• You must use the exact GGUF file provided. Re-quantization, weight modification, or substituting a different quant is not allowed. Organizers verify the file's SHA-256 at grading time.

• The MoE model in the prep pack (Llama-4-Scout) is not scored this year. It was provided to broaden your understanding of llama.cpp; the actual scored workload is dense-only.

Workload — 4 Tiers

Each tier is a fixed (npp, ntg) workload measured with llama-batched-bench. You decide all other flags (parallelism, KV cache type, FA, batch sizes, GPU count, context size) per tier.

Every other knob — parallelism (-npl), gpu_count, FA, KV-cache type, batch sizes, split mode, etc. — is yours to set independently per tier (see Allowed Optimizations).

Score Formula

• S_total is the final S t/s value reported by llama-batched-bench (last column of the result table).

• Your weighted throughput is the quantity above; it's scaled across teams — the highest gets the full 80, others proportionally. It is not your point total directly.

• Tiebreaker: GPU-efficiency = weighted throughput / max(gpu_count across tiers). Lower GPU usage wins ties. (score.sh reports this as gpu_efficiency.)

Perplexity Quality Gate

For every tier, score.sh runs llama-perplexity on wikitext-2 and requires:

The PPL run forwards only the quality-affecting flags from that tier's bench — -fa, --cache-type-k/-v, -sm, --tensor-split — and sets context itself (-c 4096, full offload -ngl 999). Your other bench flags (-npl, -b, -ub, and your own -c/-ngl) are ignored: llama-perplexity rejects some, and none affect PPL.

The unoptimized baseline is about 3.51 for the scored model (organizers may publish a refined number on competition day). The effective gate is therefore PPL ≤ 3.87 (baseline × 1.10, rounded up to 0.01). Tiers whose PPL exceeds the threshold score zero for that tier.

This 10% allowance covers all reasonable runtime/build optimizations (KV-cache quantization, flash attention, etc., which typically degrade PPL by <5%) while preventing quality-destroying modifications.

Allowed Optimizations

Not Allowed

One-Codebase Rule

You submit a single llama.cpp repo. The same codebase must build and run all four tiers. Per-tier flag changes are allowed; per-tier source-code branches are not.

Baseline llama.cpp commit

• Upstream: github.com/ggml-org/llama.cpp

• Reference: tag b9568 — commit 7d2b45b4f (2026-06-08)

You may build on any llama.cpp commit — newer or older — so there's no need to rebase your work onto a specific base. b9568 is the reference the organizer baselines (perplexity reference, plausibility bounds) were measured against, and a good default starting point:

If you build on a commit other than b9568, state its full SHA in your optimization report. Whatever commit you choose, the quality gate is the same: your llama-perplexity must stay within the published threshold (see Perplexity Quality Gate).

Submission

Running score.sh

Prerequisites: build your llama.cpp first — cmake -B build -DGGML_CUDA=ON && cmake --build build (produces build/bin/llama-batched-bench + llama-perplexity). The host also needs python3 + PyYAML (pip install pyyaml) and a working NVIDIA GPU (nvidia-smi).

• 4th arg (full_logs/) is the output directory; score.sh creates and populates it. Submit this directory.

• WIKI=/path/to/wiki.test.raw ./score.sh ... — override if wikitext-2-raw/wiki.test.raw isn't in CWD or one of the autodiscover paths (./models/, ./data/, $HOME/).

• TIERS="T-1" (or TIERS="T-1,T-3") — run a subset of tiers. Different tiers accumulate in the same output dir; re-running a tier overwrites that tier's files (details under Required Layout). Each successful tier needs a judge witness (see Verification).

score.sh streams everything live on your terminal as it runs — including the full llama-batched-bench / llama-perplexity output — while also saving each tier's output to full_logs/logs/. Keep the terminal visible so a judge can verify the run on the spot (see Verification).

Required Layout

Submit your work as a single directory (tar/zip the directory before uploading) with this layout. Files outside this layout may be ignored.

Rules:

• Use score.sh to produce full_logs/ (see Running score.sh above) and submit the resulting directory unchanged — don't roll your own. Hand-edited or partial logs are rejected.

• Source must build on its own. A git bundle is preferred (smaller, captures branch history). Either git clone <bundle> or cp -r llama.cpp source/llama.cpp must produce a directory where cmake -B build -DGGML_CUDA=ON && cmake --build build succeeds with no additional steps.

• Do not rename or restructure these directories. Our grading pipeline auto-discovers them by name. Extra files at the top level are fine; missing required files trigger a rejection.

• model.sha256 / wiki.sha256 — score.sh auto-generates both into full_logs/. Organizers cross-check both the model and the wikitext reference against the official files; a mismatch on either flags the submission. Don't hand-edit them.

• No build artifacts. Exclude build/, *.o, *.so, downloaded models, and the wikitext data from the submission tarball — they're large and we have our own copies.

• Files accumulate across different tiers — sequential TIERS=T-N invocations into the same full_logs/ each leave their bench/PPL logs and a T-N.json shard. Re-running the same tier overwrites that tier's files (T-N_bench.log, T-N_ppl.log, T-N.json, etc.) — only the most recent run for a given tier is kept. Missing or failed tiers score 0; the directory structure is identical either way. You may not re-run a tier a judge has already verified (see Verification).

Files (detail)

1. Source code — your modified llama.cpp repo as a git bundle (git bundle create llama.cpp_<tag>.bundle --all) under source/. A directory is also accepted.

2. Config YAML — declared flags + GPU count per tier. Example:

   model: Llama-3.1-Nemotron-70B-Instruct-HF-Q4_K_M.gguf per_tier: T-1: flags: "-ngl 999 -sm layer -ub 512 -npl 8 -c 32768" gpu_count: 2 T-2: flags: "-ngl 999 -sm layer -ub 512 -npl 4 -c 32768" gpu_count: 2 T-3: flags: "-ngl 999 -sm layer --flash-attn on --cache-type-k q8_0 --cache-type-v q4_0 -ub 512 -npl 1 -c 65536" gpu_count: 2 T-4: flags: "-ngl 999 -sm layer --flash-attn on --cache-type-k q8_0 --cache-type-v q4_0 -ub 512 -npl 1 -c 65536" gpu_count: 2

3. Optimization report (reports/optimization_report.md) — 1 page max (~250 words), bullet points only, full sentences not required. We mainly want your per-tier flag rationale: for each tier, why those flags. Also mention any code changes (what, why, how you validated) and, if you didn't build on b9568, the base commit SHA you used.

(full_logs/ and the two *.sha256 files are produced by score.sh — you don't author them; see the layout and rules above.)

Verification

You must notify a judge after every successful tier run — either in person at your booth or via the contest Slack thread — so a judge can come by and verify the live stdout before you move on to the next tier. Runs that aren't witnessed by a judge may be rejected at grading time.

A verified tier is final. Once a judge has witnessed a tier's run, that result is locked — you may not re-run, overwrite, or submit a new result for that tier. You may still run and submit other, not-yet-verified tiers into the same full_logs/.

Judges may also ask you to re-run any tier on the spot during the AI task time window.

Resources

• Pre-announcement (sent ahead of the contest): pre_announcement.md

• llama.cpp: https://github.com/ggml-org/llama.cpp

• Wikitext-2: provided alongside the model on shared storage

Good luck.

Pre-Announcement

Overview

One of this year's in-person tasks is LLM inference optimization. You will use llama.cpp, the leading open-source framework for efficient LLM inference on consumer and HPC hardware, to maximize inference throughput.

The full problem statement — including specific models, workloads, and scoring details — will be released at the start of the competition. This preview gives you enough to prepare.

What is llama.cpp?

llama.cpp is a C/C++ inference engine for large language models. It supports:

• Multiple GPU backends: CUDA (NVIDIA), ROCm (AMD), Vulkan, SYCL (Intel), Metal (Apple), and CPU

• Quantized models in GGUF format — compressed model weights that reduce memory requirements while preserving quality

• Multi-GPU inference with different parallelism strategies

• Flash Attention and other attention optimizations

• Batched inference — processing multiple sequences simultaneously

Building llama.cpp

Baseline flags to build llama.cpp. Feel free to explore the full build flag landscape.

Key tools (in build/bin/)

Run any tool with --help to see available flags. Experiment with different flags to understand what they do and the outputs that they produce.

Non-comprehensive list of concepts that help prepare for LLM task

• Prefill (prompt processing): Processing the input prompt in parallel. Compute-bound.

• Decode (token generation): Generating output tokens one at a time. Memory-bandwidth-bound.

• KV cache: Stored attention state from previous tokens. Grows with context length.

• Quantization: Reducing precision of model weights or KV cache to save memory.

• Multi-GPU parallelism: Distributing model computation across multiple GPUs. Different strategies could have different tradeoffs.

• Batching: Processing multiple independent sequences simultaneously for potentially higher throughput.

What to expect on competition day

• You will be asked to optimize inference for potentially multiple models with different architectures (think dense transformers, mixture-of-experts, etc.). Specific model(s) will be provided at the start.

• Scoring will involve potentially multiple workloads including different configurations of prompt and generation lengths. Exact workloads would be provided at the start. For each model and configuration combination, different runtime flags could be used in order to optimize performance.

• Source code modifications of llama.cpp are allowed. You are allowed to modify the llama.cpp source code to improve performance, subject to a perplexity-based quality gate.

• One codebase rule: You will be required to submit a single llama.cpp codebase with baseline being the llama.cpp main branch that should work for all provided model(s) and configs. Source code modifications are allowed - you may incorporate optimization ideas/code from other research and/or repositories.

• A perplexity-based quality gate will be in place. We use llama-perplexity to measure model output quality on a reference dataset (e.g., wikitext-2). Optimizations that degrade perplexity by more than 10% relative to the unoptimized (or closest to unoptimized) baseline will score zero for that workload configuration. This threshold is designed to accommodate perplexity-preserving build-level, runtime and code optimizations while preventing modifications that sacrifice model quality by a large margin for speed.

• Verification: You will run benchmarks on your own cluster and submit your results along with your code and configuration. Organizers will verify your submission for correctness, and you may be asked to demonstrate your results live during the competition.

How to prepare

1. Build and run llama.cpp on your hardware. Get familiar with the build system and tools.

2. Download some GGUF models from HuggingFace and experiment. Suggested starting points:

   • A small dense model (1-3B parameters) for quick iteration — e.g., Llama-3.2-1B-Instruct-Q4_K_M.gguf

   • A mixture-of-experts model — e.g., Llama-4-Scout-17B-16E-Instruct-Q4_K_M.gguf — to understand how MoE differs from dense transformers

   • Try both architectures with different flags to see what changes

3. Explore the flag space. Run llama-bench and llama-batched-bench with different combinations of flags and observe the effects.

4. Understand context length variations. Try running the same model with different prompt lengths (e.g., 512, 4096, ... tokens) and observe how throughput changes with different flags.

5. Experiment with perplexity. Run llama-perplexity on a reference dataset (e.g., wikitext-2) to understand how different flags affect output quality.

6. Think about tradeoffs. More GPUs is not always faster. Some optimizations help at one scale but hurt at another. The optimal configuration depends on the workload.

Resources

• llama.cpp GitHub: https://github.com/ggerganov/llama.cpp

• GGUF model catalog: https://huggingface.co/models?library=gguf

• llama.cpp documentation: https://github.com/ggerganov/llama.cpp/blob/master/docs/build.md

What we will provide on competition day

• The full problem statement with specific model(s), workloads, and scoring details

• Pre-downloaded model files on a shared storage accessible to all teams

• A scoring script to test your configurations locally

• The baseline perplexity value for each model (your optimized perplexity must stay within 10% of this)

Good luck!