This document defines the official Sequential Multi-Client Shared Storage Challenge task for the ISC Student Cluster Competition. The goal is to evaluate and optimize the performance of a shared storage system using realistic HPC and AI workload patterns.

Teams will benchmark sequential read and write performance using the industry-standard tool fio (Flexible I/O Tester). The primary scoring metric is aggregate throughput. Latency and fio-reported IOPS may be recorded as informational metrics for comparison.

Competition Goals

• Teach students practical storage benchmarking methodology used in HPC and AI environments.

• Demonstrate the impact of concurrency, queue depth, and block size on storage performance.

• Analyze scalability when workloads are distributed across multiple compute nodes.

• Encourage experimentation and performance tuning under time constraints.

System Architecture Assumptions

• Minimum 2 compute nodes

• Nodes have high-speed network connectivity to each other

• Storage may be mounted using NFS, SMB, or a POSIX-compatible filesystem

• Maximum benchmark runtime per test: 90 seconds

• Maximum file size per fio job: 50 GB

  The official benchmark configuration uses size=50G.

• Shared NVMe-backed storage must be used

• RAM disks, tmpfs, memory-backed filesystems, loopback devices backed by RAM, and other memory-only storage methods are prohibited

• Any storage device used for benchmarking must be persistent block storage

• Teams must be prepared to show evidence that NVMe devices are being used

  • Example validation commands: lsblk, nvme list, df -h, mount

Benchmark Tool – fio

fio (Flexible I/O Tester) is an open-source benchmarking tool capable of generating complex I/O

patterns that emulate real workloads.

Documentation: https://fio.readthedocs.io/

Installing fio

Download latest release, https://github.com/axboe/fio/archive/refs/tags/fio-3.42.tar.gz

Important fio Requirements

fio defaults to ioengine=psync if no ioengine is specified.

psync uses synchronous I/O and effectively limits queue depth to 1, even if a larger iodepth value is configured.

To ensure that queue depth and asynchronous I/O are actually exercised, teams must explicitly specify one of the following:

or

For consistency across systems, libaio is recommended.

Teams should also use below to avoid excessive page cache effects.

Teams should use:

This ensures that each fio job operates on its own file and prevents unintended file sharing between concurrent jobs.

fio Client/Server Mode

Start the fio server on each client:

Starting fio Servers on Multiple Nodes

For convenience, teams may use parallel shell tools such as pdsh to start fio servers on multiple client nodes simultaneously.

Example:

pdsh -R ssh -w node[01-04] \
'pkill -f "fio --server" >/dev/null 2>&1; \
fio --server --daemonize=/tmp/fio-server.log'

Example using an explicit host list:

pdsh -R ssh -w 192.168.10.[1-4] \
'pkill -f "fio --server" >/dev/null 2>&1; \
fio --server --daemonize=/tmp/fio-server.log'

Verify fio servers are running:

pdsh -R ssh -w node[01-04] \
'pgrep -af fio'

Teams are responsible for ensuring that fio servers are running on all intended benchmark clients before executing a benchmark.

Input1: sequential write job file (seqwrite.fio):

How to run FIO using 4 nodes:

1. Create a host file (hosts.txt) with IP addresses and start fio servers:

   192.168.10.1 192.168.10.2 192.168.10.3 192.168.10.4

2. Submit a job using the host file:

   fio --client=hosts.txt seqwrite.fio --output=seqwrite.json --output-format=json+

Important: fio size is per job, not per client.

Total storage footprint is: clients × numjobs × size

For example, 4 clients with numjobs=4 and size=50G require up to 800 GB of file footprint.

The 50G size is a file footprint target, not a total I/O volume limit. With time_based=1, fio may continue issuing I/O for the full runtime against the same file range.

Input2: sequential read job file (seqread.fio):

The sequential read workload allows teams to evaluate aggregate read scalability across multiple clients using the same dataset created during the write phase.

1. Submit a job using the host file:

   fio --client=hosts.txt seqread.fio --output=seqread.json --output-format=json+

Important:

The sequential read benchmark is intended to operate on the files generated by the sequential write benchmark. Teams should not delete or recreate benchmark files between the write and read phases.

Judges should evaluate aggregate bandwidth across all participating clients. Throughput should generally increase as additional clients are added until a storage, network, or filesystem bottleneck is reached.

Reference Implementation

A reference implementation containing example fio job files, JSON parsing utilities, score calculation logic, and scaling efficiency analysis is available at:

https://github.com/bonibruno/isc26-fio-parser

The repository contains:

parse_fio_score.py
seqwrite.fio
seqread.fio
README.md

The parser automatically:

• Extracts aggregate throughput from fio JSON output
• Detects fio client/server aggregate summary rows
• Calculates the official Storage Challenge score
• Reports throughput, IOPS, latency, and total I/O
• Optionally calculates scaling efficiency using single-client baseline results

The repository is provided as a convenience reference for students and judges. Teams may use their own scripts and tooling provided the official submission requirements are satisfied.

Tasks and Submissions

Teams must run fio workloads on multiple nodes.

Tunable Parameters

Submission:

Teams must submit the following files:

seqwrite.fio, seqread.fio, seqwrite.json, seqread.json, hosts.txt and fio version

Filesystem information:

• df -h <directory>

• mount

Official scoring will be performed using the aggregate throughput reported in the fio JSON output.

For fio client/server mode, the JSON output contains an aggregate summary row:

jobname = "All clients"

This aggregate row will be used for official score calculation.

Official Score

Final Score =
Aggregate Sequential Write Throughput +
Aggregate Sequential Read Throughput

Example:

Write = 74.55 GB/s
Read = 159.40 GB/s

Final Score = 233.95 GB/s

When present, the fio JSON row where:

jobname = "All clients"

is considered authoritative and should be used instead of summing individual client rows.

Reference parser:

https://github.com/bonibruno/isc26-fio-parser

Official Scoring Format

1. Official scoring and validation will use the JSON output files generated using:

   fio --client=hosts.txt <jobfile>.fio --output=<output>.json --output-format=json+

   Examples:

   fio --client=hosts.txt seqwrite.fio --output=seqwrite.json --output-format=json+ 2>&1 | tee seqwrite.txt

   fio --client=hosts.txt seqread.fio --output=seqread.json --output-format=json+ 2>&1 | tee seqread.txt

   The JSON output format allows judges to verify:

2. Aggregate multi-client bandwidth

   • Per-client participation

   • Runtime duration

   • Job count (numjobs)

   • Queue depth (iodepth)

   • Block size

   • Latency metrics

   • fio version and configuration consistency

3. Storage Validation

   Teams must demonstrate how the storage was mounted on each client node.

   Example validation commands:

   mount | grep <storage_path>

   df -h | grep <storage_path>

   Judging Notes

   Judges must evaluate aggregate performance across all participating clients rather than a single client or single job result.

   For each submission, judges should record:

   • Aggregate sequential write throughput (GB/s)

   • Aggregate sequential read throughput (GB/s)

   • Average completion latency

   • Number of participating clients

   Storage systems are expected to demonstrate increasing aggregate throughput as additional clients are added until a storage, network, or filesystem bottleneck is reached.

   Important Capacity Note

   fio size is specified per job, not per client. Total storage footprint is calculated as:

   Total Capacity = Number of Clients × numjobs × size

   For example:

   4 clients × 4 numjobs × 50 GB = 800 GB

   4 clients × 4 numjobs × 100 GB = 1.6 TB

   4 clients × 4 numjobs × 200 GB = 3.2 TB

   Teams should ensure sufficient storage capacity exists before testing.

4. Final Score =
   Aggregate Sequential Write Throughput (GB/s) +
   Aggregate Sequential Read Throughput (GB/s)

5. Scoring Philosophy

   The Storage Challenge evaluates aggregate shared-storage performance under concurrent multi-client access. Teams may utilize different numbers of client nodes depending on their cluster architecture and available resources. Final rankings are based on aggregate sequential write and read throughput achieved by the complete storage solution. The benchmark is intended to reward effective storage system design, deployment, and scalability rather than normalize performance on a per-client basis.

   Judges may additionally record informational metrics such as scaling efficiency, throughput per client, and latency; however, these metrics are not included in the final score.

FAQ:

Q1 - What kinds of fio optimization are allowed? For example, are we allowed to change fio parameters while keeping the fio source code unchanged?

Answer to Q1:  Teams are encouraged to optimize fio through parameter tuning only, while keeping the fio source code unchanged. The challenge intentionally allows tuning of workload behavior because the goal is to teach real-world performance engineering techniques rather than benchmark script memorization. Students may tune parameters such as numjobs, iodepth, file layout (directory, filename, nrfiles), the number of participating clients, filesystem mount options, and network or storage layout tuning. However, fixed parameters such as ioengine=libaio, direct=1, block size, runtime, file size, and workload type may not be modified. Modifying fio source code, patching fio behavior, or using custom fio binaries with altered functionality is not allowed. The objective is to reward system engineering and tuning rather than changes to the benchmark itself.

Q2 - Are we allowed to introduce or use a new parallel filesystem for the task?

Answer to Q2: Yes. Teams may use a new filesystem or storage architecture, including distributed or parallel filesystems, provided the solution satisfies all challenge requirements. Storage must be persistent, cannot be RAM-backed, must use NVMe-backed storage, and must provide POSIX-compatible access with support for concurrent multi-client operation. The challenge intentionally allows experimentation because filesystem architecture and deployment choices are part of real-world HPC and AI system design. The objective is not to force teams into a predefined storage stack, but rather to encourage creative engineering and allow teams to demonstrate how effectively they can design, deploy, tune, and optimize a storage solution under realistic multi-node conditions.

Q3 - Are closed-source or proprietary parallel filesystems allowed?

Answer for Q3: Yes. Closed-source and proprietary filesystems are allowed.

Q4 - If closed-source filesystems are allowed, how will their validity be verified?

Answer for Q4: Closed-source filesystems are permitted, but teams must provide sufficient evidence demonstrating that the solution is legitimate, reproducible, and operating on persistent storage. The challenge is intended to evaluate system engineering and optimization rather than require public source availability. Teams should provide the filesystem name and version, a high-level architecture description, the storage protocol and mount configuration used, filesystem type information, storage validation output, fio command lines, node topology, and an explanation of tuning modifications made during testing. Judges may additionally request validation outputs from commands such as mount, lsblk, nvme list, df -h, and fio --version to verify the configuration. The objective is not to force disclosure of proprietary intellectual property, but rather to ensure that the solution is operating on real persistent infrastructure, follows the competition rules, and can be reasonably understood and evaluated for fairness and comparability.

Q5 - Are teams allowed to run fio directly against local block devices (for example, /dev/nvme0n1 or /dev/sda) on the client nodes?

Answer to Q5: No. The objective of the Storage Challenge is to evaluate and optimize shared storage performance across multiple client nodes. fio workloads must be executed against files located on a shared filesystem that is concurrently accessible from multiple clients.

Examples of acceptable shared filesystems include NFS, SMB, Lustre, BeeGFS, WEKA, CephFS, IBM Spectrum Scale (GPFS), and similar shared-storage solutions.

Teams may use block devices on backend storage servers as part of the storage implementation. For example, NVMe SSDs may be used as storage targets within a distributed or parallel filesystem. However, fio may not be run directly against local block devices attached to individual client nodes, as this would bypass the shared-storage architecture being evaluated.

Judges may request validation output from commands such as:

mount
df -h
lsblk
nvme list

fio --version

to verify that the benchmark is operating against shared persistent storage.

Q6 - Are compression, de-duplication, or other data-reduction technologies allowed?

Answer to Q6: Yes, provided they are part of the normal operation of the storage solution and are enabled consistently throughout testing.

However, teams may not intentionally manipulate the benchmark data to artificially inflate performance through unrealistic compression or de-duplication effects. The benchmark should represent a realistic storage workload.

Judges reserve the right to request details regarding compression, de-duplication, or other data-reduction settings if observed performance appears inconsistent with the underlying hardware configuration.

Teams should disclose any enabled data-reduction technologies as part of their submission documentation.

The challenge is intended to reward storage system design, deployment, and optimization rather than exploitation of benchmark-specific data patterns.

Q7 - Will teams using more client nodes have an advantage?

Answer to Q7: Potentially yes. The Storage Challenge is designed to evaluate aggregate shared-storage performance under concurrent multi-client access. As additional clients are added, storage systems may be able to generate higher aggregate throughput by increasing parallelism.

The objective of the challenge is to evaluate how effectively teams design, deploy, and tune a shared storage architecture to scale across multiple clients. Aggregate throughput is therefore considered part of the intended benchmark behavior rather than an unfair advantage.

Q8 - How is the official score calculated?

Answer to Q8: The official score is calculated using aggregate throughput reported by fio for all participating clients.

Final Score =
Aggregate Sequential Write Throughput +
Aggregate Sequential Read Throughput

When fio is executed in client/server mode, the JSON output contains a summary row with:

jobname = "All clients"

This aggregate row represents the combined throughput of all participating clients and will be used for official scoring.

Reference parser and examples are available at:

https://github.com/bonibruno/isc26-fio-parser

Q9 - Is scaling efficiency part of the official score?

Answer to Q9: No.

Scaling efficiency is an optional informational metric that may be useful for understanding how effectively a storage solution scales as additional clients are added, but it is not part of the official competition score.

A common formula is:

Scaling Efficiency =
Multi-Client Score /
(Single-Client Score × Number of Clients)

Example:

Single-Client Score = 59.93 GB/s
Multi-Client Score = 233.95 GB/s
Clients = 4

Scaling Efficiency = 97.60%

The official competition score remains:

Final Score =
Aggregate Sequential Write Throughput +
Aggregate Sequential Read Throughput

Q10 - Can the fio launch node also be one of the benchmark clients?

Answer to Q10: Yes.

The fio command may be launched from either a dedicated control node or from a participating benchmark client.

When fio is executed in client/server mode, the JSON output may contain both individual client statistics and an aggregate summary row:

jobname = "All clients"

As a result, the number of entries in client_stats[] may be larger than the number of participating benchmark clients.

The aggregate summary row represents the combined throughput of all participating clients and should be used for official score calculation.

Teams should not determine the official score by counting client_stats[] entries or by summing the aggregate summary row together with individual client rows.

A dedicated control node may simplify validation and troubleshooting but is not required.

Reference examples and parser:

https://github.com/bonibruno/isc26-fio-parser

Q11 - How can I quickly validate my fio JSON output?

Answer to Q11:

The fio JSON output contains both per-client statistics and an aggregate summary row. The following jq commands may be useful for quick validation and troubleshooting.

Show all participating clients:

jq -r '
.client_stats[]
| .hostname
' seqwrite.json

Show per-client write throughput:

jq -r '
.client_stats[]
| select(.jobname != "All clients")
| [
.hostname,
(.write.bw_bytes / 1000000000)
]
| @tsv
' seqwrite.json

Example output:

172.31.18.70 21.20
172.31.18.71 18.47
172.31.18.72 14.15
172.31.18.73 20.76

Show aggregate write throughput:

jq -r '
.client_stats[]
| select(.jobname == "All clients")
| .write.bw_bytes / 1000000000
' seqwrite.json

Show aggregate read throughput:

jq -r '
.client_stats[]
| select(.jobname == "All clients")
| .read.bw_bytes / 1000000000
' seqread.json

Show aggregate score:

WRITE=$(jq -r '
.client_stats[]
| select(.jobname == "All clients")
| .write.bw_bytes / 1000000000
' seqwrite.json)

READ=$(jq -r '
.client_stats[]
| select(.jobname == "All clients")
| .read.bw_bytes / 1000000000
' seqread.json)

awk -v w="$WRITE" -v r="$READ" \
'BEGIN { printf "Final Score: %.2f GB/s\n", w+r }'