For this year, ISC21 SCC, we will have a coding challenge.
The main target is to analyze MPI_alltoallv patterns within the application.

MPI_alltoallv is a an MPI function call in which each rank sends to each other rank up to one message, similar to alltoall, but the size of each message could be different. More info about MPI_alltoallv can be found here.

In this task, you will be using an alltoallv collective profiler to generate MPI traces when running applications. After the run, you will use a webUI GUI to access the results and present them.

For your tests, use 4 nodes with full ppn, from the Niagara cluster only.

Coding Challenge - Overview presentation

by HPC Architect Geoffroy Vallee, Nvidia:

Download the slides here:

Open

Tasks

Task 1: Understand MPI_alltoallv calls

Write a program with an input flag for pattern, on the Niagara cluster using 4 nodes, each with 40 ppn (full), total of 160 ppn.

The program should run 1000 iterations of MPI_alltoallv using the following characteristics.

Assuming the ranks mapping is as follows:

ranks 0-39 → node 1

ranks 40-79 → node 2

ranks 80-119 → node 3

ranks 120-159 → node 4

MPI_alltoallv unbalanced Pattern:

• ranks 0-39 on node 1 are sending 1MB to ranks 40-79 on node 2, and 0B to the rest (40*40MB)

• ranks 40-79 on nodes 2 are sending 1MB to ranks 0-39 on node 1, and 0B to the rest (40*40MB)

• ranks 80-159 on nodes 3,4 are sending 0B to all ranks.

MPI_alltoallv balanced Pattern:

• ranks 0-19 ranks on nodes 1 are sending 1MB to ranks 40-59 and to ranks 80-99 (20*40MB)

• ranks 40-59 ranks on nodes 2 are sending 1MB to ranks 100-119 and to ranks 120-139 (20*40MB)

• ranks 80-99 ranks on nodes 3 are sending 1MB to ranks 140-159 and to ranks 0-19 (20*40MB)

• ranks 120-139 ranks on nodes 4 are sending 1MB to ranks 20-39 and to ranks 60-79 (20*40MB)

Compare the time it takes on both patterns. What are the differences, why?

Submit the code and results (zip file)

Task 2: Getting to know the MPI_alltoallv profiler

Download the profiler from here: https://github.com/gvallee/collective_profiler/releases/tag/ISC21_cc_v1

Use the ISC21_cc_v1 tag for the source code, follow README.md and Install.md to get started.

Run the profiler on GPAW and WRF applications, get the MPI traces for MPI_alltoallv calls. Run the WebUI and load the traces and view the results.

Note: the webUI can be used in any linux machine or laptop.

Nothing to submit here. You will need to present the results at the interview.

Task 3: Pattern display

Applications may have many MPI_alltoallv calls, but only few patterns (or templates) of those calls.

A pattern for n ranks can be defined as group of all <X_i,Y_i,M_i> - where rank X_i is sending to rank Y_i The Message length M_i. Each MPI_alltoallv have a different pattern, however, in many cases, there could be very large number of MPI_alltoallv calls, but only few patterns.

Pattern example:

• 3 ranks

• 4 MPI_alltoallv calls (in bytes)

In this example, we have 2 patterns while having 4 MPI_alltoallv calls (call #1,3,4 have the same pattern).

The pattern with more calls, might have higher impact on the run time results.

We can define the patten weight as the number of calls used for each pattern.

The Task

Display a list of up to 10 most heavy patterns (high weight) from high to low.

In this task you will need to get those patterns from the profiler, and develop the code to present it as heatmap in the webUI.

Per pattern show a heatmap graph with the following:

X axis -> rank 0..n (sender)

Y axis -> rank 0..n (receiver)

The value is a message size heat-map presented in 1x1 square with 8 color histogram:

0 -> white color

1-10 -> yellow color

11-100-> orange color

101->1,000 -> green color

1,001->10,000->red color

10,001->100,000-> purple color

100,001-> 1,000,000 -> brown color

1,000,001-> max -> black color

For example, for 4 rank MPI_alltoallv patten (this is specific pattern) we can get similar map to this (different colors).

(need to add X,Y rank labels)

Open

For your test, use 4 nodes with full ppn, from the Niagara cluster.

Submit the code and demonstrate the results in your report and ppt to the judges.

Note: This code should be added to the profiler as a clone from gitbub to your own private github location (details below).

Task 4

Create a map of all patterns on a different tab of the WebUI, where you present in the WebUI a map that is a sum of all patterns.

In cell (x_j,y_k) you will have the color based on the weighted sum (W_i) of all heatmap table (T) in the cell (x_j,y_k) For i =1 to number of patterns. j,k = 1 to number of ranks(=160) :=

cell (x,y) ← Sum{W_i*T_i(x_j,y_k)}

Task 5

Have the number of bins/colors as a fixed in the GUI to 256 colors, keeping a logarithmic scale. 8 colors is the example above for example.

Use 3 different color schemes

1. Linear distribution: e.g. the cell will get the color of min { Message length / 4KB , 255}

2. Logarithmic distribution: for example:

In general (a possible example, might need adjustment for the edges)

X=number of bytes in a cell // 0 to max integer.

n the number of colors // e.g. 0 to 255

The cell will get the color i ( 0<=i<=n)

1.6*log₂(X)*log₂n -1 ← color i

For example: for X=1MB and 256 colors, that would get 1.6*20*8-1=255 for the last color (black).

Above 1MB can be black.

Note: The formula may not reach all colors in some cases, and need special care for 0 message size.

3. Come up with your own distribution.

Submit the code and demonstrate the results in your ppt to the judges.

Task 6

Find a way to map ranks to cores in a more balanced way (using MPI rankfile for example) for WRF application, using the output you got, so each node will send/receive in average similar amount of bytes for MPI_alltoallv calls. Try to make this generic as possible.

Task 7 (Bonus)

Open task: Find ways to reduce the running time of the profiler, show the running time with and without the enhancement and show the code.

Submit the code and demonstrate the results in your ppt to the judges.

Task 8 (Bonus)

Find a way to reduce the disk space used for the alltoallv_profiler. Show your work via graph and code changes.

Submit the code and demonstrate the results in your ppt to the judges.

Work environment

Once you get the code, you will need to clone it to your own gitbub environment and to allow to our judge(s) access to your private area, to review your work.

Submissions

You will need to submit your code via (github pull request in your private area) and create a ppt report for this task to send and present to the judges.

Profiler output files

The profiler generates several different output files. For the heatmap, look on the following files :

Open the files:

• heat-map-send.md

• send-counters.job0.rank0.txt

• alltoallv_heat-map.rank0-recv.md

The counters will give you detailed count, per call, per rank. while the other files will give a summary of the count.

• Note that the message size is the count * datatype size.

send-counters.job0.rank0.txt

File output example, in this example Rank 430 is sending some data, but not to all ranks.

heat-map-send.md

output example:

This file shows the total bytes sent from each rank for the entire run.

alltoallv_heat-map.rank0-recv.md

This file shows the number of bytes sent per MPI_alltoallv call, per rank.

patterns-summary-job0-rank0.md

alltoallv_hosts-heat-map.rank0-send.md