ChaNGa Challenge

Cosmological simulation framework “ChaNGa” is a collaborative project with Prof. Thomas Quinn (University of Washington: N-Body Shop) supported by the NSF. ChaNGa (Charm N-body GrAvity solver) is a code to perform collisionless N-body simulations Cosmological simulation. It can perform cosmological simulations with periodic boundary conditions in comoving coordinates or simulations of isolated stellar systems. It also can include hydrodynamics using the Smooth Particle Hydrodynamics (SPH) technique. It uses a Barnes-Hut tree to calculate gravity, with hexadecapole expansion of nodes and Ewald summation for periodic forces. Timestepping is done with a leapfrog integrator with individual timesteps for each particle. ChaNGa runs over Charm++ framework.

In ISC20 you will run a cosmological simulation of a galaxy cluster with an unprecedented resolution.

The input cosmological simulation for this task will be given for you in flash disk.

Task 1: Build and Compile Charm++ for ChaNGa with several machine layers (several builds), you can refer to HowTo Build and Run ChaNGa over Charm++ using UCX/MPI Machine layer as a reference example.

Task 2: Build ChaNGa and run it with the given input, for the charm++ builds you created:

./charmrun +p96 ./ChaNGa h1.768.vis.param

Task 3 : Create short video out of the output (using *.ppm) files, you can refer here for an example: How to create an mp4 video from set of ppm images or use another technic.

Task 4: Add a team name-tag with/or team photo in small into the video (to mark it) and post to your team’s twitter account if you have, or linkedIn account with the hashtags #ISC20 #ISC20_SCC. if you don’t use social media, you can submit the *mp4 file via the flash memory.

We will monitor “Big Step” average values here for the performance numbers.

$ ./charmrun +p96 ./ChaNGa h1.768.vis.param
 
ChaNGa version 3.4, commit v3.4-36-g9f2b05e
Running on 96 processors/ 96 nodes with 32768 TreePieces

...
Step: 2031.000000 Time: 0.327722 Rungs 0 to 3. Gravity Active: 63066348, Gas Active: 15330135
Domain decomposition ... Sorter: Histograms balanced after 11 iterations.
total 2.90383 seconds.
Load balancer ... Orb3dLB_notopo: Step 120
numActiveObjects: 32768, numInactiveObjects: 0
active PROC range: 0 to 95
Migrating all: numActiveObjects: 32768, numInactiveObjects: 0
[Orb3dLB_notopo] sorting
***************************
Orb3dLB_notopo stats: maxObjLoad 2.825454
Orb3dLB_notopo stats: minWall 10.692922 maxWall 10.715588 avgWall 10.702433 maxWall/avgWall 1.001229
Orb3dLB_notopo stats: minIdle 0.621588 maxIdle 10.644512 avgIdle 9.349612 minIdle/avgIdle 0.066483
Orb3dLB_notopo stats: minPred 46.212934 maxPred 46.496672 avgPred 46.357328 maxPred/avgPred 1.003006
Orb3dLB_notopo stats: minPiece 276.000000 maxPiece 416.000000 avgPiece 341.333333 maxPiece/avgPiece 1.218750
Orb3dLB_notopo stats: minBg 0.031479 maxBg 1.458530 avgBg 0.232613 maxBg/avgBg 6.270197
Orb3dLB_notopo stats: orb migrated 28082 refine migrated 0 objects
took 2.02924 seconds.
Building trees ... took 0.686058 seconds.
Calculating gravity (tree bucket, theta = 0.700000) ... Kick Close:
 Rung 0: 8.06111e-05
 Rung 1: 4.03055e-05
 Rung 2: 2.01528e-05
 Rung 3: 1.00764e-05
Calculating gravity and SPH took 50.9711 seconds.
Kick took 0.0476675 seconds.
Finish NodeCache took 0.977946 seconds.
Big step 2031 took 136.851713 seconds.
KillAT: Stopping after 2121.087017 seconds
Done.

Note: For HPC-X support for PBS you will need to build with the following flag:

--with-tm=/opt/pbs

Note 2: For Performance results, you can comment out dDumpFrameStep parameter, and submit the performance results.