[gmx-users] Maximising Hardware Performance on Local node: Optimal settings

[Matthew Fisher]

Dear all,

We're currently running some experiments with a new hardware configuration and attempting to maximise performance from it. Our system contains 1x V100 and 2x 12 core (24 logical) Xeon Silver 4214 CPUs which, after optimisation of CUDA drivers & kernels etc., we've been able to get a performance of 210 ns/day for 60k atoms with GROMACS 2019.3 (allowing mdrun to select threads, which has surprised us as it only creates 24 OpenMP threads for our 48 logical core system). Furthermore we have a surprising amount of wasted GPU time. Therefore, we were wondering if anyone had any advice on how we could maximise our hardware output? We've enclosed the real cycle and time accounting display below.

Any help will be massively appreciated

Thanks,
Matt

     R E A L   C Y C L E   A N D   T I M E   A C C O U N T I N G

On 1 MPI rank, each using 24 OpenMP threads

 Computing:          Num   Num      Call    Wall time         Giga-Cycles
                     Ranks Threads  Count      (s)         total sum    %
-----------------------------------------------------------------------------
 Neighbor search        1   24      12501      32.590       1716.686   3.2
 Launch GPU ops.        1   24    2500002     105.169       5539.764  10.2
 Force                  1   24    1250001     140.283       7389.414  13.6
 Wait PME GPU gather    1   24    1250001      79.714       4198.902   7.7
 Reduce GPU PME F       1   24    1250001      25.159       1325.260   2.4
 Wait GPU NB local      1   24    1250001     264.961      13956.769  25.7
 NB X/F buffer ops.     1   24    2487501     177.862       9368.871  17.3
 Write traj.            1   24        252       5.748        302.799   0.6
 Update                 1   24    1250001      81.151       4274.601   7.9
 Constraints            1   24    1250001      70.231       3699.389   6.8
 Rest                                          47.521       2503.167   4.6
-----------------------------------------------------------------------------
 Total                                       1030.389      54275.623 100.0
-----------------------------------------------------------------------------

               Core t (s)   Wall t (s)        (%)
       Time:    24729.331     1030.389     2400.0
                 (ns/day)    (hour/ns)
Performance:      209.630        0.114