[gmx-users] Performance GROMACS on GPU

[Talarico Carmine]

Hi,
I run three simulations on 1 node with 3 GPU by using an increasing number of GPUs.

This is my system:
___________________________________________________________________
1 node with total 36 cores, 72 logical cores, 3 compatible GPUs

GROMACS version:    2018.2
Precision:          single
Memory model:       64 bit
MPI library:        thread_mpi
OpenMP support:     enabled (GMX_OPENMP_MAX_THREADS = 64)
GPU support:        CUDA
SIMD instructions:  AVX2_256
FFT library:        fftw-3.3.8-sse2-avx-avx2-avx2_128-avx512
RDTSCP usage:       enabled
TNG support:        enabled
Hwloc support:      hwloc-1.11.0
Tracing support:    disabled
Built on:           2018-08-01 09:03:03

GPU info:
    Number of GPUs detected: 3
    #0: NVIDIA Tesla V100-PCIE-32GB, compute cap.: 7.0, ECC: yes, stat: compatible
    #1: NVIDIA Tesla V100-PCIE-32GB, compute cap.: 7.0, ECC: yes, stat: compatible
    #2: NVIDIA Tesla V100-PCIE-32GB, compute cap.: 7.0, ECC: yes, stat: compatible
___________________________________________________________________

These are the launched commands on two different system size and related performances:

Command

ns/day

h/ns

Alchol Dehydrogenase system (95561 atoms)

gmx mdrun -deffnm topol -nb gpu -pme gpu -ntmpi 1 -ntomp 12

53.355

0.45

gmx mdrun -deffnm topol -nb gpu -pme gpu -ntmpi 2 -ntomp 12 -npme 1 -gputasks 01

53.176

0.451

gmx mdrun -deffnm topol -nb gpu -pme gpu -ntmpi 3 -ntomp 12 -npme 1 -gputasks 012

50.024

0.48

Villin system (4723 atoms)

gmx mdrun -deffnm topol -nb gpu -pme gpu -ntmpi 1 -ntomp 12

589.635

0.041

gmx mdrun -deffnm topol -nb gpu -pme gpu -ntmpi 2 -ntomp 12 -npme 1 -gputasks 01

727.139

0.033

gmx mdrun -deffnm topol -nb gpu -pme gpu -ntmpi 3 -ntomp 12 -npme 1 -gputasks 012

664.695

0.036


Despite the performances seems very strange, because by increasing the number of GPUs in a big system I can't seeing a speedup, while in a small system the performance's peak is reached with 2 GPU,
can I ask to all of you if I'm using the GPU's selection options in the right way?

Moreover, I'm not sure about the right usage of -ntomp option, I thought to ask in another session.

Thanks a lot!
Carmine

-----------

CONFIDENTIALITY NOTICE
 
This message and its attachments are addressed solely to the persons
above and may contain confidential information. If you have received
the message in error, be informed that any use of the content hereof
is prohibited. Please return it immediately to the sender and delete
the message.
 

        Thank you