[gmx-users] [Performance] poor performance with NV V100

[Jimmy Chen]

Hi,

I'm using NV v100 to evaluate if it's suitable to do purchase.
But I can't get similar test result as referenced performance data
which was got from internet.
https://developer.nvidia.com/hpc-application-performance
https://www.hpc.co.jp/images/pdf/benchmark/Molecular-Dynamics-March-2018.pdf


No matter using docker tag 18.02 from
https://ngc.nvidia.com/catalog/containers/hpc:gromacs/tags

or gromacs source code from
ftp://ftp.gromacs.org/pub/gromacs/gromacs-2019.3.tar.gz

test data set is ADH dodec and water 1.5M
gmx grompp -f pme_verlet.mdp
gmx mdrun -ntmpi 1 -nb gpu -pin on -v -noconfout -nsteps 5000 -s topol.tpr
-ntomp 4
and  gmx mdrun -ntmpi 2 -nb gpu -pin on -v -noconfout -nsteps 5000 -s
topol.tpr -ntomp 4

My CPU is Intel(R) Xeon(R) Gold 6142 CPU @ 2.60GHz
and GPU is NV V100 16GB PCIE.

For ADH dodec,
The perf data of 2xV100 16GB PCIE in
https://developer.nvidia.com/hpc-application-performance is 176 (ns/day).
But I only can get 28 (ns/day). actually I can get 67(ns/day) with 1xV100.
I don't know why I got poorer result with 2xV100.

For water 1.5M
The perf data of 1xV100 16GB PCIE in
https://www.hpc.co.jp/images/pdf/benchmark/Molecular-Dynamics-March-2018.pdf is
9.83(ns/day) and 2xV100 is 10.41(ns/day).
But what I got is 6.5(ns/day) with 1xV100 and 2(ns/day) with 2xV100.

Could anyone give me some suggestions about how to clarify what's problem
to result to this perf data in my environment? Is my command to perform the
testing wrong? any suggested command to perform the testing?
or which source code version is recommended to use now?

btw, after checking the code, it seems MPI doesn't go through PCIE P2p or
RDMA, is it correct? any plan to implement this in MPI?

Best regards,
Jimmy