[gmx-users] Running gmx-4.6.x over multiple homogeneous nodes with GPU acceleration

[João Henriques]

Dear all,

Since gmx-4.6 came out, I've been particularly interested in taking
advantage of the native GPU acceleration for my simulations. Luckily, I
have access to a cluster with the following specs PER NODE:

CPU
2 E5-2650 (2.0 Ghz, 8-core)

GPU
2 Nvidia K20

I've become quite familiar with the "heterogenous parallelization" and
"multiple MPI ranks per GPU" schemes on a SINGLE NODE. Everything works
fine, no problems at all.

Currently, I'm working with a nasty system comprising 608159 tip3p water
molecules and it would really help to accelerate things up a bit.
Therefore, I would really like to try to parallelize my system over
multiple nodes and keep the GPU acceleration.

I've tried many different command combinations, but mdrun seems to be blind
towards the GPUs existing on other nodes. It always finds GPUs #0 and #1 on
the first node and tries to fit everything into these, completely
disregarding the existence of the other GPUs on the remaining requested
nodes.

Once again, note that all nodes have exactly the same specs.

Literature on the official gmx website is not, well... you know... in-depth
and I would really appreciate if someone could shed some light into this
subject.

Thank you,
Best regards,

-- 
João Henriques