[gmx-users] Re: Gromacs-4.6 on two Titans GPUs

Szilárd Páll pall.szilard at gmail.com
Tue Nov 5 22:11:59 CET 2013 

Hi Dwey,

First and foremost, make sure to read the
http://www.gromacs.org/Documentation/Acceleration_and_parallelization
page, in particular the "Multiple MPI ranks per GPU" section which
applies in your case.

Secondly, please do post log files (pastebin is your friend), the
performance table at the end of the log tells much the performance
story and based on that I/we can make suggestions.

Using multiple GPU requires domain-decomposition which does have a
considerable overhead, especially comparing no DD with DD (i.e. 1 GPU
run with 2 GPU run). However, in your case I suspect that the
bottleneck is multi-threaded scaling on the AMD CPUs and you should
probably decrease the number of threads per MPI rank and share GPUs
between 2-4 ranks.

Regarding scaling across nodes, you can't expect much from gigabit
ethernet - especially not from the cheaper cards/switches, in my
experience even reaction field runs don't scale across nodes with 10G
ethernet if you have more than 4-6 ranks per node trying to
communicate (let alone with PME). However, on infiniband clusters we
have seen scaling to 100 atoms/core (at peak).

Cheers,
--
Szilárd

On Tue, Nov 5, 2013 at 9:29 PM, Dwey <mpi566 at gmail.com> wrote:
> Hi Mike,
>
>
>     I have similar configurations except a cluster of AMD-based linux
> platforms with 2 GPU cards.
>
> Your  suggestion works. However, the performance of 2 GPU  discourages
> me  because , for example,  with 1 GPU, our computer node can easily
> obtain a  simulation of 31ns/day for a protein of 300 amino acids but
> with 2 GPUs, it goes as far as 38 ns/day. I am very curious as to  why
>  the performance of 2 GPUs is under expectation. Is there any overhead
> that we should pay attention to ?  Note that these 2GPU cards are
> linked by a SLI bridge within the same node.
>
> Since the computer nodes of our cluster have at least one GPU  but
> they are connected by slow network cards ( 1GB/sec), unfortunately, I
> reasonably doubt that the performance will not be proportional to the
> total number of  GPU cards.  I am wondering if you have any suggestion
> about a cluster of GPU nodes.   For example, will a infiniband
> networking help increase a final performance when we execute a mpi
> task ? or what else ?  or forget about mpi and use single GPU instead.
>
> Any suggestion is highly appreciated.
> Thanks.
>
> Dwey
>
>> Date: Tue, 5 Nov 2013 16:20:39 +0100
>> From: Mark Abraham <mark.j.abraham at gmail.com>
>> Subject: Re: [gmx-users] Gromacs-4.6 on two Titans GPUs
>> To: Discussion list for GROMACS users <gmx-users at gromacs.org>
>> Message-ID:
>>         <CAMNuMASM5Ht40UB+unPpV7GMhQzXSb6psewMa+HbLV+GNb2k0g at mail.gmail.com>
>> Content-Type: text/plain; charset=ISO-8859-1
>>
>> On Tue, Nov 5, 2013 at 12:55 PM, James Starlight <jmsstarlight at gmail.com>wrote:
>>
>>> Dear Richard,
>>>
>>>
>>> 1)  mdrun -ntmpi 1 -ntomp 12 -gpu_id 0 -v  -deffnm md_CaM_test
>>> gave me performance about 25ns/day for the explicit solved system consisted
>>> of 68k atoms (charmm ff. 1.0 cutoofs)
>>>
>>> gaves slightly worse performation in comparison to the 1)
>>>
>>>
>> Richard suggested
>>
>> mdrun -ntmpi 2 -ntomp 6 -gpu_id 01 -v  -deffnm md_CaM_test,
>>
>> which looks correct to me. -ntomp 6 is probably superfluous
>>
>> Mark
>>
>>
>>> finally
>>>
>>> 3) mdrun -deffnm md_CaM_test
>>> running in the same regime as in the 2) so its also gave me 22ns/day for
>>> the same system.
>>>
>>> How the efficacy of using of dual-GPUs could be increased?
>>>
>>> James
>>>
>>>
>>> 2013/11/5 Richard Broadbent <richard.broadbent09 at imperial.ac.uk>
>>>
>>> > Dear James,
>>> >
>>> >
>>> > On 05/11/13 11:16, James Starlight wrote:
>>> >
>>> >> My suggestions:
>>> >>
>>> >> 1) During compilstion using -march=corei7-avx-i I have obtained error
>>> that
>>> >> somethng now found ( sorry I didnt save log) so I compile gromacs
>>> without
>>> >> this flag
>>> >>
>>> >> 2) I have twice as better performance using just 1 gpu by means of
>>> >>
>>> >> mdrun -ntmpi 1 -ntomp 12 -gpu_id 0 -v  -deffnm md_CaM_test
>>> >>
>>> >> than using of both gpus
>>> >>
>>> >> mdrun -ntmpi 2 -ntomp 12 -gpu_id 01 -v  -deffnm md_CaM_test
>>> >>
>>> >> in the last case I have obtained warning
>>> >>
>>> >> WARNING: Oversubscribing the available 12 logical CPU cores with 24
>>> >> threads.
>>> >>           This will cause considerable performance loss!
>>> >>
>>> >>  here you are requesting 2 thread mpi processes each with 12 openmp
>>> > threads, hence a total of 24 threads however even with hyper threading
>>> > enabled there are only 12 threads on your machine. Therefore, only
>>> allocate
>>> > 12. Try
>>> >
>>> > mdrun -ntmpi 2 -ntomp 6 -gpu_id 01 -v  -deffnm md_CaM_test
>>> >
>>> > or even
>>> >
>>> > mdrun -v  -deffnm md_CaM_test
>>> >
>>> > I believe it should autodetect the GPUs and run accordingly for details
>>> of
>>> > how to use gromacs with mpi/thread mpi openmp and GPUs see
>>> >
>>> > http://www.gromacs.org/Documentation/Acceleration_and_parallelization
>>> >
>>> > Which describes how to use these systems
>>> >
>>> > Richard
>>> >
>>> >
>>> >  How it could be fixed?
>>> >> All gpu are recognized correctly
>>> >>
>>> >>
>>> >> 2 GPUs detected:
>>> >>    #0: NVIDIA GeForce GTX TITAN, compute cap.: 3.5, ECC:  no, stat:
>>> >> compatible
>>> >>    #1: NVIDIA GeForce GTX TITAN, compute cap.: 3.5, ECC:  no, stat:
>>> >> compatible
>>> >>
>>> >>
>>> >> James
>>> >>
>>> >>
>>> >> 2013/11/4 Szilárd Páll <pall.szilard at gmail.com>
>>> >>
>>> >>  You can use the "-march=native" flag with gcc to optimize for the CPU
>>> >>> your are building on or e.g. -march=corei7-avx-i for Intel Ivy Bridge
>>> >>> CPUs.
>>> >>> --
>>> >>> Szilárd Páll
>>> >>>
>>> >>>
>>> >>> On Mon, Nov 4, 2013 at 12:37 PM, James Starlight <
>>> jmsstarlight at gmail.com
>>> >>> >
>>> >>> wrote:
>>> >>>
>>> >>>> Szilárd, thanks for suggestion!
>>> >>>>
>>> >>>> What kind of CPU optimisation should I take into account assumint that
>>> >>>>
>>> >>> I'm
>>> >>>
>>> >>>> using dual-GPU Nvidia TITAN workstation with 6 cores i7 (recognized as
>>> >>>> 12
>>> >>>> nodes in Debian).
>>> >>>>
>>> >>>> James
>>> >>>>
>>> >>>>
>>> >>>> 2013/11/4 Szilárd Páll <pall.szilard at gmail.com>
>>> >>>>
>>> >>>>  That should be enough. You may want to use the -march (or equivalent)
>>> >>>>> compiler flag for CPU optimization.
>>> >>>>>
>>> >>>>> Cheers,
>>> >>>>> --
>>> >>>>> Szilárd Páll
>>> >>>>>
>>> >>>>>
>>> >>>>> On Sun, Nov 3, 2013 at 10:01 AM, James Starlight <
>>> >>>>>
>>> >>>> jmsstarlight at gmail.com>
>>> >>>
>>> >>>> wrote:
>>> >>>>>
>>> >>>>>> Dear Gromacs Users!
>>> >>>>>>
>>> >>>>>> I'd like to compile lattest 4.6 Gromacs with native GPU supporting
>>> on
>>> >>>>>>
>>> >>>>> my
>>> >>>
>>> >>>> i7
>>> >>>>>
>>> >>>>>> cpu with dual GeForces Titans gpu mounted. With this config I'd like
>>> >>>>>>
>>> >>>>> to
>>> >>>
>>> >>>> perform simulations using cpu as well as both gpus simultaneously.
>>> >>>>>>
>>> >>>>>> What flags besides
>>> >>>>>>
>>> >>>>>> cmake .. -DGMX_GPU=ON -DCUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda-5.5
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> should I define to CMAKE for compiling optimized gromacs on such
>>> >>>>>>
>>> >>>>> workstation?
>>> >>>>>
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> Thanks for help
>>> >>>>>>
>>> >>>>>> James
>>> >>>>>> --
>>> >>>>>> gmx-users mailing list    gmx-users at gromacs.org
>>> >>>>>> http://lists.gromacs.org/mailman/listinfo/gmx-users
>>> >>>>>> * Please search the archive at
>>> >>>>>>
>>> >>>>> http://www.gromacs.org/Support/Mailing_Lists/Search before posting!
>>> >>>>>
>>> >>>>>> * Please don't post (un)subscribe requests to the list. Use the
>>> >>>>>> www interface or send it to gmx-users-request at gromacs.org.
>>> >>>>>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>>> >>>>>>
>>> >>>>> --
>>> >>>>> gmx-users mailing list    gmx-users at gromacs.org
>>> >>>>> http://lists.gromacs.org/mailman/listinfo/gmx-users
>>> >>>>> * Please search the archive at
>>> >>>>> http://www.gromacs.org/Support/Mailing_Lists/Search before posting!
>>> >>>>> * Please don't post (un)subscribe requests to the list. Use the
>>> >>>>> www interface or send it to gmx-users-request at gromacs.org.
>>> >>>>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>>> >>>>>
>>> >>>>>  --
>>> >>>> gmx-users mailing list    gmx-users at gromacs.org
>>> >>>> http://lists.gromacs.org/mailman/listinfo/gmx-users
>>> >>>> * Please search the archive at
>>> >>>>
>>> >>> http://www.gromacs.org/Support/Mailing_Lists/Search before posting!
>>> >>>
>>> >>>> * Please don't post (un)subscribe requests to the list. Use the
>>> >>>> www interface or send it to gmx-users-request at gromacs.org.
>>> >>>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>>> >>>>
>>> >>> --
>>> >>> gmx-users mailing list    gmx-users at gromacs.org
>>> >>> http://lists.gromacs.org/mailman/listinfo/gmx-users
>>> >>> * Please search the archive at
>>> >>> http://www.gromacs.org/Support/Mailing_Lists/Search before posting!
>>> >>> * Please don't post (un)subscribe requests to the list. Use the
>>> >>> www interface or send it to gmx-users-request at gromacs.org.
>>> >>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>>> >>>
>>> >>>  --
>>> > gmx-users mailing list    gmx-users at gromacs.org
>>> > http://lists.gromacs.org/mailman/listinfo/gmx-users
>>> > * Please search the archive at http://www.gromacs.org/
>>> > Support/Mailing_Lists/Search before posting!
>>> > * Please don't post (un)subscribe requests to the list. Use the www
>>> > interface or send it to gmx-users-request at gromacs.org.
>>> > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>>> >
>>> --
>>> gmx-users mailing list    gmx-users at gromacs.org
>>> http://lists.gromacs.org/mailman/listinfo/gmx-users
>>> * Please search the archive at
>>> http://www.gromacs.org/Support/Mailing_Lists/Search before posting!
>>> * Please don't post (un)subscribe requests to the list. Use the
>>> www interface or send it to gmx-users-request at gromacs.org.
>>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>>>
>>
>>
>> ------------------------------
>>
>> --
>> gmx-users mailing list
>> gmx-users at gromacs.org
>> http://lists.gromacs.org/mailman/listinfo/gmx-users
>> Please search the archive at http://www.gromacs.org/Support/Mailing_Lists/Search before posting!
>>
>> End of gmx-users Digest, Vol 115, Issue 22
>> ******************************************
> --
> gmx-users mailing list    gmx-users at gromacs.org
> http://lists.gromacs.org/mailman/listinfo/gmx-users
> * Please search the archive at http://www.gromacs.org/Support/Mailing_Lists/Search before posting!
> * Please don't post (un)subscribe requests to the list. Use the
> www interface or send it to gmx-users-request at gromacs.org.
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists

More information about the gromacs.org_gmx-users mailing list