[gmx-users] Gromacs 2018 and GPU PME

[Alex]

Mark,

You're right, sorry. I actually mixed up -nt (total number of threads) 
with -ntomp (per rank),  lack of sleep will do that to me sometimes.

> Yes. The -ntomp, -nt, and -ntmpi options have always been different from
> each other, and still work as they always did. See
> http://manual.gromacs.org/documentation/2018-latest/user-guide/mdrun-performance.html.
> There are 8 ranks. You specified one to do PME work, and for all PP work
> and all PME work to go on GPUs, and that 7 of the GPU tasks go on GPU 0 and
> one on GPU 1. Please check out the docs for each option rather than guess
> :-)
>
Yup, i will go through it again. -gputasks appears to provide much more 
control over what specifically happens on the GPU side, so i'll try my 
best to use it without guessing. Feel free to tell me to RTFM anytime. ;)

Thanks,

Alex

>
>
> On 2/9/2018 8:25 AM, Szilárd Páll wrote:
>>> Hi,
>>>
>>> First of all,have you read the docs (admittedly somewhat brief):
>>>
>> http://manual.gromacs.org/documentation/2018/user-guide/mdrun-performance.html#types-of-gpu-tasks
>>> The current PME GPU was optimized for single-GPU runs. Using multiple
>> GPUs
>>> with PME offloaded works, but this mode hasn't been an optimization
>> target
>>> and it will often not give very good performance. Using multiple GPUs
>>> requires a separate PME rank (as you have realized), only one can be used
>>> (as we don't support PME decomposition on GPUs) and it comes some
>> inherent
>>> scaling drawbacks. For this reason, unless you _need_ your single run to
>> be
>>> as fast as possible, you'll be better off running multiple simulations
>>> side-by side.
>>>
>>> A few tips for tuning the performance of a multi-GPU run with PME
>> offload:
>>> * expect to get at best 1.5 scaling to 2 GPUs (rarely 3 if the tasks
>> allow)
>>> * generally it's best to use about the same decomposition that you'd use
>>> with nonbonded-only offload, e.g. in your case 6-8 ranks
>>> * map the GPU task alone or at most together with 1 PP rank to a GPU,
>> i.e.
>>> use the new -gputasks option
>>> e.g. for your case I'd expect the following to work ~best:
>>> gmx mdrun -v -deffnm md -pme gpu -nb gpu -ntmpi 8 -ntomp 6 -npme 1
>>> -gputasks 00000001
>>> or
>>> gmx mdrun -v -deffnm md -pme gpu -nb gpu -ntmpi 8 -ntomp 6 -npme 1
>>> -gputasks 00000011
>>>
>>>
>>> Let me know if that gave some improvement.
>>>
>>> Cheers,
>>>
>>> --
>>> Szilárd
>>>
>>> On Fri, Feb 9, 2018 at 8:51 AM, Gmx QA <gmxquestions at gmail.com> wrote:
>>>
>>>> Hi list,
>>>>
>>>> I am trying out the new gromacs 2018 (really nice so far), but have a
>> few
>>>> questions about what command line options I should specify, specifically
>>>> with the new gnu pme implementation.
>>>>
>>>> My computer has two CPUs (with 12 cores each, 24 with hyper threading)
>> and
>>>> two GPUs, and I currently (with 2018) start simulations like this:
>>>>
>>>> $ gmx mdrun -v -deffnm md -pme gpu -nb gpu -ntmpi 2 -npme 1 -ntomp 24
>>>> -gpu_id 01
>>>>
>>>> this works, but gromacs prints the message that 24 omp threads per mpi
>> rank
>>>> is likely inefficient. However, trying to reduce the number of omp
>> threads
>>>> I see a reduction in performance. Is this message no longer relevant
>> with
>>>> gpu pme or am I overlooking something?
>>>>
>>>> Thanks
>>>> /PK
>>>> --
>>>> Gromacs Users mailing list
>>>>
>>>> * Please search the archive at http://www.gromacs.org/
>>>> Support/Mailing_Lists/GMX-Users_List before posting!
>>>>
>>>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>>>>
>>>> * For (un)subscribe requests visit
>>>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
>>>> send a mail to gmx-users-request at gromacs.org.
>>>>
>> --
>> Gromacs Users mailing list
>>
>> * Please search the archive at
>> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
>> posting!
>>
>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>>
>> * For (un)subscribe requests visit
>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
>> send a mail to gmx-users-request at gromacs.org.