[gmx-users] Can we set the number of pure PME nodes when using GPU&CPU?

Szilárd Páll pall.szilard at gmail.com
Mon Aug 25 15:44:40 CEST 2014 

On Mon, Aug 25, 2014 at 8:08 AM, Mark Abraham <mark.j.abraham at gmail.com> wrote:
> On Mon, Aug 25, 2014 at 5:01 AM, Theodore Si <sjyzhxw at gmail.com> wrote:
>
>> Hi,
>>
>> https://onedrive.live.com/redir?resid=990FCE59E48164A4!
>> 2572&authkey=!AP82sTNxS6MHgUk&ithint=file%2clog
>> https://onedrive.live.com/redir?resid=990FCE59E48164A4!
>> 2482&authkey=!APLkizOBzXtPHxs&ithint=file%2clog
>>
>> These are 2 log files. The first one  is using 64 cpu cores(64 / 16 = 4
>> nodes) and 4nodes*2 = 8 GPUs, and the second is using 512 cpu cores, no GPU.
>> When we look at the 64 cores log file, we find that in the  R E A L   C Y
>> C L E   A N D   T I M E   A C C O U N T I N G table, the total wall time is
>> the sum of every line, that is 37.730=2.201+0.082+...+1.150. So we think
>> that when the CPUs is doing PME, GPUs are doing nothing. That's why we say
>> they are working sequentially.
>>
>
> Please note that "sequential" means "one phase after another." Your log
> files don't show the timing breakdown for the GPUs, which is distinct from
> showing that the GPUs ran and then the CPUs ran (which I don't think the
> code even permits!). References to "CUDA 8x8 kernels" do show the GPU was
> active. There was an issue with mdrun not always being able to gather and
> publish the GPU timing results; I don't recall the conditions (Szilard
> might remember), but it might be fixed in a later release.

It is a limitation (well, I'd say borderline bug) in CUDA that if you
have multiple work-queues (=streams), reliable timing using the CUDA
built-in mechanisms is impossible. There may be a way to work around
this, but that won't happen in the current versions. What's important
is to observe the wait time on the CPU sideand of course, if the Op is
profiling this is not an issue.

> In any case, you
> should probably be doing performance optimization on a GROMACS version that
> isn't a year old.
>
> I gather that you didn't actually observe the GPUs idle - e.g. with a
> performance monitoring tool? Otherwise, and in the absence of a description
> of your simulation system, I'd say that log file looks somewhere between
> normal and optimal. For the record, for better performance, you should
> probably be following the advice of the install guide and not compiling
> FFTW with AVX support, and using one of the five gcc minor versions
> released since 4.4 ;-)

And besides avoiding ancient gcc versions, I suggest using CUDA 5.5
(which you can use because you have version 5.5 driver which I see in
your log file):

Additionally, I suggest avoiding MKL and using FFTW instead. For the
grid sizes of our interest all benchmarks I did in the past showed
considerably higher FFTW performance. Same goes for icc, but feel free
to benchmark and please report back if you find the opposite.

> As for the 512 cores log file, the total wall time is approximately the sum
>> of PME mesh and PME wait for PP. We think this is because PME-dedicated
>> nodes finished early, and the total wall time is the time spent on PP
>> nodes, therefore time spent on PME is covered.
>
>
> Yes, using an offload model makes it awkward to report CPU timings, because
> there are two kinds of CPU ranks. The total of the "Wall t" column adds up
> to twice the total time taken (which is noted explicitly in more recent
> mdrun versions). By design, the PME ranks do finish early, as you know from
> Figure 3.16 of the manual. As you can see in the table, the PP ranks spend
> 26% of their time waiting for the results from the PME ranks, and this is
> the origin of the note (above the table) that you might want to balance
> things better.
>
> Mark
>
> On 8/23/2014 9:30 PM, Mark Abraham wrote:
>>
>>> On Sat, Aug 23, 2014 at 1:47 PM, Theodore Si <sjyzhxw at gmail.com> wrote:
>>>
>>>  Hi,
>>>>
>>>> When we used 2 GPU nodes (each has 2 cpus and 2 gpus) to do a mdrun(with
>>>> no PME-dedicated node), we noticed that when CPU are doing PME, GPU are
>>>> idle,
>>>>
>>>
>>> That could happen if the GPU completes its work too fast, in which case
>>> the
>>> end of the log file will probably scream about imbalance.
>>>
>>> that is they are doing their work sequentially.
>>>
>>>
>>> Highly unlikely, not least because the code is written to overlap the
>>> short-range work on the GPU with everything else on the CPU. What's your
>>> evidence for *sequential* rather than *imbalanced*?
>>>
>>>
>>>  Is it supposed to be so?
>>>>
>>>
>>> No, but without seeing your .log files, mdrun command lines and knowing
>>> about your hardware, there's nothing we can say.
>>>
>>>
>>>  Is it the same reason as GPUs on PME-dedicated nodes won't be used during
>>>> a run like you said before?
>>>>
>>>
>>> Why would you suppose that? I said GPUs do work from the PP ranks on their
>>> node. That's true here.
>>>
>>> So if we want to exploit our hardware, we should map PP-PME ranks
>>> manually,
>>>
>>>> right? Say, use one node as PME-dedicated node and leave the GPUs on that
>>>> node idle, and use two nodes to do the other stuff. How do you think
>>>> about
>>>> this arrangement?
>>>>
>>>>  Probably a terrible idea. You should identify the cause of the
>>> imbalance,
>>> and fix that.
>>>
>>> Mark
>>>
>>>
>>>  Theo
>>>>
>>>>
>>>> On 8/22/2014 7:20 PM, Mark Abraham wrote:
>>>>
>>>>  Hi,
>>>>>
>>>>> Because no work will be sent to them. The GPU implementation can
>>>>> accelerate
>>>>> domains from PP ranks on their node, but with an MPMD setup that uses
>>>>> dedicated PME nodes, there will be no PP ranks on nodes that have been
>>>>> set
>>>>> up with only PME ranks. The two offload models (PP work -> GPU; PME work
>>>>> ->
>>>>> CPU subset) do not work well together, as I said.
>>>>>
>>>>> One can devise various schemes in 4.6/5.0 that could use those GPUs, but
>>>>> they either require
>>>>> * each node does both PME and PP work (thus limiting scaling because of
>>>>> the
>>>>> all-to-all for PME, and perhaps making poor use of locality on
>>>>> multi-socket
>>>>> nodes), or
>>>>> * that all nodes have PP ranks, but only some have PME ranks, and the
>>>>> nodes
>>>>> map their GPUs to PP ranks in a way that is different depending on
>>>>> whether
>>>>> PME ranks are present (which could work well, but relies on the DD
>>>>> load-balancer recognizing and taking advantage of the faster progress of
>>>>> the PP ranks that have better GPU support, and requires that you get
>>>>> very
>>>>> dirty hands laying out PP and PME ranks onto hardware that will later
>>>>> match
>>>>> the requirements of the DD load balancer, and probably that you balance
>>>>> PP-PME load manually)
>>>>>
>>>>> I do not recommend the last approach, because of its complexity.
>>>>>
>>>>> Clearly there are design decisions to improve. Work is underway.
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Mark
>>>>>
>>>>>
>>>>> On Fri, Aug 22, 2014 at 10:11 AM, Theodore Si <sjyzhxw at gmail.com>
>>>>> wrote:
>>>>>
>>>>>   Hi Mark,
>>>>>
>>>>>> Could you tell me why that when we are GPU-CPU nodes as PME-dedicated
>>>>>> nodes, the GPU on such nodes will be idle?
>>>>>>
>>>>>>
>>>>>> Theo
>>>>>>
>>>>>> On 8/11/2014 9:36 PM, Mark Abraham wrote:
>>>>>>
>>>>>>   Hi,
>>>>>>
>>>>>>> What Carsten said, if running on nodes that have GPUs.
>>>>>>>
>>>>>>> If running on a mixed setup (some nodes with GPU, some not), then
>>>>>>> arranging
>>>>>>> your MPI environment to place PME ranks on CPU-only nodes is probably
>>>>>>> worthwhile. For example, all your PP ranks first, mapped to GPU nodes,
>>>>>>> then
>>>>>>> all your PME ranks, mapped to CPU-only nodes, and then use mdrun
>>>>>>> -ddorder
>>>>>>> pp_pme.
>>>>>>>
>>>>>>> Mark
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Aug 11, 2014 at 2:45 AM, Theodore Si <sjyzhxw at gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>    Hi Mark,
>>>>>>>
>>>>>>>  This is information of our cluster, could you give us some advice as
>>>>>>>> regards to our cluster so that we can make GMX run faster on our
>>>>>>>> system?
>>>>>>>>
>>>>>>>> Each CPU node has 2 CPUs and each GPU node has 2 CPUs and 2 Nvidia
>>>>>>>> K20M
>>>>>>>>
>>>>>>>>
>>>>>>>> Device Name     Device Type     Specifications  Number
>>>>>>>> CPU Node        IntelH2216JFFKRNodes    CPU: 2×Intel Xeon E5-2670(8
>>>>>>>> Cores,
>>>>>>>> 2.6GHz, 20MB Cache, 8.0GT)
>>>>>>>> Mem: 64GB(8×8GB) ECC Registered DDR3 1600MHz Samsung Memory     332
>>>>>>>> Fat Node        IntelH2216WPFKRNodes    CPU: 2×Intel Xeon E5-2670(8
>>>>>>>> Cores,
>>>>>>>> 2.6GHz, 20MB Cache, 8.0GT)
>>>>>>>> Mem: 256G(16×16G) ECC Registered DDR3 1600MHz Samsung Memory    20
>>>>>>>> GPU Node        IntelR2208GZ4GC         CPU: 2×Intel Xeon E5-2670(8
>>>>>>>> Cores,
>>>>>>>> 2.6GHz, 20MB Cache, 8.0GT)
>>>>>>>> Mem: 64GB(8×8GB) ECC Registered DDR3 1600MHz Samsung Memory     50
>>>>>>>> MIC Node        IntelR2208GZ4GC         CPU: 2×Intel Xeon E5-2670(8
>>>>>>>> Cores,
>>>>>>>> 2.6GHz, 20MB Cache, 8.0GT)
>>>>>>>> Mem: 64GB(8×8GB) ECC Registered DDR3 1600MHz Samsung Memory     5
>>>>>>>> Computing Network Switch        Mellanox Infiniband FDR Core Switch
>>>>>>>> 648× FDR Core Switch MSX6536-10R, Mellanox Unified Fabric Manager
>>>>>>>>   1
>>>>>>>> Mellanox SX1036 40Gb Switch     36× 40Gb Ethernet Switch SX1036, 36×
>>>>>>>> QSFP
>>>>>>>> Interface     1
>>>>>>>> Management Network Switch       Extreme Summit X440-48t-10G 2-layer
>>>>>>>> Switch
>>>>>>>> 48× 1Giga Switch Summit X440-48t-10G, authorized by ExtremeXOS
>>>>>>>>  9
>>>>>>>> Extreme Summit X650-24X 3-layer Switch  24× 10Giga 3-layer Ethernet
>>>>>>>> Switch
>>>>>>>> Summit X650-24X, authorized by ExtremeXOS    1
>>>>>>>> Parallel Storage        DDN Parallel Storage System     DDN SFA12K
>>>>>>>> Storage
>>>>>>>> System       1
>>>>>>>> GPU     GPU Accelerator         NVIDIA Tesla Kepler K20M        70
>>>>>>>> MIC     MIC     Intel Xeon Phi 5110P Knights Corner     10
>>>>>>>> 40Gb Ethernet Card      MCX314A-BCBT    Mellanox ConnextX-3 Chip 40Gb
>>>>>>>> Ethernet Card
>>>>>>>> 2× 40Gb Ethernet ports, enough QSFP cables      16
>>>>>>>> SSD     Intel SSD910    Intel SSD910 Disk, 400GB, PCIE  80
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 8/10/2014 5:50 AM, Mark Abraham wrote:
>>>>>>>>
>>>>>>>>    That's not what I said.... "You can set..."
>>>>>>>>
>>>>>>>>  -npme behaves the same whether or not GPUs are in use. Using
>>>>>>>>> separate
>>>>>>>>> ranks
>>>>>>>>> for PME caters to trying to minimize the cost of the all-to-all
>>>>>>>>> communication of the 3DFFT. That's still relevant when using GPUs,
>>>>>>>>> but
>>>>>>>>> if
>>>>>>>>> separate PME ranks are used, any GPUs on nodes that only have PME
>>>>>>>>> ranks
>>>>>>>>> are
>>>>>>>>> left idle. The most effective approach depends critically on the
>>>>>>>>> hardware
>>>>>>>>> and simulation setup, and whether you pay money for your hardware.
>>>>>>>>>
>>>>>>>>> Mark
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Sat, Aug 9, 2014 at 2:56 AM, Theodore Si <sjyzhxw at gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>     Hi,
>>>>>>>>>
>>>>>>>>>   You mean no matter we use GPU acceleration or not, -npme is just a
>>>>>>>>>
>>>>>>>>>> reference?
>>>>>>>>>> Why we can't set that to a exact value?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 8/9/2014 5:14 AM, Mark Abraham wrote:
>>>>>>>>>>
>>>>>>>>>>     You can set the number of PME-only ranks with -npme. Whether
>>>>>>>>>> it's
>>>>>>>>>> useful
>>>>>>>>>>
>>>>>>>>>>   is
>>>>>>>>>>
>>>>>>>>>>> another matter :-) The CPU-based PME offload and the GPU-based PP
>>>>>>>>>>> offload
>>>>>>>>>>> do not combine very well.
>>>>>>>>>>>
>>>>>>>>>>> Mark
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Aug 8, 2014 at 7:24 AM, Theodore Si <sjyzhxw at gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>      Hi,
>>>>>>>>>>>
>>>>>>>>>>>    Can we set the number manually with -npme when using GPU
>>>>>>>>>>>
>>>>>>>>>>>  acceleration?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Gromacs Users mailing list
>>>>>>>>>>>>
>>>>>>>>>>>> * Please search the archive at http://www.gromacs.org/
>>>>>>>>>>>> Support/Mailing_Lists/GMX-Users_List before posting!
>>>>>>>>>>>>
>>>>>>>>>>>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>>>>>>>>>>>>
>>>>>>>>>>>> * For (un)subscribe requests visit
>>>>>>>>>>>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_
>>>>>>>>>>>> gmx-users
>>>>>>>>>>>> or
>>>>>>>>>>>> send a mail to gmx-users-request at gromacs.org.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>     --
>>>>>>>>>>>>
>>>>>>>>>>>>   Gromacs Users mailing list
>>>>>>>>>>>>
>>>>>>>>>>> * Please search the archive at http://www.gromacs.org/
>>>>>>>>>> Support/Mailing_Lists/GMX-Users_List before posting!
>>>>>>>>>>
>>>>>>>>>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>>>>>>>>>>
>>>>>>>>>> * For (un)subscribe requests visit
>>>>>>>>>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users
>>>>>>>>>> or
>>>>>>>>>> send a mail to gmx-users-request at gromacs.org.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>    --
>>>>>>>>>>
>>>>>>>>>>  Gromacs Users mailing list
>>>>>>>>>
>>>>>>>> * Please search the archive at http://www.gromacs.org/
>>>>>>>> Support/Mailing_Lists/GMX-Users_List before posting!
>>>>>>>>
>>>>>>>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>>>>>>>>
>>>>>>>> * For (un)subscribe requests visit
>>>>>>>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users
>>>>>>>> or
>>>>>>>> send a mail to gmx-users-request at gromacs.org.
>>>>>>>>
>>>>>>>>
>>>>>>>>   --
>>>>>>>>
>>>>>>> Gromacs Users mailing list
>>>>>>
>>>>>> * Please search the archive at http://www.gromacs.org/
>>>>>> Support/Mailing_Lists/GMX-Users_List before posting!
>>>>>>
>>>>>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>>>>>>
>>>>>> * For (un)subscribe requests visit
>>>>>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
>>>>>> send a mail to gmx-users-request at gromacs.org.
>>>>>>
>>>>>>
>>>>>>  --
>>>> Gromacs Users mailing list
>>>>
>>>> * Please search the archive at http://www.gromacs.org/
>>>> Support/Mailing_Lists/GMX-Users_List before posting!
>>>>
>>>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>>>>
>>>> * For (un)subscribe requests visit
>>>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
>>>> send a mail to gmx-users-request at gromacs.org.
>>>>
>>>>
>>>>
>>>>
>>
>>
> --
> Gromacs Users mailing list
>
> * Please search the archive at http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a mail to gmx-users-request at gromacs.org.

More information about the gromacs.org_gmx-users mailing list