[gmx-developers] Collect all coordinates/velocities/virial in one MPI rank in mdlib/constr.cpp::Impl::apply()

[Erik Lindahl]

Hi,

Think algorithm first, implementation later.

Hopefully the reason for never collecting all data on a single node is
clear. We don't even calculate the temperature at each timestep when
pushing scaling for this reason - the communication latency & bandwidth
will kill performance (apart from the memory requirements).

How to actually solve it depends on the algorithm. The nonbonded
interactions in gromacs are handled with a few tricks where we make sure
each node has roughly 1/8th of a sphere of all neighbors within the cutoff,
so we can calculate them on at least one (specific) node.

For constraints, the big challenge for parallelization is that there's
often some sort of iterative component, and trying to iterate an unspecific
number of times where each step requires communication and/or knowledge
about all connected atoms is again going to kill performance & scaling. For
p-LINCS this is adressed by a limited sphere of influence (meaning you only
need bond information up to a given number of bonds away), and then it's
usually most efficient to communicate that specifically. However, when
running in parallel it's usually better to increase the order
(corresponding to the sphere of influence) to avoid having to perform
multiple iterations.  For scaling, you basically need to think "computation
is free, communication kills you".

I think the central challenge for any new parallel constraints is that one
has to start with finding an algorithm that minimizes communication and
avoids iterations. Exactly what has to be communicated will be specific to
the algorithm, but you can likely learn a lot just by looking at how
p-LINCS finds what atoms are connected up to N bonds away.

Cheers,

Erik



On Tue, Jan 11, 2022 at 11:47 AM Pablo García Risueño <
garcia.risueno at gmail.com> wrote:

> Dear Prof Lindahl
>
> Thank you for your reply. Can you give us a clue on the second point that
> you mention? I guess that in the past the issue of transferring information
> to close neighboring nodes has been tackled by the Gromacs developers
> community.
>
> With best regards.
>
> El mar, 11 ene 2022 a las 11:30, Erik Lindahl (<erik.lindahl at gmail.com>)
> escribió:
>
>> Hi Lorién,
>>
>> On Tue, Jan 11, 2022 at 11:21 AM Lorién López Villellas <
>> lorien.lopez at bsc.es> wrote:
>>
>>> Hi all.
>>>
>>> As a first approach, we want to collect all the coordinates on the
>>> master rank, execute the solver, and send the updated results to the other
>>> ranks.
>>>
>>
>> Stop right there ;-)
>>
>> This approach is never going to scale, and you might as well stick with a
>> non-MPI implementation. Don't think of having two nodes, but imagine having
>> a molecule with ~500,000 atoms split over 100 chains that is run on 1000
>> nodes.  If you try to execute part of the problem on a single node you will
>> first be killed by the communication, and then by the master node having
>> 10x more work to do even if your algorithm is only 1% of the runtime
>> (meaning a 10x performance loss).
>>
>> To have any chance of getting impact for a new parallel constraints
>> algorithm (which is a VERY worthwhile effort per se), you need to find a
>> way to
>>
>> 1) Balance the computational load such that all nodes in the system can
>> help
>> 2) Find a way where you only need to communicate to close neighbors -
>> never ever collect all data on a single node every step.
>>
>>
>>
>> Cheers,
>>
>> Erik
>>
>>
>> --
>> Erik Lindahl <erik.lindahl at dbb.su.se>
>> Professor of Biophysics, Dept. Biochemistry & Biophysics, Stockholm
>> University
>> Science for Life Laboratory, Box 1031, 17121 Solna, Sweden
>>
>> Note: I frequently do email outside office hours because it is a
>> convenient time for me to write, but please do not interpret that as an
>> expectation for you to respond outside your work hours.
>>
>
>
> --
> --
>
> Dr. Pablo García Risueño
>


-- 
Erik Lindahl <erik.lindahl at dbb.su.se>
Professor of Biophysics, Dept. Biochemistry & Biophysics, Stockholm
University
Science for Life Laboratory, Box 1031, 17121 Solna, Sweden

Note: I frequently do email outside office hours because it is a convenient
time for me to write, but please do not interpret that as an expectation
for you to respond outside your work hours.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://maillist.sys.kth.se/pipermail/gromacs.org_gmx-developers/attachments/20220111/3beea93f/attachment-0003.html>