[gmx-users] Re: MPI scaling (was RE: MPI tips)

[David Mathog]

Lubos Vrbka wrote:

> David Mathog wrote:
> > I don't really understand your reasoning for increasing the number
> > of steps since the nodes have to do some sort of communication at
> > each step, and whatever the ratio of that time to computation time
> > per step is, it shouldn't vary with the number of steps.
> in my opinion, the overhead mentioned above is the problem. imagine the 
> following situation:
> 
> 1) simulation setup
> 	takes 30 seconds (allocation of resources, initial creation of pair 
> lists, fft optimization, lincs initialization, or whatever else might 
> happen)
> 2) simulation itself
> 	takes 30 seconds on 1 processor
> 	takes 15 seconds on 2 processors
> 
> total 4:3 (30+30)/(30+15) instead of 'real' scaling 2:1 (30:15).

Interesting theory but not one that can be easily tested since the
programs don't emit "set up took N seconds" and "processing took M
seconds".   There's another chunk of the program which should
probably also count as "set up" which is the time needed at
each step for the nodes to reconcile their chunks of the model
before the next calculation.  (My guess is that this last part doesn't
scale well at all.)  Perhaps you're counting this second thing in
your setup?  In any case, I know that the _initial_ setup before the
steps run doesn't take all that long, since tail -f on the log file
shows the steps after not too many seconds, so the initial setup
is not a key factor.  The per step (re)setup can't be measured but
could be more significant.

Again, I don't think it's actually any of these since the CPU time on
the compute nodes falls down towards zero.  If they were setting up they
would still be consuming CPU time. It looks a lot more like most of
the nodes stall while waiting for data before they can proceed
to the next step.

mdrun doesn't seem to have any command line options to turn on timing
measurements.  Is there maybe a build option that can do this?

Regards,

David Mathog
mathog at caltech.edu
Manager, Sequence Analysis Facility, Biology Division, Caltech