[gmx-developers] MPI crash with ghost molecules

[Anton Feenstra]

Jason de Joannis wrote:
> Quoting Anton Feenstra <feenstra at few.vu.nl>:
> 
>> Jason de Joannis wrote:
>>> I start with a 128 DPPC bilayer. I also have 128 D8PC which I turn 
>>> off at the beginning of the simulation. When I say 'turn off' I mean 
>>> that the atom types are set to 13 (same as HW) and the charges are 
>>> zeroed. Therefore these molecules will only experience their own 
>>> intramolecular forces and will not exert any influence on the system. 
>>> They are probably absent from the neighbor lists too right?
>>>
>>> Now. During the simulation (in between move_x and ns) I pick a DPPC 
>>> molecule and turn it off while turning on a corresponding D8PC 
>>> molecule. The D8PC coordinates and velocities are mapped onto those 
>>> of the outgoing DPPC.
>>>
>>> This is successful in single-thread runs but fails in MPI np=2 mode. 
>>> For example after such a conversion on step 50, the LINCS constraints 
>>> diverge.
>>>
>>> Any thoughts?
>>
>> How are the coordinates of both molecules (in- and out-going) coupled?
>> If you copy over the coordinates (or something similar), but they end 
>> up on different CPU's in MPI, you may be reading the wrong data 
>> because not all CPU's have all coordinates all the time.
>>
> The problem goes away when I use -sort and -shuffle. I don't know why I 
> wasn't using them at first. But anyways, why does the system crash 
> without them? I know that they sort molecules by their X-coordinate and 
> distribute them to the nodes accordingly. In what case might one not 
> want to use them?

Sorting on X will increase the chance that both molecules are on the 
same CPU. So, I'd suggest looking carefully at the code that links the 
coordinates of both molecules. My guess is you're not reading the right 
ones from one molecule...

Not using -sort/-shuffle is general practice because it is done one-time 
(by grompp), so during simulation your x-sorting will 'decay', and your 
output (trajectories, gro) will be sorted and shuffled as well. This 
complicates analysis greatly. But if the performance gain is significant 
to you, it may be worth the trouble. (You could even re-sort every ns or 
so in a script by re-running grompp and re-starting mdrun, but I don't 
know of anyone who has gone that length. Gromacs 4 should solve all 
this, and more... ;-0 -- it'll also do your hair and the laundry!)


-- 
Groetjes,

Anton

  _____________ _______________________________________________________
|             |                                                       |
|  _   _  ___,| K. Anton Feenstra                                     |
| / \ / \'| | | IBIVU/Bioinformatics - Vrije Universiteit Amsterdam   |
|(   |   )| | | De Boelelaan 1083 - 1081A HV Amsterdam - Netherlands  |
| \_/ \_/ | | | Tel +31 20 59 87783 - Fax +31 20 59 87653 - Room P440 |
|             | Feenstra at few.vu.nl - www.few.vu.nl/~feenstra/         |
|             | "If You See Me Getting High, Knock Me Down" (RHCP)    |
|_____________|_______________________________________________________|