[gmx-developers] gromacs software development

[Erik Lindahl]

Hi,

"I'm back", as certain governators would say :-)

First, I think we should separate the issue of domain decomposition  
from FFT, so the first task would be to create a truly parallel 3D FFT.

Currently, the algorithm I implemented is essentially the same as  
used in FFTW2, where the grid is only decomposed along the x-axis.  
This is bad for two reasons: (i) it limits the number of nodes we can  
use for PME, and (ii) even when the direct-space system has beautiful  
balanced domain decomposition they have to send coordinates to a more  
or less orthogonal x-axis decomposition in fourier space.

What we'd like is simply an algorithm that decomposed FFTs in 1D/2D/ 
3D. This is not rocket science in terms of mathematics, but requires  
a bit of juggling with data and communication to make sure things end  
up in the right place. There is a pretty good but superficial paper  
by Eleftheriou that describes the high-level approach, but  
essentially it's a matter of:

1. Starting with a 3D domain decomposed grid
2. Communicate so each node has whole arrays e.g. along the z axis
3. Do 1D FFTs
4. Communicate/transponse so each node has arrays e.g. along the y axis
3. Do 1D FFTs
4. Communicate/transponse so each node has arrays along the last axis
5. Don't communicate more, do the convolution in transposed coordinates
6. Repeat everything backwards.

I do think the "2 weeks" estimate is very optimistic, though - I'd  
guess a month for the basic implementation, and a bit more time for  
tweaking to minimize communication!

Cheers,

Erik



On Jun 29, 2007, at 3:08 PM, David van der Spoel wrote:

> Carsten Kutzner wrote:
>> Hi,
>> I would be happy to contribute to any of the projects, especially  
>> if they address
>> the scaling of mdrun. In this respect the 3D FFT decomposition  
>> seems most promising
>> to me (apart from the fast multipole solver ;) David, could you  
>> expand a bit on
>> the 3d FFT topic? I guess you refer to a Daresbury/DAFT-style  
>> algorithm?
>
> I think Erik knows a bit more about this, but he's traveling. It  
> would be an algorithm that does a true 3d decomposition of the PME  
> grid, to allow for using more PME nodes (right now this is limited  
> to the number of grid cells in the x dimension, which means you can  
> not use more than three or four times this number of nodes).
>> Another question that I have is, we need an extra option in the  
>> pull code which
>> allows us to rotate a group of atoms around a given axis. I would  
>> now start to
>> implement this unless some code is already around that we could use.
> I am not aware of this. Maybe Berk or Justin knows?
>
>> Carsten
>> _______________________________________________
>> gmx-developers mailing list
>> gmx-developers at gromacs.org
>> http://www.gromacs.org/mailman/listinfo/gmx-developers
>> Please don't post (un)subscribe requests to the list. Use the www  
>> interface or send it to gmx-developers-request at gromacs.org.
>
>
> -- 
> David.
> ______________________________________________________________________ 
> __
> David van der Spoel, PhD, Assoc. Prof., Molecular Biophysics group,
> Dept. of Cell and Molecular Biology, Uppsala University.
> Husargatan 3, Box 596,  	75124 Uppsala, Sweden
> phone:	46 18 471 4205		fax: 46 18 511 755
> spoel at xray.bmc.uu.se	spoel at gromacs.org   http://folding.bmc.uu.se
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 
> ++++
> _______________________________________________
> gmx-developers mailing list
> gmx-developers at gromacs.org
> http://www.gromacs.org/mailman/listinfo/gmx-developers
> Please don't post (un)subscribe requests to the list. Use the www  
> interface or send it to gmx-developers-request at gromacs.org.