[gmx-users] PME on BlueGene

[Mark Abraham]

Jakob Wohlert wrote:
> Mark Abraham wrote:
>> Jakob Wohlert wrote:
>>> Hi,
>>>
>>> I'm trying to compile Gromacs 4.0.4 for BlueGene/P, and using the
>>> configuration options from the wiki I have succeeded insofar that I 
>>> have a
>>> working program as long as I don't use PME.
>>>
>>> I have tried many different variants of fftw - 2.1.5, 3.2.1, single
>>> precision, double precision, different compiler optimizations and so on,
>>> but it all ends the same: mdrun getting stuck somewhere in the
>>> initialization process.
>>>
>>> However, by using the built in fft library FFTPACK instead of FFTW, PME
>>> will work, but that is not really an alternative.
>>>
>>> In at least a few cases I have been able to pinpoint the location 
>>> where it
>>> hangs - it's in pme.c, subroutine pme_dd_sendrecv. The program calls
>>> MPI_Sendrecv, but then nothing else happens as far as I can tell.
>>>
>>> I'm confused and I have sort of ran out of ideas right now. Has anyone
>>> else encountered a problem like this, or has anyone any suggestions 
>>> how to
>>> proceed from here?
> Thanks for your answer!
> 
>> That looks to me like the separate PME nodes are dying through some 
>> linking problem and the problem is only manifest on node 0 when its 
>> sendrecv doesn't complete. Forcing mdrun -npme 0 may confirm this when 
>> all the nodes die at the first point they refer to a symbol in the FFT 
>> library.
>>
>> Otherwise, looking at warnings/errors from the linker will be required. 
> Ok, can you be a little more specific? Do you mean when compiling fftw, 
> gromacs or both? I'm not very experienced with these kind of things.

GROMACS links with the FFTW library, so there ought to be warnings of 
potential problems at link time, i.e. the final stage of a GROMACS "make 
mdrun" (having started with a clean configuration, either freshly 
unpacked or after "make clean"). The last 100 or so lines of the output 
of "make mdrun" should have any relevant data.

>> Are you compiling an FFT library version for the back end, or the 
>> front end?
> I'm trying to get it to work on the back end, on the front end it works 
> fine! (So, I have fftw libraries for both).

My BG/L FFTW-3.2 configure line is

../configure --prefix /hpc/home/mja163/progs CC=blrts_xlc --host=powerpc 
--build=ppc64 CFLAGS=-qbgl -qarch=440d -qtune=440 -qnoautoconfig -O5 
--disable-fortran --enable-float

but you may need to tweak that for BG/P for all I know. Let me know 
when/if something works and I'll update the wiki.

Mark