[gmx-developers] thread problem on AMD 12 core chips?

Tue Jan 17 09:23:44 CET 2012

On 2012-01-16 16:55, Sander Pronk wrote:
> The only reason the system I could imagine the call could fail would be a permissions problem.
> I've submitted a patch to release-4-5-patches that doesn't check for errors in setting thread affinity.
>

This seems to work!

Starting 32 threads

Will use 24 particle-particle and 8 PME only nodes
This is a guess, check the performance at the end of the log file
Making 2D domain decomposition 8 x 1 x 3

FYI, this is on this OS:

Linux t-mn01.hpc2n.umu.se 2.6.32-35-server #78-Ubuntu SMP Tue Oct 11 
16:26:12 UTC 2011 x86_64 GNU/Linux

Thanks!
> Sander
>
> On 13 Jan 2012, at 14:20 , David van der Spoel wrote:
>
>> On 2012-01-13 11:23, Sander Pronk wrote:
>>> This error is generated when the thread library can't create threads for some reason (out of memory, or some ulimit; I've never seen it before). It is probably due to the OS.
>>>
>>> There might be a chance that this is due to thread affinity API incompatibility: if the number of threads is equal to the number of hardware threads (cores, etc), thread_mpi will enforce thread affinity.
>>>
>>> Could you try with:
>>>
>>> mdrun -nt 30
>>>
>>> (or whichever number other than 32 that is compatible with domain decomposition)
>>>
>>> and report whether that works?
>>
>> It does indeed work with 30 threads, but not with 32.
>> Strange because I thought I tested it with 16, but apparently not, because that works as well.
>>
>> Note that the machine have 4 cpus with 8 cores (not 12 as I stated previously).
>>
>>>
>>> Sander
>>>
>>>
>>> On 12 Jan 2012, at 14:39 , David van der Spoel wrote:
>>>
>>>> On 2012-01-12 13:31, Ake Sandgren wrote:
>>>>> On Thu, 2012-01-12 at 13:18 +0100, David van der Spoel wrote:
>>>>>> On 2012-01-12 11:36, Berk Hess wrote:
>>>>>>> On 01/12/2012 11:24 AM, David van der Spoel wrote:
>>>>>>>> On 2012-01-12 11:17, Berk Hess wrote:
>>>>>>>>> Which compiler is this?
>>>>>>>>> We get lots of warnings with gcc4.6, but we run regularly on 32 and 64
>>>>>>>>> core nodes.
>>>>>>>>
>>>>>>>> Thread model: posix
>>>>>>>> gcc version 4.4.3 (Ubuntu 4.4.3-4ubuntu5)
>>>>>>>>
>>>>>>>> Is intel more reliable?
>>>>>>> This is not a matter of platform I would think.
>>>>>>> We have only used AMD platform with 32 or more MPI threads.
>>>>>>> I would guess this is a thread-mpi bug or a compiler issue.
>>>>>> Compiling with Intel C 12.0.3.174 gives the same error, but the same
>>>>>> pthread library is linked in.
>>>>>>
>>>>>> Other tips for debugging this?
>>>>>>
>>>>>>>
>>>>>>> Berk
>>>>>>>>>
>>>>>>>>> Berk
>>>>>>>>>
>>>>>>>>> On 01/12/2012 11:04 AM, David van der Spoel wrote:
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> I'm trying to compile and install gromacs release-4-5-patches on a new
>>>>>>>>>> cluster with four 12-core AMD chips (abisko in Umea, Sweden). However
>>>>>>>>>> the threaded code bails out with the following message:
>>>>>>>>>>
>>>>>>>>>> Reading file topol.tpr, VERSION 4.5.5-dev-20120111-9181e (double
>>>>>>>>>> precision)
>>>>>>>>>> Starting 32 threads
>>>>>>>>>> tMPI error: tMPI Initialization error (in valid comm)
>>>>>>>>>>
>>>>>>>>>> First, I'm a bit confused why the code detects only 32 cores, second
>>>>>>>>>> it shows above error and quits.
>>>>>>>>>>
>>>>>>>>>> Any clues?
>>>>>
>>>>>
>>>>> Abisko's current nodes are 4-socket 8-core (the 12-cores are still under
>>>>> test)
>>>>>
>>>>> If you are using openmpi it does not have support for MPI threads
>>>>> compiled in (the openib part of openmpi doesn't support this yet) that
>>>>> probably explains your problem.
>>>>>
>>>> I see, that explains the 32. But gromacs uses it's own mpi-over-threads implementation that does not use any MPI whatsoever.
>>>>
>>>>
>>>> --
>>>> David van der Spoel, Ph.D., Professor of Biology
>>>> Dept. of Cell&   Molec. Biol., Uppsala University.
>>>> Box 596, 75124 Uppsala, Sweden. Phone:	+46184714205.
>>>> spoel at xray.bmc.uu.se    http://folding.bmc.uu.se
>>>> --
>>>> gmx-developers mailing list
>>>> gmx-developers at gromacs.org
>>>> http://lists.gromacs.org/mailman/listinfo/gmx-developers
>>>> Please don't post (un)subscribe requests to the list. Use the www interface or send it to gmx-developers-request at gromacs.org.
>>>
>>
>>
>> --
>> David van der Spoel, Ph.D., Professor of Biology
>> Dept. of Cell&  Molec. Biol., Uppsala University.
>> Box 596, 75124 Uppsala, Sweden. Phone:	+46184714205.
>> spoel at xray.bmc.uu.se    http://folding.bmc.uu.se
>> --
>> gmx-developers mailing list
>> gmx-developers at gromacs.org
>> http://lists.gromacs.org/mailman/listinfo/gmx-developers
>> Please don't post (un)subscribe requests to the list. Use the www interface or send it to gmx-developers-request at gromacs.org.
>

-- 
David van der Spoel, Ph.D., Professor of Biology
Dept. of Cell & Molec. Biol., Uppsala University.
Box 596, 75124 Uppsala, Sweden. Phone:	+46184714205.
spoel at xray.bmc.uu.se    http://folding.bmc.uu.se