[gmx-developers] Gromacs trying to use OpenMP instead of thread-mpi despite trying to convince it?

Shirts, Michael (mrs5pt) mrs5pt at eservices.virginia.edu
Mon Dec 3 22:21:58 CET 2012 

Sorry, I meant cmake options.

Best,
~~~~~~~~~~~~
Michael Shirts
Assistant Professor
Department of Chemical Engineering
University of Virginia
michael.shirts at virginia.edu
(434)-243-1821


> From: Szilárd Páll <szilard.pall at cbr.su.se>
> Date: Mon, 3 Dec 2012 22:20:26 +0100
> To: "michael.shirts at virginia.edu" <michael.shirts at virginia.edu>, Discussion
> list for GROMACS development <gmx-developers at gromacs.org>
> Subject: Re: [gmx-developers] Gromacs trying to use OpenMP instead of
> thread-mpi despite trying to convince it?
> 
> On Mon, Dec 3, 2012 at 9:38 PM, Shirts, Michael (mrs5pt) <
> mrs5pt at eservices.virginia.edu> wrote:
> 
>> Just to chime in, the illegal instruction error is definitely a bad one.
>> However, a FAQ that clearly flags this error and that the solution is
>> fixing
>> those two compiler flags as the likely fix is probably enough for now.
>> 
> 
> Note that modifying compiler flags won't help because in these cases you'd
> typically need to "lower" the CPU acceleration. This sis currently not
> possible without re-configuring.
> 
> --
> Szilárd
> 
> 
>> 
>> Best,
>> ~~~~~~~~~~~~
>> Michael Shirts
>> Assistant Professor
>> Department of Chemical Engineering
>> University of Virginia
>> michael.shirts at virginia.edu
>> (434)-243-1821
>> 
>> 
>>> From: Roland Schulz <roland at utk.edu>
>>> Reply-To: Discussion list for GROMACS development <
>> gmx-developers at gromacs.org>
>>> Date: Mon, 3 Dec 2012 13:18:27 -0500
>>> To: Discussion list for GROMACS development <gmx-developers at gromacs.org>
>>> Subject: Re: [gmx-developers] Gromacs trying to use OpenMP instead of
>>> thread-mpi despite trying to convince it?
>>> 
>>> On Mon, Dec 3, 2012 at 1:02 PM, Szilárd Páll <szilard.pall at cbr.su.se>
>> wrote:
>>> 
>>>> 
>>>>  On Mon, Dec 3, 2012 at 5:12 AM, Roland Schulz <roland at utk.edu> wrote:
>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>>  On Sun, Dec 2, 2012 at 12:58 PM, Shirts, Michael (mrs5pt) <
>>>>> mrs5pt at eservices.virginia.edu> wrote:
>>>>> 
>>>>>> So, more progress, but no simulations running yet.
>>>>>> 
>>>>>> mdrun -nt 8 -ntmpi 8 gives the same error as before (I actually tried
>>>>>> that
>>>>>> before, and forgot to include it in my error report)
>>>>>> 
>>>>>> mdrun -ntmpi 8 -ntomp 1 gives the error
>>>>>> Fatal error:
>>>>>> OMP_NUM_THREADS (8) and the number of threads requested on the command
>>>>>> line
>>>>>> (1) have different values
>>>>>>  For more information and tips for troubleshooting, please check the
>>>>>> GROMACS
>>>>>> website at http://www.gromacs.org/Documentation/Errors
>>>>>> 
>>>>> 
>>>>>  We probably should print a notice that OMP_NUM_THREADS is set.
>>>>> Otherwise this is really confusing if OMP_NUM_THREADS isn't set by the
>> user
>>>>> but by the system.
>>>>> 
>>>> 
>>>>  There is a note printed whenever the number of OpenMP threads is set by
>>>> OMP_NUM_THREADS instead of -ntomp.
>>>> 
>>> I think if even core developers don't understand/find a note, then
>> that's a
>>> pretty clear sign that it will be confusing to the average user ;-)
>>> 
>>> 
>>>> 
>>>> 
>>>>> 
>>>>> 
>>>>>> Fatal error:
>>>>>> OMP_NUM_THREADS is invalid: '0'
>>>>>> 
>>>>> 
>>>>>  This is also for ntomp. Also there you wand to use 1 not 0 to disable
>>>>> OpenMP (1 because it is total number of threads and thus 1 means
>> serial).
>>>>> 
>>>>>> 
>>>>>> /var/spool/PBS/mom_priv/jobs/2053253.lc5.itc.virginia.edu.SC: line
>> 22:
>>>>>> 12201
>>>>>> Illegal instruction     /h3/n1/shirtsgroup/gromac
>>>>>> s_46/install/bin/mdrun_d -ntomp 1 -ntmpi -8 -deffnm
>>>>>> /bigtmp/mrs5pt/eth.vrescale.50
>>>>>> 
>>>>> 
>>>>>  I suppose the CPU on the compute node is different from the build
>> host.
>>>>> You need to change GMX_CPU_ACCELERATION to the one correct for the
>> compute
>>>>> node. It could also help to set GMX_DISTRIBUTABLE_BUILD (both are cmake
>>>>> options).
>>>>> 
>>>> 
>>>>  GMX_DISTRIBUTABLE_BUILD only does one thing, it turns off rdtscp. Is
>> the
>>>> intention to provide more features for this option? If not, I don't see
>> the
>>>> point in not calling it GMX_DISABLE_RDTSCP.
>>>> 
>>> I called it that for two reasons:
>>> - Users don't know whether they want to disabled RDTSCP but they might
>> know
>>> whether they want to have a distributable build. In other words, in this
>>> case the ultimate goal ("make it work on a different CPU") makes more
>> sense
>>> to the user then how this is achieved.
>>> - It can be extended in the furture.
>>> 
>>> But I think we should consider to disable rdtscp by default. Unless we
>> add
>>> a runtime detection. The advantage is to small to cause problems. And as
>>> far as I know the only other option which can cause illegal instruction
>> is
>>> GMX_CPU_ACCELERATION. But that one is pretty obvious to the user (it is a
>>> non-advanced cmake option and I think people are much more likely to have
>>> heard of SSE then rdtscp).
>>> 
>>> Roland
>>> 
>>> 
>>> 
>>>> 
>>>>  Cheers,
>>>> --
>>>> Szilárd
>>>> 
>>>> 
>>>>> 
>>>>>  Roland
>>>>> 
>>>>> 
>>>>>> 
>>>>>> Best,
>>>>>> 
>>>>>> ~~~~~~~~~~~~
>>>>>> Michael Shirts
>>>>>> Assistant Professor
>>>>>> Department of Chemical Engineering
>>>>>> University of Virginia
>>>>>> michael.shirts at virginia.edu
>>>>>> (434)-243-1821
>>>>>> 
>>>>>> 
>>>>>>> From: Berk Hess <hess at kth.se>
>>>>>>> Date: Sun, 2 Dec 2012 09:34:11 +0100
>>>>>>> To: "michael.shirts at virginia.edu" <michael.shirts at virginia.edu>,
>>>>>> Discussion
>>>>>>> list for GROMACS development <gmx-developers at gromacs.org>
>>>>>>> Subject: Re: [gmx-developers] Gromacs trying to use OpenMP instead of
>>>>>>> thread-mpi despite trying to convince it?
>>>>>>> 
>>>>>> 
>>>>>>> Hi,
>>>>>>> 
>>>>>>> You queuing system probably doesn't set OMP_NUM_THREADS then
>>>>>>> and I assume this machine has at least 16 (HT) cores.
>>>>>>> mdrun -ntmpi 8 -ntomp 1
>>>>>>> will do what you want, or:
>>>>>>> mdrun -nt 8 -ntmpi 8
>>>>>>> 
>>>>>>> Cheers,
>>>>>>> 
>>>>>>> Berk
>>>>>>> 
>>>>>>> On 12/02/2012 08:28 AM, Shirts, Michael (mrs5pt) wrote:
>>>>>>>> Quick question:
>>>>>>>> 
>>>>>>>> Compiling the most recent code in release-4-6, I compile without
>>>>>> OpenMP
>>>>>>>> (because using group rather than verlet cutoffs), and using any of
>> the
>>>>>>>> below:
>>>>>>>> 
>>>>>>>> mdrun_d -ntmpi 8 -deffnm ethrun
>>>>>>>> or
>>>>>>>> mdrun_d -nt 8 -deffnm ethrun
>>>>>>>> or
>>>>>>>> mdrun_d -deffnm ethrun
>>>>>>>> or
>>>>>>>> mdrun_d -ntomp 0 -deffnm ethrun
>>>>>>>> or
>>>>>>>> mdrun_d -ntomp 0 -ntmpi8 -deffnm ethrun
>>>>>>>> 
>>>>>>>> I get:
>>>>>>>> Fatal error:
>>>>>>>> OpenMP threads are requested, but Gromacs was compiled without
>> OpenMP
>>>>>>>> support
>>>>>>>> For more information and tips for troubleshooting, please check the
>>>>>> GROMACS
>>>>>>>> website at http://www.gromacs.org/Documentation/Errors
>>>>>>>> 
>>>>>>>> Even though I'm presumably requesting thread-mpi.  Worked fine with
>>>>>> -nt
>>>>>>>> previously (before the new -nt options introduced a few months
>> back).
>>>>>>>> 
>>>>>>>> Any suggestions or something I'm doing wrong?  Perhaps gromacs is
>>>>>>>> interpreting the cluster environment as requesting OpenMP somehow?
>>>>>> FWIW, the
>>>>>>>> PBS script request line is "#PBS -l select=1:mpiprocs=8:ncpus=8".
>>>>>>>> 
>>>>>>>> Apologies if I missed this answers somewhere out there already.
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> ~~~~~~~~~~~~
>>>>>>>> Michael Shirts
>>>>>>>> Assistant Professor
>>>>>>>> Department of Chemical Engineering
>>>>>>>> University of Virginia
>>>>>>>> michael.shirts at virginia.edu
>>>>>>>> (434)-243-1821
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> --
>>>>>> gmx-developers mailing list
>>>>>> gmx-developers at gromacs.org
>>>>>> http://lists.gromacs.org/mailman/listinfo/gmx-developers
>>>>>> Please don't post (un)subscribe requests to the list. Use the
>>>>>> www interface or send it to gmx-developers-request at gromacs.org.
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>>>  --
>>>>> ORNL/UT Center for Molecular Biophysics cmb.ornl.gov
>>>>> 865-241-1537, ORNL PO BOX 2008 MS6309
>>>>> 
>>>>> --
>>>>> gmx-developers mailing list
>>>>> gmx-developers at gromacs.org
>>>>> http://lists.gromacs.org/mailman/listinfo/gmx-developers
>>>>> Please don't post (un)subscribe requests to the list. Use the
>>>>> www interface or send it to gmx-developers-request at gromacs.org.
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> --
>>> ORNL/UT Center for Molecular Biophysics cmb.ornl.gov
>>> 865-241-1537, ORNL PO BOX 2008 MS6309
>>> --
>>> gmx-developers mailing list
>>> gmx-developers at gromacs.org
>>> http://lists.gromacs.org/mailman/listinfo/gmx-developers
>>> Please don't post (un)subscribe requests to the list. Use the
>>> www interface or send it to gmx-developers-request at gromacs.org.
>> 
>> --
>> gmx-developers mailing list
>> gmx-developers at gromacs.org
>> http://lists.gromacs.org/mailman/listinfo/gmx-developers
>> Please don't post (un)subscribe requests to the list. Use the
>> www interface or send it to gmx-developers-request at gromacs.org.
>>

More information about the gromacs.org_gmx-developers mailing list