[gmx-developers] Gromacs trying to use OpenMP instead of thread-mpi despite trying to convince it?

[Shirts, Michael (mrs5pt)]

Just to chime in, the illegal instruction error is definitely a bad one.
However, a FAQ that clearly flags this error and that the solution is fixing
those two compiler flags as the likely fix is probably enough for now.

Best,
~~~~~~~~~~~~
Michael Shirts
Assistant Professor
Department of Chemical Engineering
University of Virginia
michael.shirts at virginia.edu
(434)-243-1821


> From: Roland Schulz <roland at utk.edu>
> Reply-To: Discussion list for GROMACS development <gmx-developers at gromacs.org>
> Date: Mon, 3 Dec 2012 13:18:27 -0500
> To: Discussion list for GROMACS development <gmx-developers at gromacs.org>
> Subject: Re: [gmx-developers] Gromacs trying to use OpenMP instead of
> thread-mpi despite trying to convince it?
> 
> On Mon, Dec 3, 2012 at 1:02 PM, Szilárd Páll <szilard.pall at cbr.su.se> wrote:
> 
>> 
>>  On Mon, Dec 3, 2012 at 5:12 AM, Roland Schulz <roland at utk.edu> wrote:
>> 
>>> 
>>> 
>>> 
>>>  On Sun, Dec 2, 2012 at 12:58 PM, Shirts, Michael (mrs5pt) <
>>> mrs5pt at eservices.virginia.edu> wrote:
>>> 
>>>> So, more progress, but no simulations running yet.
>>>> 
>>>> mdrun -nt 8 -ntmpi 8 gives the same error as before (I actually tried
>>>> that
>>>> before, and forgot to include it in my error report)
>>>> 
>>>> mdrun -ntmpi 8 -ntomp 1 gives the error
>>>> Fatal error:
>>>> OMP_NUM_THREADS (8) and the number of threads requested on the command
>>>> line
>>>> (1) have different values
>>>>  For more information and tips for troubleshooting, please check the
>>>> GROMACS
>>>> website at http://www.gromacs.org/Documentation/Errors
>>>> 
>>> 
>>>  We probably should print a notice that OMP_NUM_THREADS is set.
>>> Otherwise this is really confusing if OMP_NUM_THREADS isn't set by the user
>>> but by the system.
>>> 
>> 
>>  There is a note printed whenever the number of OpenMP threads is set by
>> OMP_NUM_THREADS instead of -ntomp.
>> 
> I think if even core developers don't understand/find a note, then that's a
> pretty clear sign that it will be confusing to the average user ;-)
> 
> 
>> 
>> 
>>> 
>>> 
>>>> Fatal error:
>>>> OMP_NUM_THREADS is invalid: '0'
>>>> 
>>> 
>>>  This is also for ntomp. Also there you wand to use 1 not 0 to disable
>>> OpenMP (1 because it is total number of threads and thus 1 means serial).
>>> 
>>>> 
>>>> /var/spool/PBS/mom_priv/jobs/2053253.lc5.itc.virginia.edu.SC: line 22:
>>>> 12201
>>>> Illegal instruction     /h3/n1/shirtsgroup/gromac
>>>> s_46/install/bin/mdrun_d -ntomp 1 -ntmpi -8 -deffnm
>>>> /bigtmp/mrs5pt/eth.vrescale.50
>>>> 
>>> 
>>>  I suppose the CPU on the compute node is different from the build host.
>>> You need to change GMX_CPU_ACCELERATION to the one correct for the compute
>>> node. It could also help to set GMX_DISTRIBUTABLE_BUILD (both are cmake
>>> options).
>>> 
>> 
>>  GMX_DISTRIBUTABLE_BUILD only does one thing, it turns off rdtscp. Is the
>> intention to provide more features for this option? If not, I don't see the
>> point in not calling it GMX_DISABLE_RDTSCP.
>> 
> I called it that for two reasons:
> - Users don't know whether they want to disabled RDTSCP but they might know
> whether they want to have a distributable build. In other words, in this
> case the ultimate goal ("make it work on a different CPU") makes more sense
> to the user then how this is achieved.
> - It can be extended in the furture.
> 
> But I think we should consider to disable rdtscp by default. Unless we add
> a runtime detection. The advantage is to small to cause problems. And as
> far as I know the only other option which can cause illegal instruction is
> GMX_CPU_ACCELERATION. But that one is pretty obvious to the user (it is a
> non-advanced cmake option and I think people are much more likely to have
> heard of SSE then rdtscp).
> 
> Roland
> 
> 
> 
>> 
>>  Cheers,
>> --
>> Szilárd
>> 
>> 
>>> 
>>>  Roland
>>> 
>>> 
>>>> 
>>>> Best,
>>>> 
>>>> ~~~~~~~~~~~~
>>>> Michael Shirts
>>>> Assistant Professor
>>>> Department of Chemical Engineering
>>>> University of Virginia
>>>> michael.shirts at virginia.edu
>>>> (434)-243-1821
>>>> 
>>>> 
>>>>> From: Berk Hess <hess at kth.se>
>>>>> Date: Sun, 2 Dec 2012 09:34:11 +0100
>>>>> To: "michael.shirts at virginia.edu" <michael.shirts at virginia.edu>,
>>>> Discussion
>>>>> list for GROMACS development <gmx-developers at gromacs.org>
>>>>> Subject: Re: [gmx-developers] Gromacs trying to use OpenMP instead of
>>>>> thread-mpi despite trying to convince it?
>>>>> 
>>>> 
>>>>> Hi,
>>>>> 
>>>>> You queuing system probably doesn't set OMP_NUM_THREADS then
>>>>> and I assume this machine has at least 16 (HT) cores.
>>>>> mdrun -ntmpi 8 -ntomp 1
>>>>> will do what you want, or:
>>>>> mdrun -nt 8 -ntmpi 8
>>>>> 
>>>>> Cheers,
>>>>> 
>>>>> Berk
>>>>> 
>>>>> On 12/02/2012 08:28 AM, Shirts, Michael (mrs5pt) wrote:
>>>>>> Quick question:
>>>>>> 
>>>>>> Compiling the most recent code in release-4-6, I compile without
>>>> OpenMP
>>>>>> (because using group rather than verlet cutoffs), and using any of the
>>>>>> below:
>>>>>> 
>>>>>> mdrun_d -ntmpi 8 -deffnm ethrun
>>>>>> or
>>>>>> mdrun_d -nt 8 -deffnm ethrun
>>>>>> or
>>>>>> mdrun_d -deffnm ethrun
>>>>>> or
>>>>>> mdrun_d -ntomp 0 -deffnm ethrun
>>>>>> or
>>>>>> mdrun_d -ntomp 0 -ntmpi8 -deffnm ethrun
>>>>>> 
>>>>>> I get:
>>>>>> Fatal error:
>>>>>> OpenMP threads are requested, but Gromacs was compiled without OpenMP
>>>>>> support
>>>>>> For more information and tips for troubleshooting, please check the
>>>> GROMACS
>>>>>> website at http://www.gromacs.org/Documentation/Errors
>>>>>> 
>>>>>> Even though I'm presumably requesting thread-mpi.  Worked fine with
>>>> -nt
>>>>>> previously (before the new -nt options introduced a few months back).
>>>>>> 
>>>>>> Any suggestions or something I'm doing wrong?  Perhaps gromacs is
>>>>>> interpreting the cluster environment as requesting OpenMP somehow?
>>>> FWIW, the
>>>>>> PBS script request line is "#PBS -l select=1:mpiprocs=8:ncpus=8".
>>>>>> 
>>>>>> Apologies if I missed this answers somewhere out there already.
>>>>>> 
>>>>>> Thanks,
>>>>>> ~~~~~~~~~~~~
>>>>>> Michael Shirts
>>>>>> Assistant Professor
>>>>>> Department of Chemical Engineering
>>>>>> University of Virginia
>>>>>> michael.shirts at virginia.edu
>>>>>> (434)-243-1821
>>>>>> 
>>>>> 
>>>> 
>>>> --
>>>> gmx-developers mailing list
>>>> gmx-developers at gromacs.org
>>>> http://lists.gromacs.org/mailman/listinfo/gmx-developers
>>>> Please don't post (un)subscribe requests to the list. Use the
>>>> www interface or send it to gmx-developers-request at gromacs.org.
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>>  --
>>> ORNL/UT Center for Molecular Biophysics cmb.ornl.gov
>>> 865-241-1537, ORNL PO BOX 2008 MS6309
>>> 
>>> --
>>> gmx-developers mailing list
>>> gmx-developers at gromacs.org
>>> http://lists.gromacs.org/mailman/listinfo/gmx-developers
>>> Please don't post (un)subscribe requests to the list. Use the
>>> www interface or send it to gmx-developers-request at gromacs.org.
>>> 
>> 
>> 
> 
> 
> -- 
> ORNL/UT Center for Molecular Biophysics cmb.ornl.gov
> 865-241-1537, ORNL PO BOX 2008 MS6309
> -- 
> gmx-developers mailing list
> gmx-developers at gromacs.org
> http://lists.gromacs.org/mailman/listinfo/gmx-developers
> Please don't post (un)subscribe requests to the list. Use the
> www interface or send it to gmx-developers-request at gromacs.org.