[gmx-developers] mpi and thread command line options

Wed Jul 11 19:40:25 CEST 2012

2012/7/11 Berk Hess <hess at kth.se>:
> On 07/11/2012 06:51 PM, Christoph Junghans wrote:
>>
>> 2012/7/11 Roland Schulz <roland at utk.edu>:
>>>
>>> On Wed, Jul 11, 2012 at 7:19 AM, Alexey Shvetsov
>>> <alexxy at omrb.pnpi.spb.ru> wrote:
>>>>
>>>> Roland Schulz писал 2012-07-11 10:47:
>>>>>
>>>>> On Wed, Jul 11, 2012 at 2:33 AM, Alexey Shvetsov
>>>>> <alexxy at omrb.pnpi.spb.ru> wrote:
>>>>>>
>>>>>> Hi!
>>>>>>
>>>>>> mvpich2/mvapich (as well as its derviations like platform
>>>>>> mpi,pc-mpi,intel-mpi) also will behave differently. So user can get
>>>>>> cryptic message about launching mpd, in case of launching mdrun -np
>>>>>> directly
>>>>>
>>>>> Not quite. mpich2 requires for MPI_Comm_spawn to work that the
>>>>> application is run with mpirun/mpiexec. See
>>>>>
>>>>> http://lists.mcs.anl.gov/pipermail/mpich-discuss/2012-June/012638.html
>>>>> for the details. We would need to detect that and don't try to spawn
>>>>> in that case (and run in serial with a warning).
>>>>> Thus mpich2 would require: mpirun mdrun -np x. Of course that isn't
>>>>> more convenient than mpirun -np x mdrun. The only advantage would be
>>>>> that as with tmpi "mpirun mdrun" would automatically use as many
>>>>> cores
>>>>> as are available and are useful for the system, whereas without spawn
>>>>> the user needs to decides the number of cores and we can't have any
>>>>> automatic mechanism helping the user.
>>>>>
>>>>> Roland
>>>>
>>>> Ok. But how this will work with batch systems that automaticaly send
>>>> number of processes to mpiexec, effective launch command will be
>>>>
>>>> $ mpiexec mdrun_mpi $mdrunargs
>>>
>>> The idea was to only do any spawn if mdrun_mpi is started in serial
>>> (mpiexec -n 1). It was only meant to make mdrun_mpi behave the same as
>>> tmpi mdrun for a single node. On clusters with batch system nothing
>>> would have changed over the current situation.
>>
>> 1.) I agree with Roland that keeping the -nt option would be
>> misleading, even if -nt gets a new meaning - "number of tasks", where
>> tasks can be threads or mpi tasks.
>> Also from my experience, half of the users are not aware of the -nt
>> option either, they just start mdrun without any special setting,
>> which means "guess" and we should keep that.
>> So making -nt obsolete is not bad, I think.
>
> In my last mail the proposition was to only use -nt for the total number of
> threads
> and not with MPI. I think this is useful, since the user usually would want
> to limit
> the total number of threads (or cores used) and would not know what to
> choose
> for -ntmpi and -ntomp.
You are right, that sounds good. I think, the real issue is to
communicate the difference between OMP and tMPI threads to the user.

>>
>> 2.) One class of system that has not been discussed yet, are the one
>> with different number of OMP threads per node (like the Intel MIC),
>> that should also be possible by explicitly defining OMP_NUM_THREADS on
>> each node.
> How doesn MIC have different numbers of threads per node?
Sorry, I should been more detailed about that. The MIC runs as an
acceleration together with an normal Intel core.
>From the user side the MIC looks like a XX( >32) cpu linux machine, so
one would like to run mdrun with 2 MPI tasks, one of which uses XX
threads, while the other uses Y (=8/16) threads on the Intel core.

>
>>
>> 3.) With all these thread options and combination, we will need
>> something like g_tune_mdrun, which I guess could be an extension of
>> g_tune_pme.
>
> Except for the scaling limit, there is usually once best choice.
Good to know.
> But for running close to the scaling limit a g_tune_mdrun could indeed help.

>
> Cheers,
>
> Berk
>
>>
>> Christoph
>>
>>
>>
>>> Roland
>>>
>>>>
>>>>
>>>>>> Roland Schulz писал 2012-07-11 05:09:
>>>>>>>
>>>>>>> On Tue, Jul 10, 2012 at 8:09 PM, Szilárd Páll
>>>>>>> <szilard.pall at cbr.su.se> wrote:
>>>>>>>>
>>>>>>>> On Tue, Jul 10, 2012 at 11:15 PM, Berk Hess <hess at kth.se> wrote:
>>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> We are working on the final part of the 4.6 release, which is
>>>>>>>>> making the MPI
>>>>>>>>> and OpenMP thread setup automated, fully checked and user
>>>>>>>>> friendly.
>>>>>>>>> We have to decide on the naming of the options.
>>>>>>>>> Roland has an implementation of mpi spawn ready. This would allow
>>>>>>>>> to do
>>>>>>>>> mdrun -np #processes instead of using mpirun (at least with
>>>>>>>>> openmpi).
>>>>>>>>
>>>>>>>> Would this feature add anything but the convenience of being able
>>>>>>>> to
>>>>>>>> run without mpirun on a single node? Without MPI spawning working
>>>>>>>> reliably in most cases (or with the ability to detect with a high
>>>>>>>> certainty when it does not), enabling an -np mdrun option would
>>>>>>>> just
>>>>>>>> lead to confusion when mdrun exits with cryptic MPI error due to
>>>>>>>> not
>>>>>>>> being able to spawn.
>>>>>>>
>>>>>>> The idea was to make mdrun behave the same whether it is compiled
>>>>>>> with
>>>>>>> real MPI or tMPI. Thus also only support a single node. But MPICH
>>>>>>> is
>>>>>>> behaving quite stupid and they also don't seem to care. And only
>>>>>>> supporting it for OpenMPI is probably also more confusing then
>>>>>>> helpful
>>>>>>> (then tmpi+OpenMPI would behave the same but MPICH/MVAPICH would
>>>>>>> behave different). So you are probably right that it is better to
>>>>>>> not
>>>>>>> add spawn at all.
>>>>>>>
>>>>>>>> Therefore, I'd be OK with a new *hidden* -np option that only
>>>>>>>> works
>>>>>>>> in
>>>>>>>> single-node case, but not with a non-hidden one advertised in the
>>>>>>>> documentation/wiki.
>>>>>>>
>>>>>>> As a hidden option it would only help for testing. But I don't
>>>>>>> think
>>>>>>> it is worth adding it for just that.
>>>>>>>
>>>>>>> Roland
>>>>>>
>>>>>> --
>>>>>> Best Regards,
>>>>>> Alexey 'Alexxy' Shvetsov
>>>>>> Petersburg Nuclear Physics Institute, NRC Kurchatov Institute,
>>>>>> Gatchina, Russia
>>>>>> Department of Molecular and Radiation Biophysics
>>>>>> Gentoo Team Ru
>>>>>> Gentoo Linux Dev
>>>>>> mailto:alexxyum at gmail.com
>>>>>> mailto:alexxy at gentoo.org
>>>>>> mailto:alexxy at omrb.pnpi.spb.ru
>>>>>> --
>>>>>> gmx-developers mailing list
>>>>>> gmx-developers at gromacs.org
>>>>>> http://lists.gromacs.org/mailman/listinfo/gmx-developers
>>>>>> Please don't post (un)subscribe requests to the list. Use the
>>>>>> www interface or send it to gmx-developers-request at gromacs.org.
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> ORNL/UT Center for Molecular Biophysics cmb.ornl.gov
>>>>> 865-241-1537, ORNL PO BOX 2008 MS6309
>>>>
>>>> --
>>>> Best Regards,
>>>> Alexey 'Alexxy' Shvetsov
>>>> Petersburg Nuclear Physics Institute, NRC Kurchatov Institute,
>>>> Gatchina, Russia
>>>> Department of Molecular and Radiation Biophysics
>>>> Gentoo Team Ru
>>>> Gentoo Linux Dev
>>>> mailto:alexxyum at gmail.com
>>>> mailto:alexxy at gentoo.org
>>>> mailto:alexxy at omrb.pnpi.spb.ru
>>>> --
>>>> gmx-developers mailing list
>>>> gmx-developers at gromacs.org
>>>> http://lists.gromacs.org/mailman/listinfo/gmx-developers
>>>> Please don't post (un)subscribe requests to the list. Use the
>>>> www interface or send it to gmx-developers-request at gromacs.org.
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> ORNL/UT Center for Molecular Biophysics cmb.ornl.gov
>>> 865-241-1537, ORNL PO BOX 2008 MS6309
>>> --
>>> gmx-developers mailing list
>>> gmx-developers at gromacs.org
>>> http://lists.gromacs.org/mailman/listinfo/gmx-developers
>>> Please don't post (un)subscribe requests to the list. Use the
>>> www interface or send it to gmx-developers-request at gromacs.org.
>>
>>
>>
>
>
> --
> gmx-developers mailing list
> gmx-developers at gromacs.org
> http://lists.gromacs.org/mailman/listinfo/gmx-developers
> Please don't post (un)subscribe requests to the list. Use the www interface
> or send it to gmx-developers-request at gromacs.org.


-- 
Christoph Junghans
Web: http://www.compphys.de