[gmx-developers] mpi and thread command line options

[Justin A. Lemkul]

On 7/10/12 5:38 PM, Alexey Shvetsov wrote:
> Hi all!
>
> I think -np for number of mpi process and -nth would be good. But how it will
> handle situation for example with 48 core node (e.g. 4x opteron 61xx) so it will
> have 8 numa nodes in one node. Optimal situation may be (in case we have more
> then one interconnect port per platform) to run one mpi process per numa node
> (total 8 mpi processe) and 6 omp threades per numa node (mpi porcess)
>

These are the kinds of things that scare me ;)  I am not a specialist on 
computer hardware, compilers, libraries, etc.  I suspect many on the development 
list are, but the majority of the user base is likely not.

> Berk Hess писал 2012-07-11 01:15:
>> Hi,
>>
>> We are working on the final part of the 4.6 release, which is making the MPI
>> and OpenMP thread setup automated, fully checked and user friendly.

This sounds lovely to those of us who are blissfully ignorant of what much of 
this all means.

>> We have to decide on the naming of the options.
>> Roland has an implementation of mpi spawn ready. This would allow to do
>> mdrun -np #processes instead of using mpirun (at least with openmpi).
>> The question now is how to name the (thread-)mpi option and the openmp
>> thread option. I suggested -np and -nt, but I now think we should not change
>> the functionality of -nt.
>> So I would suggest -np and -nth (or -ntsm for shared memory threads).
>> Then -nt can be kept in 4.6 for backward compatibility, with a warning.
>> I think we don't want to use -ntomp as we might move away from OpenMP.

Would there be any value in keeping -np and -nt, and adding an additional flag 
(like -thtype or something) that sets the type of thread to be invoked?  Default 
behavior can act like what we have now.  Again, I know very little about what 
all the distinctions are here between thread types (-nth vs. -ntsm, for 
instance), but perhaps that's a useful perspective to have.

>> Another question is if the processes MPI and thread-mpi option should
>> have the same name. An inexperienced user might submit a thread-mpi mdrun
>> job on multiple nodes and get unwanted effects. A way to catch this could
>> be to give a fatal error when you ask for more threads then there are cores
>> in the node, with the hint that you shoud compile with MPI to run
>> over multiple
>> nodes and a hint that you can set an env.var. if you really want to use
>> more threads than cores.

Error messages that are very (very, very) explicit are great.  From a 
documentation standpoint (where I've made the most contributions), I would like 
to see some "official" implementation examples in the manual, appendix A.3.1. 
Much of that appendix probably needs updating, anyway.

All the new features sound wonderful, we just have to make sure that any changes 
are obvious and that the implementation is accessible to those of us who have a 
more casual understanding of parallelization and such.  I'm a biochemist; for 
me, computers were a hobby that became very important :)

-Justin

-- 
========================================

Justin A. Lemkul, Ph.D.
Research Scientist
Department of Biochemistry
Virginia Tech
Blacksburg, VA
jalemkul[at]vt.edu | (540) 231-9080
http://www.bevanlab.biochem.vt.edu/Pages/Personal/justin

========================================