[gmx-developers] mpi and thread command line options

[Berk Hess]

On 07/11/2012 09:36 AM, Roland Schulz wrote:
> On Wed, Jul 11, 2012 at 3:12 AM, Berk Hess <hess at kth.se> wrote:
>> Hi all,
>>
>> Thanks for all the feedback.
>>
>> What I forgot to say is that most users will not have to look and
>> probably will not look
>> at all these options, as we will automate things as much as possible.
>> Furthermore, with the group cut-off scheme (the only scheme pre-4.6) openmp
>> threads are not even supported, so there is not much option. With the
>> Verlet scheme
>> combining MPI processes/threads with openmp threads is slower, except at
>> the scaling limit. Within a single CPU openmp threads might be faster
>> than mpi.
>> We can handle all these cases, except for optimization at the scaling limit,
>> automatically when using all available cores.
>>
>> While thinking of this, and if we don't add mpi spawn, I came up with a
>> new proposal:
>> -nt total number of threads (effectively the same functionality as in 4.5)
>> -ntmpi number of MPI threads (advanced option)
>> -ntomp number of OpenMP threads (advanced option)
> I like this proposal for a binary with tMPI.
> What if the binary is compiled with real MPI (GMX_MPI)? Would then
> "-nt" and "-ntmpi" be invalid options and "-ntomp" have the same
> meaning as for tMPI? Or would "-nt" also be a valid option and would
> have the same meaning as "-ntomp"? I slightly prefer making "-nt"
> invalid, because it would give an error if the user thinks the mdrun
> binary is compiled with tMPI and wants to use "-nt" for total number
> of threads.
>
> What about setting a different number of OpenMP threads for PME nodes?
> Do we want to have a (potential hidden) command line flag for that
> (e.g. ntomp_pme) or will this only be possible with environment
> variables?
>
> Roland
>
I forgot to add PME openmp count (and to be more precise):
-ntomp number of OpenMP threads per MPI process/thread (advanced option)
-ntomppme number of OpenMP threads for PME, default: set to -ntomp 
((very) advanced option)

David, as I tried to explain, combining MPI processes/threads with 
OpenMP threads
is always slower, except at the scaling limit, so most users don't have 
to worry about this.
Note that we hope to change this in the future, but we might need 
something else than OpenMP.

With process MPI we could still use -nt as setting the total number of 
threads (only
meaningful with the Verlet scheme), this would then indirectly control 
-ntomp.
This might not be particularly useful, but at least it makes things 
consistent.
So with process mpi the only option which does nothing is -ntmpi.

Cheers,

Berk