[gmx-users] Gromacs 4.6.7 with MPI and OpenMP

[Mark Abraham]

Hi,

On Fri, May 8, 2015 at 3:01 PM Malcolm Tobias <mtobias at wustl.edu> wrote:

>
> Hi Mark,
>
> On Friday 08 May 2015 11:51:03 Mark Abraham wrote:
> > >
> > > I'm attempting to build gromacs on a new cluster and following the same
> > > recipies that I've used in the past, but encountering a strange
> behavior:
> > > It claims to be using both MPI and OpenMP, but I can see by 'top' and
> the
> > > reported core/walltime that it's really only generating the MPI
> processes
> > > and no threads.
> > >
> >
> > I wouldn't take the output from top completely at face value. Do you get
> > the same performance from -ntomp 1 as -ntomp 4?
>
> I'm not relying on top. I also mentioned that the core/walltime as
> reported by Gromacs suggests that it's only utilizing 2 cores.  I've also
> been comparing the performance to an older cluster.
>

What kind of simulation are you testing with? A reaction-field water box
will have almost nothing to do on the CPU, so no real change with #threads.
Check with your users, but a PME test case is often more appropriate.

> > We're running a hetergenous environment, so I tend to build with
> > > MPI/OpenMP/CUDA and the Intel compilers, but I'm seeing this same sort
> of
> > > behavior with the GNU compilers.  Here's how I'm configuring things:
> > >
> > > [root at login01 build2]# cmake -DGMX_FFT_LIBRARY=mkl -DGMX_MPI=ON
> > > -DGMX_GPU=ON -DCUDA_TOOLKIT_ROOT_DIR=/opt/cuda -DGMX_OPENMP=ON
> > > -DCMAKE_INSTALL_PREFIX=/act/gromacs-4.6.7_take2 .. | tee cmake.out
> > >
> >
> > You need root access for "make install." Period.
>
> Yes Mark, I ran 'make install' as root.
>
> > Routinely using root means you've probably hosed your system some time...
>
> In 20+ years of managing Unix systems I've managed to hose many a system.
>
> > > Using 2 MPI processes
> > > Using 4 OpenMP threads per MPI process
> > >
> > > although I do see this warning:
> > >
> > > Number of CPUs detected (16) does not match the number reported by
> OpenMP
> > > (1).
> > >
> >
> > Yeah, that happens. There's not really a well-defined standard, so once
> the
> > OS, MPI and OpenMP libraries all combine, things can get messy.
>
> Understood.  On top of that we're using CPUSETs with our queuing system
> which can interfere with how the tasks are distributed.  I've tried running
> the job outside of the queuing system and have seen the same behavior.
>

OK. Well that 1 reported by mdrun is literally the return value from
calling omp_get_num_procs(), so the solution is to look for what part of
the ecosystem is setting that to 1 and give that a slap ;-) IIRC the use of
-ntomp 4 means mdrun will go and use 4 threads anyway, but it'd be good to
fix the wider context.


> > But if people go around using root routinely... ;-)
>
> As soon as I figure out how to manage a computing cluster without becoming
> root I'll let you know  ;-)
>

Sure, you need root access. You don't need it for running cmake when that
runs a pile of unsecure code ;-)

I've got dozens of Gromacs users, so I'm attempting to build the fastest,
> most versatile binary that I can.  Any help that people can offer is
> certainly appreciated.
>

YMMV but hyperthreads were generally not useful with GROMACS 4.6. That is
changing for newer hardware and GROMACS, however.

Mark


> Cheers,
> Malcolm
>
>
> --
> Malcolm Tobias
> 314.362.1594
>
>
> --
> Gromacs Users mailing list
>
> * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> send a mail to gmx-users-request at gromacs.org.
>