[gmx-users] Gromacs 4.6.7 with MPI and OpenMP
mark.j.abraham at gmail.com
Fri May 8 15:48:34 CEST 2015
On Fri, May 8, 2015 at 3:01 PM Malcolm Tobias <mtobias at wustl.edu> wrote:
> Hi Mark,
> On Friday 08 May 2015 11:51:03 Mark Abraham wrote:
> > >
> > > I'm attempting to build gromacs on a new cluster and following the same
> > > recipies that I've used in the past, but encountering a strange
> > > It claims to be using both MPI and OpenMP, but I can see by 'top' and
> > > reported core/walltime that it's really only generating the MPI
> > > and no threads.
> > >
> > I wouldn't take the output from top completely at face value. Do you get
> > the same performance from -ntomp 1 as -ntomp 4?
> I'm not relying on top. I also mentioned that the core/walltime as
> reported by Gromacs suggests that it's only utilizing 2 cores. I've also
> been comparing the performance to an older cluster.
What kind of simulation are you testing with? A reaction-field water box
will have almost nothing to do on the CPU, so no real change with #threads.
Check with your users, but a PME test case is often more appropriate.
> > We're running a hetergenous environment, so I tend to build with
> > > MPI/OpenMP/CUDA and the Intel compilers, but I'm seeing this same sort
> > > behavior with the GNU compilers. Here's how I'm configuring things:
> > >
> > > [root at login01 build2]# cmake -DGMX_FFT_LIBRARY=mkl -DGMX_MPI=ON
> > > -DGMX_GPU=ON -DCUDA_TOOLKIT_ROOT_DIR=/opt/cuda -DGMX_OPENMP=ON
> > > -DCMAKE_INSTALL_PREFIX=/act/gromacs-4.6.7_take2 .. | tee cmake.out
> > >
> > You need root access for "make install." Period.
> Yes Mark, I ran 'make install' as root.
> > Routinely using root means you've probably hosed your system some time...
> In 20+ years of managing Unix systems I've managed to hose many a system.
> > > Using 2 MPI processes
> > > Using 4 OpenMP threads per MPI process
> > >
> > > although I do see this warning:
> > >
> > > Number of CPUs detected (16) does not match the number reported by
> > > (1).
> > >
> > Yeah, that happens. There's not really a well-defined standard, so once
> > OS, MPI and OpenMP libraries all combine, things can get messy.
> Understood. On top of that we're using CPUSETs with our queuing system
> which can interfere with how the tasks are distributed. I've tried running
> the job outside of the queuing system and have seen the same behavior.
OK. Well that 1 reported by mdrun is literally the return value from
calling omp_get_num_procs(), so the solution is to look for what part of
the ecosystem is setting that to 1 and give that a slap ;-) IIRC the use of
-ntomp 4 means mdrun will go and use 4 threads anyway, but it'd be good to
fix the wider context.
> > But if people go around using root routinely... ;-)
> As soon as I figure out how to manage a computing cluster without becoming
> root I'll let you know ;-)
Sure, you need root access. You don't need it for running cmake when that
runs a pile of unsecure code ;-)
I've got dozens of Gromacs users, so I'm attempting to build the fastest,
> most versatile binary that I can. Any help that people can offer is
> certainly appreciated.
YMMV but hyperthreads were generally not useful with GROMACS 4.6. That is
changing for newer hardware and GROMACS, however.
> Malcolm Tobias
> Gromacs Users mailing list
> * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> send a mail to gmx-users-request at gromacs.org.
More information about the gromacs.org_gmx-users