[gmx-users] AVX libraries with GMX4.6.3
Ali Sinan Saglam
asinansaglam at gmail.com
Tue Sep 3 18:17:53 CEST 2013
Hi Mark,
1e-9 is indeed an abnormally high tolerance to use, I was just testing
higher precisions to see if I can get better energy conservation.
I ran some 1ns tests with different tolerances and compared avx and sse
instruction sets. I'm still seeing a discrepancy between the two
instruction sets as can be seen;
AVX : https://www.dropbox.com/s/glmhyj1l18va8xp/avx_tol_comp_2.png
SSE : https://www.dropbox.com/s/6bwz4j9d0xywe2w/sse_tol_comp_2.png
where in SET_TEMP_X, SET=the instruction set used for the binary
compilation and X=(ewald_rtol=1e-X). For X=5,6,7 they are basically the
same but for X=8,9 SSE is an order of magnitude better than AVX. For a more
normal value like X=5,6 the acceleration types certainly agree so this
might not be very important but I wanted to post what I have seen
nonetheless.
Best,
Ali Sinan Saglam
On Fri, Aug 30, 2013 at 2:32 PM, Mark Abraham <mark.j.abraham at gmail.com>wrote:
> ewald_rtol of 1e-9 is a very long way from the normal range of values,
> and your other electrostatics settings seem pretty normal. I'd expect
> these results to be of doubtful value, but I am surprised to see
> that/if there is a marked different with acceleration type. I suggest
> you try a normal simulation protocol and see how you go.
>
> Mark
>
> On Fri, Aug 30, 2013 at 7:31 PM, Ali Sinan Saglam
> <asinansaglam at gmail.com> wrote:
> > Hi Mark,
> >
> > I have tried GCC 4.7.1, 4.7.2 and ICC 13 (with and without MKL, my
> > colleague also tried double precision) compilers, all showed the same
> > behavior.
> >
> > I will add the .mdp file I have used but the problem consists for all
> input
> > files my colleagues tried (they had their own input files, independent to
> > me). For example the first graph is Barnase - Barstar in a large box
> > (~100,000 atoms, explicit tip3p waters, neutralized and a small amount of
> > salt, roughly 60-70 atoms of NaCl) and the mdp file used was;
> > ;##################################################### INTEGRATOR
> > ######################################################
> > integrator = md ; Molecular
> > dynamics
> > dt = 0.002 ; Timestep
> (ps)
> > nsteps = 500000 ; Simulation
> > duration (timesteps)
> > nstcomm = 1000 ; Center of
> > mass motion removal interval (timesteps)
> > comm-grps = system ; Remove
> center
> > of mass motion of system
> > ;###################################################### ENSEMBLE
> > #######################################################
> > Pcoupl = no ; Disable
> > barostat
> > tcoupl = no ; Disable
> > thermostat
> > ;################################################ BONDED INTERACTIONS
> > ##################################################
> > constraints = hbonds ; Disable
> > constraints
> > constraint_algorithm = LINCS ; Constrain
> > bonds using LINCS
> > lincs_iter = 1 ; Number of
> > LINCS iterations
> > ;############################################### NONBONDED INTERACTIONS
> > ################################################
> > coulombtype = PME ; Switch/PME
> > long-range electrostatics
> > fourierspacing = 0.1
> > pme_order = 4
> > ewald_rtol = 0.000000001
> > pbc = xyz ; Periodic
> > boundary condition
> > rcoulomb = 1.00 ; Short-range
> > electrostatic cutoff (nm)
> > ;rcoulomb_switch = 0.9 ;
> Short-range
> > electrostatic switch cutoff (nm)
> > vdwtype = Switch ; Switch van
> > der Waals interactions
> > rvdw = 0.9 ; Van der
> Waals
> > cutoff (nm)
> > rvdw_switch = 0.8 ; Van der
> Waals
> > switch cutoff (nm)
> > DispCorr = EnerPres ; Long-range
> > dispersion correction to energy and pressure
> > ns_type = grid ; Update
> > neighbor list using grid
> > nstlist = 10 ; Neighbor
> list
> > update interval (timesteps)
> > rlist = 1.00 ; Neighbor
> list
> > cut-off (nm)
> > continuation = yes
> > ;####################################################### OUTPUT
> > ########################################################
> > nstlog = 500 ; Energy log
> > output interval (timesteps)
> > nstenergy = 500000 ; Energy
> output
> > interval (timesteps)
> > nstxout = 500000 ;
> > Full-resolution trajectory output interval (timesteps)
> > nstvout = 500000 ;
> > Full-resolution velocity output interval (timesteps)
> > nstfout = 500000 ;
> > Full-resolution force output interval (timesteps)
> > nstxtcout = 500 ;
> > Reduced-resolution trajectory output interval (timesteps)
> > xtc-precision = 10000 ;
> > Reduced-resolution trajectory output precision
> >
> > While the second graph is for EG5 (~130,000 atoms I believe, I don't have
> > the exact number but I remember that it's a bigger system than mine,
> same
> > water model). I'm unsure what my 3rd colleague tested but had similar
> > results (his system heated to 400+K) and we didn't share any input files.
> > All conserved energy fine with SSE instruction set (my results were
> losing
> > energy with 1.1kcal/mol.ps)
> >
> > Best,
> > Ali Sinan Saglam
> >
> >
> > On Fri, Aug 30, 2013 at 12:03 PM, Mark Abraham <mark.j.abraham at gmail.com
> >wrote:
> >
> >> With what compiler? (e.g. consult mdrun -version) There are many known
> >> bugs in early point versions of each minor release of gcc, for
> >> example. This is why the installation instructions stress getting the
> >> latest version of your compiler.
> >>
> >> Otherwise, what is your .mdp file and simulation system contents?
> >>
> >> Mark
> >>
> >> On Fri, Aug 30, 2013 at 5:48 PM, Ali Sinan Saglam
> >> <asinansaglam at gmail.com> wrote:
> >> > Hi,
> >> >
> >> > I have been running some energy conservation tests with GMX4.6.3 and
> had
> >> > encountered some issues when using the AVX-256 instruction set.
> >> >
> >> > I first noticed that my systems were freezing on a cluster that uses
> >> > Sandy-bridge CPUs and did not on a different cluster using SSE. After
> >> > realizing that the problems were not in my configuration and setup I
> >> > started compiling multiple copies of Gromacs and saw that only the
> >> AVX-256
> >> > compiled binaries showed this behavior.
> >> >
> >> > After a few colleagues reproduced the same results, I'm now pretty
> sure
> >> > this is the case. Two colleagues started completely independently and
> >> > reproduced the same behavior on 2 different clusters and 2 desktop
> >> machines
> >> > (all sandy-brige, in all cases AVX yielded really weird temperatures,
> >> > sometimes exponential freezing sometimes heating etc.).
> >> >
> >> > Gromacs was compiled with -DGMX_CPU_ACCELERATION=AVX or =SSE4.1 (or
> any
> >> > version of SSE) and that was the only difference between the two
> >> binaries.
> >> > The method of simulation was exactly the same outside of the mdrun
> that
> >> was
> >> > used. The temperature results from my simulations can be seen here;
> >> > https://www.dropbox.com/s/ul7g0fb4il17wxm/avx_sse_temp.png
> >> > Another example with a different person reproducing the results;
> >> > https://www.dropbox.com/s/x7mr3kcgmvd78ie/adam_avx_sse_temp.png
> >> >
> >> > Cmake command was;
> >> > cmake SOURCE -DGMX_BUILD_OWN_FFTW=on -DGMX_OPENMM=off -DGMX_MPI=off
> >> > -DGMX_GPU=off -DGMX_THREAD_MPI=on -DGMX_CPU_ACCELERATION=AVX (or
> SSE4.1)
> >> > (also tried THREAD_MPI=off, similar results)
> >> >
> >> > Also all builds passed regression tests.
> >> >
> >> > I just wondered if this is a known problem or are we making a mistake
> >> while
> >> > compiling?
> >> >
> >> > Best,
> >> > Ali Sinan Saglam
> >> > --
> >> > gmx-users mailing list gmx-users at gromacs.org
> >> > http://lists.gromacs.org/mailman/listinfo/gmx-users
> >> > * Please search the archive at
> >> http://www.gromacs.org/Support/Mailing_Lists/Search before posting!
> >> > * Please don't post (un)subscribe requests to the list. Use the
> >> > www interface or send it to gmx-users-request at gromacs.org.
> >> > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >> --
> >> gmx-users mailing list gmx-users at gromacs.org
> >> http://lists.gromacs.org/mailman/listinfo/gmx-users
> >> * Please search the archive at
> >> http://www.gromacs.org/Support/Mailing_Lists/Search before posting!
> >> * Please don't post (un)subscribe requests to the list. Use the
> >> www interface or send it to gmx-users-request at gromacs.org.
> >> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >>
> >
> >
> >
> > --
> > Ali Sinan Saglam
> > Graduate Student in Chemistry
> > Chong Lab, Room 338, Eberly Hall
> > University of Pittsburgh
> > Pittsburgh, PA 15260
> > --
> > gmx-users mailing list gmx-users at gromacs.org
> > http://lists.gromacs.org/mailman/listinfo/gmx-users
> > * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/Search before posting!
> > * Please don't post (un)subscribe requests to the list. Use the
> > www interface or send it to gmx-users-request at gromacs.org.
> > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> --
> gmx-users mailing list gmx-users at gromacs.org
> http://lists.gromacs.org/mailman/listinfo/gmx-users
> * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/Search before posting!
> * Please don't post (un)subscribe requests to the list. Use the
> www interface or send it to gmx-users-request at gromacs.org.
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
--
Ali Sinan Saglam
Graduate Student in Chemistry
Chong Lab, Room 338, Eberly Hall
University of Pittsburgh
Pittsburgh, PA 15260
More information about the gromacs.org_gmx-users
mailing list