[gmx-users] results produced by auto-tuning of Coulomb cut-off/grid for PME can not be reproduced by manually setting the Coulomb cut-off and grid spacing
Mark Abraham
mark.j.abraham at gmail.com
Thu Jan 22 23:41:06 CET 2015
On Thu, Jan 22, 2015 at 8:36 PM, Mark Abraham <mark.j.abraham at gmail.com>
wrote:
>
>
> On Thu, Jan 22, 2015 at 8:34 PM, Mark Abraham <mark.j.abraham at gmail.com>
> wrote:
>
>>
>>
>> On Thu, Jan 22, 2015 at 8:09 PM, Jiaqi Lin <jqlin at mit.edu> wrote:
>>
>>> Hi Mark
>>>
>>> Thank you for your patient reply.
>>>
>>> I've tried using the exact same set up (rcoulombic =3.258 fourierspacing
>>> = 0.325, rlist =1.4, rlistlong =3.258, rvdw=1.2) as the auto-tune of would
>>> have changed the parameters to, and still it won't reproduce the result
>>> obtained by auto-tune, just the result I tired previously using similar
>>> parameters. Therefore, I believe there is something going on in the code of
>>> auto-tune that is not what it is appear to be, or, the auto-tune change the
>>> parameters in a special way that in theory shouldn't affect simulation
>>> result but in my case the system is very sensitive to these changes. As you
>>> pointed out, the code is not straightforward, therefore much can happen at
>>> this stage. I'd very much like to figure out what going on, but spending
>>> two months on the code is not something I'm willing to do.
>>
>>
>> One test you can do that is straightforward is to do an auto-tuning run
>> and write the final output whenever the tuning completes.
>>
>
> Actually, get that frame and another one later. I don't remember whether
> the e.g. step-1000 energy output is with the last-tried or
> newly-decided-as-optimal PME regime.
>
And use nstfout to get the full-precision forces in both frames for both
runs, too. A problem could lie in either part, in principle.
Mark
> That trajectory should have only one frame in it. Then do a
>> http://www.gromacs.org/Documentation/How-tos/Single-Point_Energy
>> calculation on that trajectory file, using a .tpr that should match the
>> auto-tuning result. If that gave different energy components, then I think
>> that would be good evidence of a code bug. (Either way, as I've already
>> said, you should just go and use the Verlet scheme for running this system
>> on that amount of hardware...)
>>
>>
>>> So I think I will keep changing parameters and try to understand long
>>> range electrostatics in my system. Also, the simulation produced by
>>> auto-tune is unlikely to be a outlier. I kept changing -ntpme and when
>>> auto-tune give me rcoulombic larger than 3, I can always observe the
>>> "interesting" result.
>>>
>>
>> That is suggestive of a bug. If you can also reproduce it from runs that
>> have different initial velocities, then that would be quite strong evidence.
>>
>> I also tired LJPME as you suggested for long range vdw, it compress the
>>> bilayer to a large extend.
>>
>>
>> What force field was that?
>>
>>
>>> The whole force fields has to be refined if I use LJPME, but that's
>>> another story. I think electrostatics is the main driving force in my
>>> simulation system, and I'll keep trying to figure out what really happened.
>>>
>>
>> Hmm, that sounds like your model physics might be over-parameterized on
>> an invalid van der Waals approximation. I'd give serious consideration to
>> reviewing its performance on other problems quite carefully. Using it only
>> in a regime and/or software package where it is thought to be valid would
>> be the standard advice.
>>
>> Anyway, there is no way that
>> * if the model physics is correct and not over-tuned to the exact regime
>> and software that produced it, and
>> * if the GROMACS code is correct,
>> that the simulation behaviour could depend in a statistically significant
>> way on the tuning or -npme choice. Those values do not enter the model
>> physics.
>>
>> Just changing electrostatics regimes arbitrarily and cherry-picking some
>> of the results to analyse is going to get questions raised about your work
>> - either the model physics is not robust enough to use on your system, or
>> the software is wrong, or you're ignoring valid behaviour because something
>> else looks more interesting.
>>
>> Mark
>>
>>
>>> Best
>>> Jiaqi
>>>
>>>
>>> On 1/20/2015 7:18 PM, Mark Abraham wrote:
>>>
>>>> On Tue, Jan 20, 2015 at 11:48 PM, Jiaqi Lin <jqlin at mit.edu> wrote:
>>>>
>>>> Hi Szilard,
>>>>>
>>>>> - I've tired 5.0.1 and it gives the same result. So 4.6.7 or 5.0.4 is
>>>>> better, but in what way?
>>>>>
>>>>> - I've tired Verlet scheme and it gives small change of cutoff and
>>>>> grid.
>>>>> But what I really interested is to manually reproduce the result that
>>>>> tune_pme give me in the first case using Group scheme.
>>>>>
>>>>> If that's the only situation you can observe it, it could be an
>>>> outlier or
>>>> your method could be unsuitable...
>>>>
>>>>
>>>> - I've also tired lincs-roder=4 and it makes no difference.
>>>>>
>>>>> - My Fourier spacing is 25% finer, but that shouldn't affect the
>>>>> results
>>>>> right? if it do affect the results, then I want to find out how.
>>>>>
>>>>> It affects your results because you could do some more sampling with
>>>> the
>>>> computer time you are spending on PME at the moment. Where to choose
>>>> your
>>>> balance of model quality vs amount of sampling is poorly understood, and
>>>> problem-dependent.
>>>>
>>>> - I happen to use the PP/PME rank split (100+28) and it gives me
>>>>
>>>>> interesting results (speed of performance is not bad actually). Then
>>>>> I'm
>>>>> very interested in how these cutoff and grid setting can affect my
>>>>> simulation results.
>>>>>
>>>>
>>>> If the implementation and model is right, then they have no effect.
>>>> That's
>>>> why we auto-tune with them. You're going to need a lot of replicates to
>>>> show that -npme 28 gives a statistically different result from a
>>>> different
>>>> value, and you won't yet have established that it's somehow a more valid
>>>> observation to pursue.
>>>>
>>>> So I tried to manually control the parameter (turning off tune_PME).
>>>> But no
>>>>
>>>>> matter how I tired, I can not reproduce the result given by tune_pme.
>>>>> So my
>>>>> biggest question is, how does tune_PME implemented in the code? What
>>>>> parameters does it actually tuned?
>>>>>
>>>>> Like it says, it varies rcoulomb and the Fourier grid, keeping rvdw
>>>> and
>>>> beta fixed. Details of how rlist and rlistlong behave are a bit messy,
>>>> but
>>>> nothing you asked for is ever missed out.
>>>>
>>>>
>>>> - When PME tuned the cutoff to such large value, the speed does not
>>>>> goes
>>>>> down noticeably. So what I suspect is that tune_PME
>>>>>
>>>>
>>>> Please be careful with names. gmx tune_pme is a different thing from the
>>>> mdrun auto-tuning.
>>>>
>>>>
>>>> does the direct space calculation without changing the neighbor list
>>>>> search distance.
>>>>>
>>>>> Your auto-tuned run number 1 had rlist = rcoulomb at the start, so
>>>> mdrun
>>>> knows you wanted a PME model with an unbuffered list whose size equals
>>>> rcoulomb, and a buffered VDW model with rlist 1.4 and rvdw 1.2. Thus,
>>>> rlistlong will stay equal to rcoulomb as it changes. The details and
>>>> code
>>>> are horrible, though, and I am looking forward to nothing so much as
>>>> ripping it all out in about 2 months!
>>>>
>>>> And like Szilard suggested, your runs are probably a long way from
>>>> maximum
>>>> throughput. Aim for lots of sampling, don't chase replicating rare
>>>> events
>>>> with brute-force simulation!
>>>>
>>>> Mark
>>>>
>>>> Thank you
>>>>
>>>>> Best
>>>>> Jiaqi
>>>>>
>>>>>
>>>>> On 1/20/2015 3:54 PM, Szilárd Páll wrote:
>>>>>
>>>>> Not (all) directly related, but a few comments/questions:
>>>>>>
>>>>>> - Have you tried 4.6.7 or 5.0.4?
>>>>>> - Have you considered using the Verlet scheme instead of doing manual
>>>>>> buffering?
>>>>>> - lincs-order=8 is very large for 2fs production runs - typically 4 is
>>>>>> used.
>>>>>> - Your fourier spacing is a lot (~25%) finer than it needs to be.
>>>>>>
>>>>>> - The PP/PME rank split of 100+28 is _very_ inconvenient and it is the
>>>>>> main cause of the horrible PME performance together with the overly
>>>>>> coarse grid. That's why you get such a huge cut-off after the PP-PME
>>>>>> load balancing. Even if you want to stick to these parameters, you
>>>>>> should tune the rank split (manually or with tune_pme).
>>>>>> - The above contributes to the high neighbor search cost too.
>>>>>>
>>>>>> --
>>>>>> Szilárd
>>>>>>
>>>>>>
>>>>>> On Tue, Jan 20, 2015 at 9:18 PM, Jiaqi Lin <jqlin at mit.edu> wrote:
>>>>>>
>>>>>> Hi Mark,
>>>>>>>
>>>>>>> Thanks for reply. I put the md.log files in the following link
>>>>>>>
>>>>>>> https://www.dropbox.com/sh/d1d2fbwreizr974/
>>>>>>> AABYhSRU03nmijbTIXKKr-rra?dl=0
>>>>>>>
>>>>>>> There are four log files
>>>>>>> 1.GMX 4.6.5 -tunepme (the coulombic cutoff is tuned to 3.253)
>>>>>>> 2.GMX 4.6.5 -notunepme rcoulomb= 3.3 , fourierspace = 0.33
>>>>>>> 3.GMX 4.6.5 -notunepme rcoulomb= 3.3 , fourierspace = 0.14
>>>>>>> 4.GMX 4.6.5 -notunepme rcoulomb= 1.4 , fourierspace = 0.14
>>>>>>>
>>>>>>> Note that the LR Coulombic energy in the first one is almost twice
>>>>>>> the
>>>>>>> value
>>>>>>> of that in the second one, whereas the grid spacing in both cases are
>>>>>>> nealy
>>>>>>> the same.
>>>>>>>
>>>>>>> Only the first one gives a strong electrostatic interaction of a
>>>>>>> nanoparticle with a lipid bilayer under ionic imbalance. In other
>>>>>>> cases
>>>>>>> I do
>>>>>>> not observe such a strong interaction.
>>>>>>>
>>>>>>> GMX 5.0.1 give the same results as GMX 4.6.5 using Group cutoff.
>>>>>>> Thanks
>>>>>>>
>>>>>>>
>>>>>>> Regards
>>>>>>> Jiaqi
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 1/19/2015 3:22 PM, Mark Abraham wrote:
>>>>>>>
>>>>>>> On Thu, Jan 15, 2015 at 3:21 AM, Jiaqi Lin <jqlin at mit.edu> wrote:
>>>>>>>>
>>>>>>>> Dear GMX developers,
>>>>>>>>
>>>>>>>>> I've encounter a problem in GROMACS concerning the auto-tuning
>>>>>>>>> feature
>>>>>>>>> of
>>>>>>>>> PME that bugged me for months. As stated in the title, the
>>>>>>>>> auto-tuning
>>>>>>>>> feature of mdrun changed my coulomb cutoff from 1.4 nm to ~3.3 nm
>>>>>>>>> (stated
>>>>>>>>> in md.log) when I set -npme to be 28 (128 total CPU cores), and
>>>>>>>>> this
>>>>>>>>> giving
>>>>>>>>> me interesting simulation results. When I use -notunepme, I found
>>>>>>>>> Coulomb
>>>>>>>>> (SR) and recip. giving me same energy but the actual simulation
>>>>>>>>> result
>>>>>>>>> is
>>>>>>>>> different. This i can understand: scaling between coulombic
>>>>>>>>> cut-off/grid
>>>>>>>>> size theoretically give same accuracy to electrostatics (according
>>>>>>>>> to
>>>>>>>>> GMX
>>>>>>>>> manual and PME papers), but there actually some numerical error
>>>>>>>>> due to
>>>>>>>>> grid
>>>>>>>>> mapping and even if the energy is the same that does not mean
>>>>>>>>> system
>>>>>>>>> configuration has to be the same (NVE ensemble: constant energy,
>>>>>>>>> different
>>>>>>>>> configuration).
>>>>>>>>>
>>>>>>>>> Total electrostatic energy should be approximately the same with
>>>>>>>>>
>>>>>>>> different
>>>>>>>> PME partitions.
>>>>>>>>
>>>>>>>>
>>>>>>>> However the thing i don't understand is the following. I am
>>>>>>>> interested
>>>>>>>>
>>>>>>>>> in
>>>>>>>>> the result under large coulomb cut-off, so I try to manually set
>>>>>>>>> cut-off
>>>>>>>>> and grid space with -notunepme, using the value tuned by mdrun
>>>>>>>>> previously.
>>>>>>>>> This give me complete different simulation result, and the energy
>>>>>>>>> is
>>>>>>>>> also
>>>>>>>>> different. I've tried to set rlist, rlistlong, or both to equal
>>>>>>>>> rcoulomb
>>>>>>>>> (~3.3) still does not give me the result produced by auto-tuning
>>>>>>>>> PME.
>>>>>>>>>
>>>>>>>>> In what sense is the result different?
>>>>>>>>
>>>>>>>>
>>>>>>>> In addition, simulation speed dramatically reduces when I set
>>>>>>>> rcoulomb
>>>>>>>>
>>>>>>>>> to
>>>>>>>>> be ~3.3 (using -tunepme the speed remains nearly the same no
>>>>>>>>> matter how
>>>>>>>>> large the cutoff is tuned to). I've tested this in both GMX 4.6.5
>>>>>>>>> and
>>>>>>>>> 5.0.1, same thing happens, so clearly it's not because of versions.
>>>>>>>>> Thus
>>>>>>>>> the question is: what exactly happened to PME calcualtion using the
>>>>>>>>> auto-tuning feature in mdrun, why it does give different results
>>>>>>>>> when I
>>>>>>>>> manually set the coulomb cutoff and grid space to the value tuned
>>>>>>>>> by
>>>>>>>>> mdrun
>>>>>>>>> without the auto-tuning feature (using -notunepme)? Thank you for
>>>>>>>>> help.
>>>>>>>>>
>>>>>>>>> For the group scheme, these should all lead to essentially the
>>>>>>>>> same
>>>>>>>>>
>>>>>>>> result
>>>>>>>> and (if tuned) performance. If you can share your various log files
>>>>>>>> on a
>>>>>>>> file-sharing service (rc 1.4, rc 3.3, various -tunepme settings,
>>>>>>>> 4.6.5
>>>>>>>> and
>>>>>>>> 5.0.1) then we can be in a position to comment further.
>>>>>>>>
>>>>>>>> Mark
>>>>>>>>
>>>>>>>>
>>>>>>>> additional info: I use Group cutoff-scheme , rvdw is 1.2.
>>>>>>>>
>>>>>>>>>
>>>>>>>>> md.log file:
>>>>>>>>> DD step 9 load imb.: force 29.4% pme mesh/force 3.627
>>>>>>>>>
>>>>>>>>> step 30: timed with pme grid 280 280 384, coulomb cutoff 1.400:
>>>>>>>>> 1026.4
>>>>>>>>> M-cycles
>>>>>>>>> step 50: timed with pme grid 256 256 324, coulomb cutoff 1.464:
>>>>>>>>> 850.3
>>>>>>>>> M-cycles
>>>>>>>>> step 70: timed with pme grid 224 224 300, coulomb cutoff 1.626:
>>>>>>>>> 603.6
>>>>>>>>> M-cycles
>>>>>>>>> step 90: timed with pme grid 200 200 280, coulomb cutoff 1.822:
>>>>>>>>> 555.2
>>>>>>>>> M-cycles
>>>>>>>>> step 110: timed with pme grid 160 160 208, coulomb cutoff 2.280:
>>>>>>>>> 397.0
>>>>>>>>> M-cycles
>>>>>>>>> step 130: timed with pme grid 144 144 192, coulomb cutoff 2.530:
>>>>>>>>> 376.0
>>>>>>>>> M-cycles
>>>>>>>>> step 150: timed with pme grid 128 128 160, coulomb cutoff 2.964:
>>>>>>>>> 343.7
>>>>>>>>> M-cycles
>>>>>>>>> step 170: timed with pme grid 112 112 144, coulomb cutoff 3.294:
>>>>>>>>> 334.8
>>>>>>>>> M-cycles
>>>>>>>>> Grid: 12 x 14 x 14 cells
>>>>>>>>> step 190: timed with pme grid 84 84 108, coulomb cutoff 4.392:
>>>>>>>>> 346.2
>>>>>>>>> M-cycles
>>>>>>>>> step 190: the PME grid restriction limits the PME load balancing
>>>>>>>>> to a
>>>>>>>>> coulomb cut-off of 4.392
>>>>>>>>> step 210: timed with pme grid 128 128 192, coulomb cutoff 2.846:
>>>>>>>>> 360.6
>>>>>>>>> M-cycles
>>>>>>>>> step 230: timed with pme grid 128 128 160, coulomb cutoff 2.964:
>>>>>>>>> 343.6
>>>>>>>>> M-cycles
>>>>>>>>> step 250: timed with pme grid 120 120 160, coulomb cutoff 3.036:
>>>>>>>>> 340.4
>>>>>>>>> M-cycles
>>>>>>>>> step 270: timed with pme grid 112 112 160, coulomb cutoff 3.253:
>>>>>>>>> 334.3
>>>>>>>>> M-cycles
>>>>>>>>> step 290: timed with pme grid 112 112 144, coulomb cutoff 3.294:
>>>>>>>>> 334.7
>>>>>>>>> M-cycles
>>>>>>>>> step 310: timed with pme grid 84 84 108, coulomb cutoff 4.392:
>>>>>>>>> 348.0
>>>>>>>>> M-cycles
>>>>>>>>> optimal pme grid 112 112 160, coulomb cutoff 3.253
>>>>>>>>> DD step 999 load imb.: force 18.4% pme mesh/force 0.918
>>>>>>>>>
>>>>>>>>> At step 1000 the performance loss due to force load imbalance is
>>>>>>>>> 6.3 %
>>>>>>>>>
>>>>>>>>> NOTE: Turning on dynamic load balancing
>>>>>>>>>
>>>>>>>>> Step Time Lambda
>>>>>>>>> 1000 20.00000 0.00000
>>>>>>>>>
>>>>>>>>> Energies (kJ/mol)
>>>>>>>>> Bond G96Angle LJ (SR) Coulomb (SR)
>>>>>>>>> Coul.
>>>>>>>>> recip.
>>>>>>>>> 1.98359e+05 1.79181e+06 -1.08927e+07 -7.04736e+06
>>>>>>>>> -2.32682e+05
>>>>>>>>> Position Rest. Potential Kinetic En. Total Energy
>>>>>>>>> Temperature
>>>>>>>>> 6.20627e+04 -1.61205e+07 4.34624e+06 -1.17743e+07
>>>>>>>>> 3.00659e+02
>>>>>>>>> Pressure (bar) Constr. rmsd
>>>>>>>>> 2.13582e+00 1.74243e-04
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Best
>>>>>>>>> Jiaqi
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Jiaqi Lin
>>>>>>>>> postdoc fellow
>>>>>>>>> The Langer Lab
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Gromacs Users mailing list
>>>>>>>>>
>>>>>>>>> * Please search the archive at http://www.gromacs.org/
>>>>>>>>> Support/Mailing_Lists/GMX-Users_List before posting!
>>>>>>>>>
>>>>>>>>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>>>>>>>>>
>>>>>>>>> * For (un)subscribe requests visit
>>>>>>>>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users
>>>>>>>>> or
>>>>>>>>> send a mail to gmx-users-request at gromacs.org.
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>>
>>>>>>>> Jiaqi Lin
>>>>>>> postdoc fellow
>>>>>>> The Langer Lab
>>>>>>>
>>>>>>> --
>>>>>>> Gromacs Users mailing list
>>>>>>>
>>>>>>> * Please search the archive at
>>>>>>> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
>>>>>>> posting!
>>>>>>>
>>>>>>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>>>>>>>
>>>>>>> * For (un)subscribe requests visit
>>>>>>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users
>>>>>>> or
>>>>>>> send a
>>>>>>> mail to gmx-users-request at gromacs.org.
>>>>>>>
>>>>>>> --
>>>>> Jiaqi Lin
>>>>> postdoc fellow
>>>>> The Langer Lab
>>>>> David H. Koch Institute for Integrative Cancer Research
>>>>> Massachusetts Institute of Technology
>>>>>
>>>>>
>>>>> --
>>>>> Gromacs Users mailing list
>>>>>
>>>>> * Please search the archive at http://www.gromacs.org/
>>>>> Support/Mailing_Lists/GMX-Users_List before posting!
>>>>>
>>>>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>>>>>
>>>>> * For (un)subscribe requests visit
>>>>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
>>>>> send a mail to gmx-users-request at gromacs.org.
>>>>>
>>>>>
>>> --
>>> Jiaqi Lin
>>> postdoc fellow
>>> The Langer Lab
>>>
>>> --
>>> Gromacs Users mailing list
>>>
>>> * Please search the archive at http://www.gromacs.org/
>>> Support/Mailing_Lists/GMX-Users_List before posting!
>>>
>>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>>>
>>> * For (un)subscribe requests visit
>>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
>>> send a mail to gmx-users-request at gromacs.org.
>>>
>>
>>
>
More information about the gromacs.org_gmx-users
mailing list