[gmx-users] results produced by auto-tuning of Coulomb cut-off/grid for PME can not be reproduced by manually setting the Coulomb cut-off and grid spacing
pall.szilard at gmail.com
Tue Jan 20 22:01:07 CET 2015
Not (all) directly related, but a few comments/questions:
- Have you tried 4.6.7 or 5.0.4?
- Have you considered using the Verlet scheme instead of doing manual buffering?
- lincs-order=8 is very large for 2fs production runs - typically 4 is used.
- Your fourier spacing is a lot (~25%) finer than it needs to be.
- The PP/PME rank split of 100+28 is _very_ inconvenient and it is the
main cause of the horrible PME performance together with the overly
coarse grid. That's why you get such a huge cut-off after the PP-PME
load balancing. Even if you want to stick to these parameters, you
should tune the rank split (manually or with tune_pme).
- The above contributes to the high neighbor search cost too.
On Tue, Jan 20, 2015 at 9:18 PM, Jiaqi Lin <jqlin at mit.edu> wrote:
> Hi Mark,
> Thanks for reply. I put the md.log files in the following link
> There are four log files
> 1.GMX 4.6.5 -tunepme (the coulombic cutoff is tuned to 3.253)
> 2.GMX 4.6.5 -notunepme rcoulomb= 3.3 , fourierspace = 0.33
> 3.GMX 4.6.5 -notunepme rcoulomb= 3.3 , fourierspace = 0.14
> 4.GMX 4.6.5 -notunepme rcoulomb= 1.4 , fourierspace = 0.14
> Note that the LR Coulombic energy in the first one is almost twice the value
> of that in the second one, whereas the grid spacing in both cases are nealy
> the same.
> Only the first one gives a strong electrostatic interaction of a
> nanoparticle with a lipid bilayer under ionic imbalance. In other cases I do
> not observe such a strong interaction.
> GMX 5.0.1 give the same results as GMX 4.6.5 using Group cutoff. Thanks
> On 1/19/2015 3:22 PM, Mark Abraham wrote:
>> On Thu, Jan 15, 2015 at 3:21 AM, Jiaqi Lin <jqlin at mit.edu> wrote:
>>> Dear GMX developers,
>>> I've encounter a problem in GROMACS concerning the auto-tuning feature of
>>> PME that bugged me for months. As stated in the title, the auto-tuning
>>> feature of mdrun changed my coulomb cutoff from 1.4 nm to ~3.3 nm (stated
>>> in md.log) when I set -npme to be 28 (128 total CPU cores), and this
>>> me interesting simulation results. When I use -notunepme, I found Coulomb
>>> (SR) and recip. giving me same energy but the actual simulation result is
>>> different. This i can understand: scaling between coulombic cut-off/grid
>>> size theoretically give same accuracy to electrostatics (according to GMX
>>> manual and PME papers), but there actually some numerical error due to
>>> mapping and even if the energy is the same that does not mean system
>>> configuration has to be the same (NVE ensemble: constant energy,
>> Total electrostatic energy should be approximately the same with different
>> PME partitions.
>>> However the thing i don't understand is the following. I am interested in
>>> the result under large coulomb cut-off, so I try to manually set cut-off
>>> and grid space with -notunepme, using the value tuned by mdrun
>>> This give me complete different simulation result, and the energy is also
>>> different. I've tried to set rlist, rlistlong, or both to equal rcoulomb
>>> (~3.3) still does not give me the result produced by auto-tuning PME.
>> In what sense is the result different?
>>> In addition, simulation speed dramatically reduces when I set rcoulomb to
>>> be ~3.3 (using -tunepme the speed remains nearly the same no matter how
>>> large the cutoff is tuned to). I've tested this in both GMX 4.6.5 and
>>> 5.0.1, same thing happens, so clearly it's not because of versions. Thus
>>> the question is: what exactly happened to PME calcualtion using the
>>> auto-tuning feature in mdrun, why it does give different results when I
>>> manually set the coulomb cutoff and grid space to the value tuned by
>>> without the auto-tuning feature (using -notunepme)? Thank you for help.
>> For the group scheme, these should all lead to essentially the same result
>> and (if tuned) performance. If you can share your various log files on a
>> file-sharing service (rc 1.4, rc 3.3, various -tunepme settings, 4.6.5 and
>> 5.0.1) then we can be in a position to comment further.
>>> additional info: I use Group cutoff-scheme , rvdw is 1.2.
>>> md.log file:
>>> DD step 9 load imb.: force 29.4% pme mesh/force 3.627
>>> step 30: timed with pme grid 280 280 384, coulomb cutoff 1.400: 1026.4
>>> step 50: timed with pme grid 256 256 324, coulomb cutoff 1.464: 850.3
>>> step 70: timed with pme grid 224 224 300, coulomb cutoff 1.626: 603.6
>>> step 90: timed with pme grid 200 200 280, coulomb cutoff 1.822: 555.2
>>> step 110: timed with pme grid 160 160 208, coulomb cutoff 2.280: 397.0
>>> step 130: timed with pme grid 144 144 192, coulomb cutoff 2.530: 376.0
>>> step 150: timed with pme grid 128 128 160, coulomb cutoff 2.964: 343.7
>>> step 170: timed with pme grid 112 112 144, coulomb cutoff 3.294: 334.8
>>> Grid: 12 x 14 x 14 cells
>>> step 190: timed with pme grid 84 84 108, coulomb cutoff 4.392: 346.2
>>> step 190: the PME grid restriction limits the PME load balancing to a
>>> coulomb cut-off of 4.392
>>> step 210: timed with pme grid 128 128 192, coulomb cutoff 2.846: 360.6
>>> step 230: timed with pme grid 128 128 160, coulomb cutoff 2.964: 343.6
>>> step 250: timed with pme grid 120 120 160, coulomb cutoff 3.036: 340.4
>>> step 270: timed with pme grid 112 112 160, coulomb cutoff 3.253: 334.3
>>> step 290: timed with pme grid 112 112 144, coulomb cutoff 3.294: 334.7
>>> step 310: timed with pme grid 84 84 108, coulomb cutoff 4.392: 348.0
>>> optimal pme grid 112 112 160, coulomb cutoff 3.253
>>> DD step 999 load imb.: force 18.4% pme mesh/force 0.918
>>> At step 1000 the performance loss due to force load imbalance is 6.3 %
>>> NOTE: Turning on dynamic load balancing
>>> Step Time Lambda
>>> 1000 20.00000 0.00000
>>> Energies (kJ/mol)
>>> Bond G96Angle LJ (SR) Coulomb (SR) Coul.
>>> 1.98359e+05 1.79181e+06 -1.08927e+07 -7.04736e+06
>>> Position Rest. Potential Kinetic En. Total Energy Temperature
>>> 6.20627e+04 -1.61205e+07 4.34624e+06 -1.17743e+07 3.00659e+02
>>> Pressure (bar) Constr. rmsd
>>> 2.13582e+00 1.74243e-04
>>> Jiaqi Lin
>>> postdoc fellow
>>> The Langer Lab
>>> Gromacs Users mailing list
>>> * Please search the archive at http://www.gromacs.org/
>>> Support/Mailing_Lists/GMX-Users_List before posting!
>>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>>> * For (un)subscribe requests visit
>>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
>>> send a mail to gmx-users-request at gromacs.org.
> Jiaqi Lin
> postdoc fellow
> The Langer Lab
> Gromacs Users mailing list
> * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting!
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a
> mail to gmx-users-request at gromacs.org.
More information about the gromacs.org_gmx-users