[gmx-users] Updating GTX670 PCIE speed from 5GT/s to 8GT/s resulted in about 10% speedup of md_run.

Henk Neefs henk.neefs at gmail.com
Sun Dec 1 04:24:58 CET 2013

By configuring 8 GT/s PCIE 3.0 for the Nvidia driver, I got a 10% speedup
on Gromacs md_run.
   45 ns/day -> 50 ns/day (1AKI protein).

This posting is just informational, my findings on how to do this, so
others can possibly also exploit this if they desire so. There is no
question that i'm asking here.

Config: Intel Ivytown (Ivybridge family processor (i7-4960X).
        Nvidia GeForce GTX670. Single GPU card installed.
        ASUS X79 Deluxe Motherboard. Single socket system.
        Fedora 19
        Nvidia driver: 331.20
Application: md_run
        Gromacs 4.6.4
        1AKI protein from tutorial

To measure PCIe Link Speed, do either of:
  1. nvidia-settings  (from cmd-line, this is part of the Nvidia drivers
     Look under PowerMizer to see what PCIe Link Speed is presently used
(will change under load).
     run md_run to generate a load. If 8GT/s is not enabled then expect to
see 5 GT/s during an md_run.

  2. lspci -vv | grep -i nvidia
     Use the device PCI reg address to get details (needs root privileges):
     lspci -vv  -s 01:00.0
     You would see something like:
        LnkCap: Port #0, Speed 8GT/s, Width x16    => PCIE Capability is
set to 8 GT/s here (this is after applying below settings).
        LnkSta: Speed 2.5GT/s, Width x16              => PCIE Link State
presently low speed as md_run is not running (to save power).

If the capability shows 5 GT/s (or speed is just 5 GT/s under md_run load)
then you can try below setting to elevate to 8 GT/s.

I'm using the parameter NVreg_EnablePCIeGen3=1 as provided by the Nvidia

1. I tried it first during a single Fedora boot and ran md_run (a 30 mins
wall-time example) to determine whether it's stable.
   During boot, when the boot options show up for the images:
    Press 'e' to edit the cmd line.
    Add to the cmd line:
    Then 'Ctrl-X' to boot with the cmd-line (note: this option will be
forgotten on the next boot).
    Run md_run or some other tests that heavily exercise the graphics card
to gauge the stability.

2. To make the change permanent (as root):
    edit /etc/default/grub and add to the GRUB_CMDLINE_LINUX option:
    grub2-mkconfig -o /boot/grub2/grub.cfg
   Now reboot and the setting will take effect from then onwards (for every

FYI: Note that I'm using an Intel i7-4960X (Ivybridge family: 15MB, 6
Cores). It seems to support PCIE 3.0 with 8 GT/s (I'm not speaking for

Disclaimer: Nvidia does not guarantee link/system stability when doing
this. I don't either. It works for me. Your mileage may vary. I only have a
single GPU card on the PCIE bus.

Henk Neefs
Gromacs User
Computer Architect (Ivytown chip architect at Intel).
Not speaking for Intel, these are all personal opinions.

More information about the gromacs.org_gmx-users mailing list