[gmx-users] different nvidia-smi/gmx GPU_IDs

[Szilárd Páll]

Hi,

The CUDA runtime tries (and AFAIK has always tried) to be smart about
device order which is what GROMACS will see in its detection. The
nvidia-smi monitoring tools however uses a different mechanism for
enumeration that will always respect the PCI identifier of the devices (~
the order of cards/slots in the box).

This can of course cause some headache in mixed setups, but you can set the
CUDA_DEVICE_ORDER=PCI_BUS_ID environment variable to tell the runtime to
avoid reordering the GPUs and expose them ordered by bus ID.

Cheers,
--
Szilárd


On Sun, Jan 13, 2019 at 2:27 PM Tamas Hegedus <tamas at hegelab.org> wrote:

> Hi,
>
> I have a node with 4 nvidia GPUs.
>  From nvidia-smi output:
>   0  Quadro P6000
>   1  GeForce RTX 208
>   2  GeForce GTX 108
>   3  GeForce RTX 208
>
> However, the order of GPUs is differently detected by gmx 2018.3
>      Number of GPUs detected: 4
>      #0: NVIDIA GeForce RTX 2080 Ti
>      #1: NVIDIA GeForce RTX 2080 Ti
>      #2: NVIDIA Quadro P6000
>      #3: NVIDIA GeForce GTX 1080 Ti
>
> Why is this? This makes difficult to plan/check the running of two gmx
> jobs on the same node.
>
> Thanks for your suggestion.
>
> Tamas
>
> --
> Tamas Hegedus, PhD
> Senior Research Fellow
> MTA-SE Molecular Biophysics Research Group
> Hungarian Academy of Sciences  | phone: (36) 1-459 1500/60233
> Semmelweis University          | fax:   (36) 1-266 6656
> Tuzolto utca 37-47             | mailto:tamas at hegelab.org
> Budapest, 1094, Hungary        | http://www.hegelab.org
>
> ---
> This email has been checked for viruses by AVG.
> https://www.avg.com
>
> --
> Gromacs Users mailing list
>
> * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> send a mail to gmx-users-request at gromacs.org.
>