[gmx-users] different nvidia-smi/gmx GPU_IDs

[paul buscemi]

Szilard,

Is the environmental variable set at build  ?

thanks
Paul

> On Jan 18, 2019, at 12:36 PM, Szilárd Páll <pall.szilard at gmail.com> wrote:
> 
> Hi,
> 
> The CUDA runtime tries (and AFAIK has always tried) to be smart about
> device order which is what GROMACS will see in its detection. The
> nvidia-smi monitoring tools however uses a different mechanism for
> enumeration that will always respect the PCI identifier of the devices (~
> the order of cards/slots in the box).
> 
> This can of course cause some headache in mixed setups, but you can set the
> CUDA_DEVICE_ORDER=PCI_BUS_ID environment variable to tell the runtime to
> avoid reordering the GPUs and expose them ordered by bus ID.
> 
> Cheers,
> --
> Szilárd
> 
> 
> On Sun, Jan 13, 2019 at 2:27 PM Tamas Hegedus <tamas at hegelab.org> wrote:
> 
>> Hi,
>> 
>> I have a node with 4 nvidia GPUs.
>> From nvidia-smi output:
>>  0  Quadro P6000
>>  1  GeForce RTX 208
>>  2  GeForce GTX 108
>>  3  GeForce RTX 208
>> 
>> However, the order of GPUs is differently detected by gmx 2018.3
>>     Number of GPUs detected: 4
>>     #0: NVIDIA GeForce RTX 2080 Ti
>>     #1: NVIDIA GeForce RTX 2080 Ti
>>     #2: NVIDIA Quadro P6000
>>     #3: NVIDIA GeForce GTX 1080 Ti
>> 
>> Why is this? This makes difficult to plan/check the running of two gmx
>> jobs on the same node.
>> 
>> Thanks for your suggestion.
>> 
>> Tamas
>> 
>> --
>> Tamas Hegedus, PhD
>> Senior Research Fellow
>> MTA-SE Molecular Biophysics Research Group
>> Hungarian Academy of Sciences  | phone: (36) 1-459 1500/60233
>> Semmelweis University          | fax:   (36) 1-266 6656
>> Tuzolto utca 37-47             | mailto:tamas at hegelab.org
>> Budapest, 1094, Hungary        | http://www.hegelab.org
>> 
>> ---
>> This email has been checked for viruses by AVG.
>> https://www.avg.com
>> 
>> --
>> Gromacs Users mailing list
>> 
>> * Please search the archive at
>> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
>> posting!
>> 
>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>> 
>> * For (un)subscribe requests visit
>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
>> send a mail to gmx-users-request at gromacs.org.
>> 
> -- 
> Gromacs Users mailing list
> 
> * Please search the archive at http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting!
> 
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> 
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a mail to gmx-users-request at gromacs.org.