[gmx-users] Gomacs and multiple GPU

Szilárd Páll pall.szilard at gmail.com
Thu Dec 1 18:28:24 CET 2016 

Hi,

Welcome!

> Thank for you reply. Yes, i have read them and know them. I use them
will success with other software.

BTW, what did you refer to here? It seems like you were focused on
figuring out the detection issue and might have ignored my other
recommendations on using multi (which incidentally would have solved
the original issue too as well as would do correct job placement for
you) and 2+ ranks per GPU?
Cheers,
--
Szilárd


On Thu, Dec 1, 2016 at 6:12 PM, Alexis Michon <alexis.michon at ibcp.fr> wrote:
> Hi,
>
> You solve my problem, Thank you.
>
> The variable CUDA_VISIBLE_DEVICES is set up by the scheduler at the
> running time of my job. So after adding commandline "unset
> $CUDA_VISIBLE_DEVICES" in my script, mdrun detect and use both GPUs.
>
> Thank you for your patience.
>
> Cheers,
> Alexis
>
> On 01/12/16 17:53, Szilárd Páll wrote:
>> Hi,
>>
>> Do any other programs, e.g. CUDA SDK samples (like deviceQuery) detect
>> both GPUs?
>> I find it *very-very* unlikely that only mdrun would not be able
>> detect both GPUs.
>>
>> Again, are you sure it's not the use of CUDA_VISIBLE_DEVICES that
>> creates the confusion? Please share your full command line (with env
>> vars an all) and resulting log output?
>>
>> Cheers,
>> --
>> Szilárd
>>
>>
>> On Thu, Dec 1, 2016 at 5:10 PM, Alexis Michon <alexis.michon at ibcp.fr> wrote:
>>> Hi,
>>>
>>> Thank for you reply. Yes, i have read them and know them. I use them
>>> will success with other software.
>>>
>>> My first and main problem is mdrun doesn't see the second GPU on the
>>> system. And i try to find the root cause of and the remediation.
>>>
>>> Hardware is a bi xeon with 2 Titan-xp GPU,  so my goal is to run 2
>>> differentes simulations with two mdrun process each with one dedicated GPU.
>>>
>>> Do you have any idea ?
>>>
>>> Cheers
>>> Alexis
>>>
>>> On 01/12/16 16:38, Szilárd Páll wrote:
>>>> Alexis,
>>>>
>>>> What is the problem? Have you read the relevant docs of what
>>>> CUDA_VISIBLE_DEVICES does?
>>>> https://docs.nvidia.com/cuda/cuda-c-programming-guide/#env-vars
>>>>
>>>>
>>>> BTW,  you can use, but you *do not need* CUDA_VISIBLE devices to
>>>> control the mapping of GPU(s) to mdrun process, the -gpu_id variable
>>>> (or equivalent env var) alone is enough.
>>>>
>>>> To run multiple processes per node, do consider:
>>>> - using mdrun -multi
>>>> - using 2 or more mdrun runs per GPU
>>>>
>>>> You can find the reason, motivating performance examples, and sample
>>>> commands here (in particular Fig 5 and related section):
>>>> dx.doi.org/10.1002/jcc.24030
>>>>
>>>> Cheers,
>>>> --
>>>> Szilárd
>>>>
>>>>
>>>> On Thu, Dec 1, 2016 at 3:38 PM, Alexis Michon <alexis.michon at ibcp.fr> wrote:
>>>>> Hi,
>>>>>
>>>>> thank you for your answer
>>>>>
>>>>> the Cmdline is:
>>>>> gmx mdrun  -deffnm nvt
>>>>>
>>>>> I played a bit with option -gpu_id, use $CUDA_VISIBLE_DEVICES  and read
>>>>> the page http://www.gromacs.org/GPU_acceleration without any succes. One
>>>>> output is in nvt.log.11,
>>>>>
>>>>> nvidia-smi returs is in  nvidia-smi.log
>>>>>
>>>>> Nvidia-smi detect 2 GPUs, but not mdrun.
>>>>>
>>>>> nvidia-smi.log : https://icloud.ibcp.fr/index.php/s/08jqFsZpWNlSUGh
>>>>> nvt.log : https://icloud.ibcp.fr/index.php/s/vC5UxCCAhyw9FuC
>>>>>
>>>>> Alexis
>>>>>
>>>>>
>>>>>
>>>>> On 01/12/16 11:34, Mark Abraham wrote:
>>>>>> Hi,
>>>>>>
>>>>>> On Wed, Nov 30, 2016 at 5:38 PM Alexis Michon <alexis.michon at ibcp.fr> wrote:
>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> We have build gromacs 2016.1 from source with "DMX_GPU=on" on a bi
>>>>>>> processor bi GPU machine, mdrun detect and run fine on only 1 GPU. How
>>>>>>> could we force mdrun to detect the second GPU ?
>>>>>>>
>>>>>> If it's compatible, powered and supported by your driver, then mdrun will
>>>>>> find it. Presumably nvidia-smi tool will help you work out what's going on.
>>>>>>
>>>>>> We would like to run 2 mdrun instances on a machine equiped with 2 GPUs,
>>>>>>> each mdrun will use 1GPU. How could we tell  mdrun to use a specific GPU ?
>>>>>>>
>>>>>> Guidance is here
>>>>>> http://manual.gromacs.org/documentation/2016.1/user-guide/mdrun-performance.html#examples-for-mdrun-on-one-node
>>>>>>
>>>>>> Mark
>>>>>>
>>>>>>
>>>>>>> Cheers,
>>>>>>> Alexis
>>>>>>>
>>>>>>> --
>>>>>>> Citation : "It’s not enough to be busy; so are the ants. The question is:
>>>>>>> what are we busy about?" - Henry David Thoreau
>>>>>>> Alexis MICHON, responsable informatique
>>>>>>> CNRS IBCP, 7 passage du vercors, 69007 LYON, France
>>>>>>> Mail : alexis.michon at ibcp.fr  Tel : 04.72.72.26.03 - 06.27.56.34.80
>>>>>>> CNRS IBCP - UMS 5760 - http://www.ibcp.fr/
>>>>>>> Empreinte : C9:45:2D:7C:79:7F:0B:79:CA:C8:0B:68:41:A2:8C:EE:EA:72:82:34
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Gromacs Users mailing list
>>>>>>>
>>>>>>> * Please search the archive at
>>>>>>> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
>>>>>>> posting!
>>>>>>>
>>>>>>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>>>>>>>
>>>>>>> * For (un)subscribe requests visit
>>>>>>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
>>>>>>> send a mail to gmx-users-request at gromacs.org.
>>>>> --
>>>>> Citation : "It’s not enough to be busy; so are the ants. The question is: what are we busy about?" - Henry David Thoreau
>>>>> Alexis MICHON, responsable informatique
>>>>> CNRS IBCP, 7 passage du vercors, 69007 LYON, France
>>>>> Mail : alexis.michon at ibcp.fr  Tel : 04.72.72.26.03 - 06.27.56.34.80
>>>>> CNRS IBCP - UMS 5760 - http://www.ibcp.fr/
>>>>> Empreinte : C9:45:2D:7C:79:7F:0B:79:CA:C8:0B:68:41:A2:8C:EE:EA:72:82:34
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Gromacs Users mailing list
>>>>>
>>>>> * Please search the archive at http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting!
>>>>>
>>>>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>>>>>
>>>>> * For (un)subscribe requests visit
>>>>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a mail to gmx-users-request at gromacs.org.
>>> --
>>> Citation : "It’s not enough to be busy; so are the ants. The question is: what are we busy about?" - Henry David Thoreau
>>> Alexis MICHON, responsable informatique
>>> CNRS IBCP, 7 passage du vercors, 69007 LYON, France
>>> Mail : alexis.michon at ibcp.fr  Tel : 04.72.72.26.03 - 06.27.56.34.80
>>> CNRS IBCP - UMS 5760 - http://www.ibcp.fr/
>>> Empreinte : C9:45:2D:7C:79:7F:0B:79:CA:C8:0B:68:41:A2:8C:EE:EA:72:82:34
>>>
>>>
>>>
>>> --
>>> Gromacs Users mailing list
>>>
>>> * Please search the archive at http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting!
>>>
>>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>>>
>>> * For (un)subscribe requests visit
>>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a mail to gmx-users-request at gromacs.org.
>
> --
> Citation : "It’s not enough to be busy; so are the ants. The question is: what are we busy about?" - Henry David Thoreau
> Alexis MICHON, responsable informatique
> CNRS IBCP, 7 passage du vercors, 69007 LYON, France
> Mail : alexis.michon at ibcp.fr  Tel : 04.72.72.26.03 - 06.27.56.34.80
> CNRS IBCP - UMS 5760 - http://www.ibcp.fr/
> Empreinte : C9:45:2D:7C:79:7F:0B:79:CA:C8:0B:68:41:A2:8C:EE:EA:72:82:34
>
>
>
> --
> Gromacs Users mailing list
>
> * Please search the archive at http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a mail to gmx-users-request at gromacs.org.

More information about the gromacs.org_gmx-users mailing list