[gmx-users] Gomacs and multiple GPU

Alexis Michon alexis.michon at ibcp.fr
Fri Dec 2 09:15:35 CET 2016 

Hi,

Others software doesn't assist the user as gromacs, they don't use
CUDA_VISIBLE without an action by the user (with an option, or a
specific environnement variable), in case the use this variable they
wrote a bit more information, like what they detect and will be used. So
i was convince gromacs works in the same way even though you wrote in
your mail. sorry :(

Alexis

On 01/12/16 18:28, Szilárd Páll wrote:
> Hi,
>
> Welcome!
>
>> Thank for you reply. Yes, i have read them and know them. I use them
> will success with other software.
>
> BTW, what did you refer to here? It seems like you were focused on
> figuring out the detection issue and might have ignored my other
> recommendations on using multi (which incidentally would have solved
> the original issue too as well as would do correct job placement for
> you) and 2+ ranks per GPU?
> Cheers,
> --
> Szilárd
>
>
> On Thu, Dec 1, 2016 at 6:12 PM, Alexis Michon <alexis.michon at ibcp.fr> wrote:
>> Hi,
>>
>> You solve my problem, Thank you.
>>
>> The variable CUDA_VISIBLE_DEVICES is set up by the scheduler at the
>> running time of my job. So after adding commandline "unset
>> $CUDA_VISIBLE_DEVICES" in my script, mdrun detect and use both GPUs.
>>
>> Thank you for your patience.
>>
>> Cheers,
>> Alexis
>>
>> On 01/12/16 17:53, Szilárd Páll wrote:
>>> Hi,
>>>
>>> Do any other programs, e.g. CUDA SDK samples (like deviceQuery) detect
>>> both GPUs?
>>> I find it *very-very* unlikely that only mdrun would not be able
>>> detect both GPUs.
>>>
>>> Again, are you sure it's not the use of CUDA_VISIBLE_DEVICES that
>>> creates the confusion? Please share your full command line (with env
>>> vars an all) and resulting log output?
>>>
>>> Cheers,
>>> --
>>> Szilárd
>>>
>>>
>>> On Thu, Dec 1, 2016 at 5:10 PM, Alexis Michon <alexis.michon at ibcp.fr> wrote:
>>>> Hi,
>>>>
>>>> Thank for you reply. Yes, i have read them and know them. I use them
>>>> will success with other software.
>>>>
>>>> My first and main problem is mdrun doesn't see the second GPU on the
>>>> system. And i try to find the root cause of and the remediation.
>>>>
>>>> Hardware is a bi xeon with 2 Titan-xp GPU,  so my goal is to run 2
>>>> differentes simulations with two mdrun process each with one dedicated GPU.
>>>>
>>>> Do you have any idea ?
>>>>
>>>> Cheers
>>>> Alexis
>>>>
>>>> On 01/12/16 16:38, Szilárd Páll wrote:
>>>>> Alexis,
>>>>>
>>>>> What is the problem? Have you read the relevant docs of what
>>>>> CUDA_VISIBLE_DEVICES does?
>>>>> https://docs.nvidia.com/cuda/cuda-c-programming-guide/#env-vars
>>>>>
>>>>>
>>>>> BTW,  you can use, but you *do not need* CUDA_VISIBLE devices to
>>>>> control the mapping of GPU(s) to mdrun process, the -gpu_id variable
>>>>> (or equivalent env var) alone is enough.
>>>>>
>>>>> To run multiple processes per node, do consider:
>>>>> - using mdrun -multi
>>>>> - using 2 or more mdrun runs per GPU
>>>>>
>>>>> You can find the reason, motivating performance examples, and sample
>>>>> commands here (in particular Fig 5 and related section):
>>>>> dx.doi.org/10.1002/jcc.24030
>>>>>
>>>>> Cheers,
>>>>> --
>>>>> Szilárd
>>>>>
>>>>>
>>>>> On Thu, Dec 1, 2016 at 3:38 PM, Alexis Michon <alexis.michon at ibcp.fr> wrote:
>>>>>> Hi,
>>>>>>
>>>>>> thank you for your answer
>>>>>>
>>>>>> the Cmdline is:
>>>>>> gmx mdrun  -deffnm nvt
>>>>>>
>>>>>> I played a bit with option -gpu_id, use $CUDA_VISIBLE_DEVICES  and read
>>>>>> the page http://www.gromacs.org/GPU_acceleration without any succes. One
>>>>>> output is in nvt.log.11,
>>>>>>
>>>>>> nvidia-smi returs is in  nvidia-smi.log
>>>>>>
>>>>>> Nvidia-smi detect 2 GPUs, but not mdrun.
>>>>>>
>>>>>> nvidia-smi.log : https://icloud.ibcp.fr/index.php/s/08jqFsZpWNlSUGh
>>>>>> nvt.log : https://icloud.ibcp.fr/index.php/s/vC5UxCCAhyw9FuC
>>>>>>
>>>>>> Alexis
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 01/12/16 11:34, Mark Abraham wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> On Wed, Nov 30, 2016 at 5:38 PM Alexis Michon <alexis.michon at ibcp.fr> wrote:
>>>>>>>
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> We have build gromacs 2016.1 from source with "DMX_GPU=on" on a bi
>>>>>>>> processor bi GPU machine, mdrun detect and run fine on only 1 GPU. How
>>>>>>>> could we force mdrun to detect the second GPU ?
>>>>>>>>
>>>>>>> If it's compatible, powered and supported by your driver, then mdrun will
>>>>>>> find it. Presumably nvidia-smi tool will help you work out what's going on.
>>>>>>>
>>>>>>> We would like to run 2 mdrun instances on a machine equiped with 2 GPUs,
>>>>>>>> each mdrun will use 1GPU. How could we tell  mdrun to use a specific GPU ?
>>>>>>>>
>>>>>>> Guidance is here
>>>>>>> http://manual.gromacs.org/documentation/2016.1/user-guide/mdrun-performance.html#examples-for-mdrun-on-one-node
>>>>>>>
>>>>>>> Mark
>>>>>>>
>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> Alexis
>>>>>>>>
>>>>>>>> --
>>>>>>>> Citation : "It’s not enough to be busy; so are the ants. The question is:
>>>>>>>> what are we busy about?" - Henry David Thoreau
>>>>>>>> Alexis MICHON, responsable informatique
>>>>>>>> CNRS IBCP, 7 passage du vercors, 69007 LYON, France
>>>>>>>> Mail : alexis.michon at ibcp.fr  Tel : 04.72.72.26.03 - 06.27.56.34.80
>>>>>>>> CNRS IBCP - UMS 5760 - http://www.ibcp.fr/
>>>>>>>> Empreinte : C9:45:2D:7C:79:7F:0B:79:CA:C8:0B:68:41:A2:8C:EE:EA:72:82:34
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Gromacs Users mailing list
>>>>>>>>
>>>>>>>> * Please search the archive at
>>>>>>>> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
>>>>>>>> posting!
>>>>>>>>
>>>>>>>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>>>>>>>>
>>>>>>>> * For (un)subscribe requests visit
>>>>>>>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
>>>>>>>> send a mail to gmx-users-request at gromacs.org.
>>>>>> --
>>>>>> Citation : "It’s not enough to be busy; so are the ants. The question is: what are we busy about?" - Henry David Thoreau
>>>>>> Alexis MICHON, responsable informatique
>>>>>> CNRS IBCP, 7 passage du vercors, 69007 LYON, France
>>>>>> Mail : alexis.michon at ibcp.fr  Tel : 04.72.72.26.03 - 06.27.56.34.80
>>>>>> CNRS IBCP - UMS 5760 - http://www.ibcp.fr/
>>>>>> Empreinte : C9:45:2D:7C:79:7F:0B:79:CA:C8:0B:68:41:A2:8C:EE:EA:72:82:34
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Gromacs Users mailing list
>>>>>>
>>>>>> * Please search the archive at http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting!
>>>>>>
>>>>>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>>>>>>
>>>>>> * For (un)subscribe requests visit
>>>>>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a mail to gmx-users-request at gromacs.org.
>>>> --
>>>> Citation : "It’s not enough to be busy; so are the ants. The question is: what are we busy about?" - Henry David Thoreau
>>>> Alexis MICHON, responsable informatique
>>>> CNRS IBCP, 7 passage du vercors, 69007 LYON, France
>>>> Mail : alexis.michon at ibcp.fr  Tel : 04.72.72.26.03 - 06.27.56.34.80
>>>> CNRS IBCP - UMS 5760 - http://www.ibcp.fr/
>>>> Empreinte : C9:45:2D:7C:79:7F:0B:79:CA:C8:0B:68:41:A2:8C:EE:EA:72:82:34
>>>>
>>>>
>>>>
>>>> --
>>>> Gromacs Users mailing list
>>>>
>>>> * Please search the archive at http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting!
>>>>
>>>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>>>>
>>>> * For (un)subscribe requests visit
>>>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a mail to gmx-users-request at gromacs.org.
>> --
>> Citation : "It’s not enough to be busy; so are the ants. The question is: what are we busy about?" - Henry David Thoreau
>> Alexis MICHON, responsable informatique
>> CNRS IBCP, 7 passage du vercors, 69007 LYON, France
>> Mail : alexis.michon at ibcp.fr  Tel : 04.72.72.26.03 - 06.27.56.34.80
>> CNRS IBCP - UMS 5760 - http://www.ibcp.fr/
>> Empreinte : C9:45:2D:7C:79:7F:0B:79:CA:C8:0B:68:41:A2:8C:EE:EA:72:82:34
>>
>>
>>
>> --
>> Gromacs Users mailing list
>>
>> * Please search the archive at http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting!
>>
>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>>
>> * For (un)subscribe requests visit
>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a mail to gmx-users-request at gromacs.org.

-- 
Citation : "It’s not enough to be busy; so are the ants. The question is: what are we busy about?" - Henry David Thoreau
Alexis MICHON, responsable informatique
CNRS IBCP, 7 passage du vercors, 69007 LYON, France
Mail : alexis.michon at ibcp.fr  Tel : 04.72.72.26.03 - 06.27.56.34.80
CNRS IBCP - UMS 5760 - http://www.ibcp.fr/
Empreinte : C9:45:2D:7C:79:7F:0B:79:CA:C8:0B:68:41:A2:8C:EE:EA:72:82:34

More information about the gromacs.org_gmx-users mailing list