[gmx-users] Fatal Error when launching gromacs 2019.2 on GPU.

[Szilárd Páll]

Hi,

Indeed, the standard way provided by CUDA to expose a subset of GPUs to an
application is the CUDA_VISIBLE_DEVICES (note the "S" ending); I did not
realize that is something you were interested in, I thought you wanted to
avoid using GPUs.

Also note (for anyone interested), a when using a queue system, if node
sharing is allowed, the scheduler should be set up to a correct
CUDA_VISIBLE_DEVICES variable as well.

Cheers,
--
Szilárd


On Mon, Oct 28, 2019 at 1:43 PM Artem Shekhovtsov <job.shekhovtsov at gmail.com>
wrote:

> Hi,
>
> Thanks, setting this variable allowed me to start GROMACS without errors
> using the CPU.
> The problem is that this method prevents me from using other free GPUs on
> the host, but I would like to do this.
> I also found out that setting the CUDA_VISIBLE_DEVICE variable to available
> GPUs at the time of launch allows us to solve this problem.
>
> Artem
>
>
> On Sat, Oct 26, 2019 at 1:50 AM Szilárd Páll <pall.szilard at gmail.com>
> wrote:
>
> > Hi,
> >
> > This is an issue in one of pre-detection checks that trips due to
> > encountering exclusive / prohibited mode devices.
> >
> > You can work around this by entirely disabling the detection using the
> > GMX_DISABLE_GPU_DETECTION environment variable.
> >
> > Cheers,
> > --
> > Szilárd
> >
> >
> > On Thu, Oct 17, 2019 at 5:01 PM Artem Shekhovtsov <
> > job.shekhovtsov at gmail.com>
> > wrote:
> >
> > > Hello!
> > > Problem: The launch of mdrun that does not require video cards exit
> with
> > > fatal error if at least one video card is busy on the host at that
> time.
> > > gmx mdrun -deffnm test -ntmpi 1 -ntomp 1 -nb cpu -bonded cpu
> > > -------------------------------------------------------
> > > Program:     gmx mdrun, version 2019.2
> > > Source file: src/gromacs/gpu_utils/gpu_utils.cu (line 100)
> > >
> > > Fatal error:
> > > cudaFuncGetAttributes failed: all CUDA-capable devices are busy or
> > > unavailable
> > >
> > > For more information and tips for troubleshooting, please check the
> > GROMACS
> > > website at http://www.gromacs.org/Documentation/Errors
> > > -------------------------------------------------------
> > >
> > > I have this error in gromacs version 2019.2, 2019.3, 2020.beta.
> > > Version - 2018.6 is not affected.
> > > All version builds with the same flags.
> > >
> > > Archive with log files and gromacs build files
> > >
> > >
> >
> https://drive.google.com/file/d/1ahn7S69CU5yvAPlLWHryXmMzcGfdWVxP/view?usp=sharing
> > >
> > >
> > > I would appreciate any help.
> > >
> > > Thanks,
> > > Artem Shekhovtsov.
> > > --
> > > Gromacs Users mailing list
> > >
> > > * Please search the archive at
> > > http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> > > posting!
> > >
> > > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> > >
> > > * For (un)subscribe requests visit
> > > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> > > send a mail to gmx-users-request at gromacs.org.
> > >
> > --
> > Gromacs Users mailing list
> >
> > * Please search the archive at
> > http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> > posting!
> >
> > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >
> > * For (un)subscribe requests visit
> > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> > send a mail to gmx-users-request at gromacs.org.
> --
> Gromacs Users mailing list
>
> * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> send a mail to gmx-users-request at gromacs.org.