[gmx-developers] cudaStreamSynchronize failed in cu_blockwait_nb

[Szilárd Páll]

Hi,

That sounds like a nasty bug that I have not seen for quite a while. This
happens generally when some serious memory corruption puts the GPU in a
"bad state". For the future, you could try to reset the GPU by reloading
the driver, but if that does not help you will have to reboot.

I was able to reproduce the bug and in fact on our development machine the
NVIDIA driver seems to get into a messed up state in which mdrun will hang,
no matter whether I launch in on the GTX 580 or 680. Reloading the driver
seems to fix this issue.

Thanks for the report, I'll looking into this bug and will give you an
update!

Cheers,
--
Szilárd


On Fri, Oct 19, 2012 at 12:01 PM, Carsten Kutzner <ckutzne at gwdg.de> wrote:

> Hi,
>
> we updated to the newest driver, but later I found that this crash is
> caused by
> a .tpr file with Coulomb-type=cutoff instead of PME:
>
> - I start with a PME .tpr file that runs with the recent 4.6 on both a
> GTX580 and 680,
>   and even using both
> - I change to cutoff setting (no other changes!); this tpr still runs on
> the 580,
>   but on the 680 produces the fatal error:
>   "cudaStreamSynchronize failed in cu_blockwait_nb: unspecified launch
> failure"
>   Moreover, after that any other mdrun using any GPU on that node will
> read in the
>   previously working, PME) .tpr file and then hang. After rebooting, I can
> again
>   run the PME .tpr file.
>
> Carsten
>
>
>
> On Oct 17, 2012, at 3:10 PM, Szilárd Páll <szilard.pall at cbr.su.se> wrote:
>
> > HI,
> >
> > Your driver might be simply too old for a GTX680. You'll need at least a
> very late 295.xx driver and preferably the 304.54 (or later).
> >
> > Cheers,
> > --
> > Szilárd
> >
> >
> > On Wed, Oct 17, 2012 at 2:10 PM, Carsten Kutzner <ckutzne at gwdg.de>
> wrote:
> > BTW this executable works on a GTX580, but shows the fatal error
> > on a GTX680 - both mounted in the same workstation.
> >
> > Carsten
> >
> >
> > On Oct 17, 2012, at 12:05 PM, Carsten Kutzner <ckutzne at gwdg.de> wrote:
> >
> > > Hi,
> > >
> > > what am I doing wrong if I get this error code:
> > >
> > > -------------------------------------------------------
> > > Program mdrun_threads, VERSION 4.6-dev-20121016-4af4561
> > > Source code file:
> /home/ckutzne/installations/git-gromacs-4-6-department/src/mdlib/nbnxn_cuda/
> nbnxn_cuda.cu, line: 558
> > >
> > > Fatal error:
> > > cudaStreamSynchronize failed in cu_blockwait_nb: unspecified launch
> failure
> > >
> > > For more information and tips for troubleshooting, please check the
> GROMACS
> > > website at http://www.gromacs.org/Documentation/Errors
> > > -------------------------------------------------------
> > >
> > > Thanks,
> > > Carsten
> > > --
> > > gmx-developers mailing list
> > > gmx-developers at gromacs.org
> > > http://lists.gromacs.org/mailman/listinfo/gmx-developers
> > > Please don't post (un)subscribe requests to the list. Use the www
> interface or send it to gmx-developers-request at gromacs.org.
> >
> >
> > --
> > Dr. Carsten Kutzner
> > Max Planck Institute for Biophysical Chemistry
> > Theoretical and Computational Biophysics
> > Am Fassberg 11, 37077 Goettingen, Germany
> > Tel. +49-551-2012313, Fax: +49-551-2012302
> > http://www.mpibpc.mpg.de/grubmueller/kutzner
> >
> > --
> > gmx-developers mailing list
> > gmx-developers at gromacs.org
> > http://lists.gromacs.org/mailman/listinfo/gmx-developers
> > Please don't post (un)subscribe requests to the list. Use the
> > www interface or send it to gmx-developers-request at gromacs.org.
> >
> > --
> > gmx-developers mailing list
> > gmx-developers at gromacs.org
> > http://lists.gromacs.org/mailman/listinfo/gmx-developers
> > Please don't post (un)subscribe requests to the list. Use the
> > www interface or send it to gmx-developers-request at gromacs.org.
>
>
> --
> Dr. Carsten Kutzner
> Max Planck Institute for Biophysical Chemistry
> Theoretical and Computational Biophysics
> Am Fassberg 11, 37077 Goettingen, Germany
> Tel. +49-551-2012313, Fax: +49-551-2012302
> http://www.mpibpc.mpg.de/grubmueller/kutzner
>
> --
> gmx-developers mailing list
> gmx-developers at gromacs.org
> http://lists.gromacs.org/mailman/listinfo/gmx-developers
> Please don't post (un)subscribe requests to the list. Use the
> www interface or send it to gmx-developers-request at gromacs.org.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://maillist.sys.kth.se/pipermail/gromacs.org_gmx-developers/attachments/20121019/22b272e1/attachment.html>