[gmx-developers] cudaStreamSynchronize failed in cu_blockwait_nb

[Szilárd Páll]

Hi,

The CUDA plain cut-off kernel's pointer was incorrectly assigned (stupid
copy-paste bug). Just pushed a bugfix: https://gerrit.gromacs.org/#/c/1553/

Cheers,
--
Szilárd


On Fri, Oct 19, 2012 at 3:20 PM, Szilárd Páll <szilard.pall at cbr.su.se>wrote:

> Hi,
>
> That sounds like a nasty bug that I have not seen for quite a while. This
> happens generally when some serious memory corruption puts the GPU in a
> "bad state". For the future, you could try to reset the GPU by reloading
> the driver, but if that does not help you will have to reboot.
>
> I was able to reproduce the bug and in fact on our development machine the
> NVIDIA driver seems to get into a messed up state in which mdrun will hang,
> no matter whether I launch in on the GTX 580 or 680. Reloading the driver
> seems to fix this issue.
>
> Thanks for the report, I'll looking into this bug and will give you an
> update!
>
> Cheers,
> --
> Szilárd
>
>
>
> On Fri, Oct 19, 2012 at 12:01 PM, Carsten Kutzner <ckutzne at gwdg.de> wrote:
>
>> Hi,
>>
>> we updated to the newest driver, but later I found that this crash is
>> caused by
>> a .tpr file with Coulomb-type=cutoff instead of PME:
>>
>> - I start with a PME .tpr file that runs with the recent 4.6 on both a
>> GTX580 and 680,
>>   and even using both
>> - I change to cutoff setting (no other changes!); this tpr still runs on
>> the 580,
>>   but on the 680 produces the fatal error:
>>   "cudaStreamSynchronize failed in cu_blockwait_nb: unspecified launch
>> failure"
>>   Moreover, after that any other mdrun using any GPU on that node will
>> read in the
>>   previously working, PME) .tpr file and then hang. After rebooting, I
>> can again
>>   run the PME .tpr file.
>>
>> Carsten
>>
>>
>>
>> On Oct 17, 2012, at 3:10 PM, Szilárd Páll <szilard.pall at cbr.su.se> wrote:
>>
>> > HI,
>> >
>> > Your driver might be simply too old for a GTX680. You'll need at least
>> a very late 295.xx driver and preferably the 304.54 (or later).
>> >
>> > Cheers,
>> > --
>> > Szilárd
>> >
>> >
>> > On Wed, Oct 17, 2012 at 2:10 PM, Carsten Kutzner <ckutzne at gwdg.de>
>> wrote:
>> > BTW this executable works on a GTX580, but shows the fatal error
>> > on a GTX680 - both mounted in the same workstation.
>> >
>> > Carsten
>> >
>> >
>> > On Oct 17, 2012, at 12:05 PM, Carsten Kutzner <ckutzne at gwdg.de> wrote:
>> >
>> > > Hi,
>> > >
>> > > what am I doing wrong if I get this error code:
>> > >
>> > > -------------------------------------------------------
>> > > Program mdrun_threads, VERSION 4.6-dev-20121016-4af4561
>> > > Source code file:
>> /home/ckutzne/installations/git-gromacs-4-6-department/src/mdlib/nbnxn_cuda/
>> nbnxn_cuda.cu, line: 558
>> > >
>> > > Fatal error:
>> > > cudaStreamSynchronize failed in cu_blockwait_nb: unspecified launch
>> failure
>> > >
>> > > For more information and tips for troubleshooting, please check the
>> GROMACS
>> > > website at http://www.gromacs.org/Documentation/Errors
>> > > -------------------------------------------------------
>> > >
>> > > Thanks,
>> > > Carsten
>> > > --
>> > > gmx-developers mailing list
>> > > gmx-developers at gromacs.org
>> > > http://lists.gromacs.org/mailman/listinfo/gmx-developers
>> > > Please don't post (un)subscribe requests to the list. Use the www
>> interface or send it to gmx-developers-request at gromacs.org.
>> >
>> >
>> > --
>> > Dr. Carsten Kutzner
>> > Max Planck Institute for Biophysical Chemistry
>> > Theoretical and Computational Biophysics
>> > Am Fassberg 11, 37077 Goettingen, Germany
>> > Tel. +49-551-2012313, Fax: +49-551-2012302
>> > http://www.mpibpc.mpg.de/grubmueller/kutzner
>> >
>> > --
>> > gmx-developers mailing list
>> > gmx-developers at gromacs.org
>> > http://lists.gromacs.org/mailman/listinfo/gmx-developers
>> > Please don't post (un)subscribe requests to the list. Use the
>> > www interface or send it to gmx-developers-request at gromacs.org.
>> >
>> > --
>> > gmx-developers mailing list
>> > gmx-developers at gromacs.org
>> > http://lists.gromacs.org/mailman/listinfo/gmx-developers
>> > Please don't post (un)subscribe requests to the list. Use the
>> > www interface or send it to gmx-developers-request at gromacs.org.
>>
>>
>> --
>> Dr. Carsten Kutzner
>> Max Planck Institute for Biophysical Chemistry
>> Theoretical and Computational Biophysics
>> Am Fassberg 11, 37077 Goettingen, Germany
>> Tel. +49-551-2012313, Fax: +49-551-2012302
>> http://www.mpibpc.mpg.de/grubmueller/kutzner
>>
>> --
>> gmx-developers mailing list
>> gmx-developers at gromacs.org
>> http://lists.gromacs.org/mailman/listinfo/gmx-developers
>> Please don't post (un)subscribe requests to the list. Use the
>> www interface or send it to gmx-developers-request at gromacs.org.
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://maillist.sys.kth.se/pipermail/gromacs.org_gmx-developers/attachments/20121022/613cd093/attachment.html>