[gmx-users] WG: WG: Issue with CUDA and gromacs

Szilárd Páll pall.szilard at gmail.com
Mon Mar 25 20:13:04 CET 2019 

Hi,



--
Szilárd


On Mon, Mar 18, 2019 at 2:34 PM Tafelmeier, Stefanie <
Stefanie.Tafelmeier at zae-bayern.de> wrote:

> Hi,
>
> Many thanks again.
>
> Regarding the tests:
> - ntmpi 1 -ntomp 22 -pin on
> >OK, so this suggests that your previously successful 22-thread runs did
> not
> turn on pinning, I assume?
> It seems so, yet it does not run successfully each time. But if done with
> 20-threads, which works usually without error, it does not look like the
> pinning is turned on.
>

Pinning is only turned on if mdrun can safely assume that the cores of the
node are not shared by multiple applications. This assumption can only be
made if all hardware threads of the entire node are used the run itself
(i.e. in your case 2x22 cores with HyperThreadince hence 2 threads each =
88 threads).

-ntmpi 1 -ntomp 1 -pin on; runs
> -ntmpi 1 -ntomp 2 -pin on; runs
>
> - ntmpi 24 -ntomp 1 -pinstride 1 -pin on; runs
> - ntmpi 24 -ntomp 1 -pinstride 2 -pin on; runs
>
> After patch supplied:
> - ntmpi 1 -ntomp 22 -pin on; sometime runs - sometimes doesn't*   ->
> md_run.log at :
> https://it-service.zae-bayern.de/Team/index.php/s/ezXWnQ2pGNeFx6T
>
>  md_norun.log at:
> https://it-service.zae-bayern.de/Team/index.php/s/wYPY7dWEJdwmqJi
> - ntmpi 1 -ntomp 22 -pin off; sometime runs - sometimes doesn't*   (ran
> before)
> - ntmpi 1 -ntomp 23 -pin off; doesn't work*  (ran before)
>
> - ntmpi 1 -ntomp 23 -pinstride 1 -pin on; doesn't work*
>
> - ntmpi 1 -ntomp 23 -pinstride 2 -pin on; doesn't work*  (ran before)
>


The suspicious thing is that the patch I made only improves the verbosity
of the error reporting, it should have no impact on whether the error is
triggered or not. Considering the above behavior it seems that pinning (at
least the patters tried) has no influence on whether the runs work.

Can you please try:
-ntmpi 1 -ntomp 11 -pin on -pinstride 1
-ntmpi 1 -ntomp 11 -pin on -pinstride 2
-ntmpi 1 -ntomp 11 -pin on -pinstride 4
-ntmpi 1 -ntomp 11 -pin on -pinstride 8
and
-ntmpi 1 -ntomp 22 -pin on -pinstride 1
-ntmpi 1 -ntomp 22 -pin on -pinstride 2
-ntmpi 1 -ntomp 22 -pin on -pinstride 4

And please run these 5 times each (-nsteps 0 is fine to make things quick).

Also, please use this patch
https://termbin.com/r8kk
The same way as you did the one before, it adds another check that might
shed some light on what's going on.

- ntmpi 24 -ntomp 1 -pinstride 1 -pin on; runs
> - ntmpi 24 -ntomp 1 -pinstride 2 -pin on; runs
>
> * Fatal error:
> Asynchronous H2D copy failed: invalid argument
>
> When compiling, the make check shows that the regressiontest-complex and
> regressiontest-essential dynamics fail.
> I am not sure if this is correlated?
>

It might be, please share the outputs of the regressiontests.

--
Szilárd


> Many thanks in advance.
> Best wishes,
> Steffi
>
>
>
>
> -----Ursprüngliche Nachricht-----
> Von: gromacs.org_gmx-users-bounces at maillist.sys.kth.se [mailto:
> gromacs.org_gmx-users-bounces at maillist.sys.kth.se] Im Auftrag von Szilárd
> Páll
> Gesendet: Freitag, 15. März 2019 17:57
> An: Discussion list for GROMACS users
> Betreff: Re: [gmx-users] WG: WG: Issue with CUDA and gromacs
>
> On Fri, Mar 15, 2019 at 5:02 PM Tafelmeier, Stefanie <
> Stefanie.Tafelmeier at zae-bayern.de> wrote:
>
> > Hi,
> >
> > about the tests:
> > - ntmpi 1 -ntomp 22 -pin on; doesn't work*
> >
>
> OK, so this suggests that your previously successful 22-thread runs did not
> turn on pinning, I assume?
> Can you please try:
> -ntmpi 1 -ntomp 1 -pin on
> -ntmpi 1 -ntomp 2 -pin on
> that is to check does pinning work at all?
> Also, please try one/both of the above (assuming they fail with) same
> binary, but CPU-only run, i.e.
> -ntmpi 1 -ntomp 1 -pin on -nb cpu
>
>
> > - ntmpi 1 -ntomp 22 -pin off; runs
> > - ntmpi 1 -ntomp 23 -pin off; runs
> > - ntmpi 1 -ntomp 23 -pinstride 1 -pin on; doesn't work*
> > - ntmpi 1 -ntomp 23 -pinstride 2 -pin on; runs
> > - ntmpi 23 -ntomp 1 -pinstride 1 -pin on; doesn't work**
> > - ntmpi 23 -ntomp 1 -pinstride 2 -pin on; doesn't work**
> >
>
> Just to confirm, can you please run the **'s with either -ntmpi 24 (to
> avoid the DD error).
>
>
> >
> > *Error as known.
> >
> > **The number of ranks you selected (23) contains a large prime factor 23.
> > In
> > most cases this will lead to bad performance. Choose a number with
> smaller
> > prime factors or set the decomposition (option -dd) manually.
> >
> > The log file is at:
> > https://it-service.zae-bayern.de/Team/index.php/s/fypKB9iZJz8yXq8
> >
>
> Will have a look and get back with more later.
>
>
> >
> > Many thanks again,
> > Steffi
> >
> > -----Ursprüngliche Nachricht-----
> > Von: gromacs.org_gmx-users-bounces at maillist.sys.kth.se [mailto:
> > gromacs.org_gmx-users-bounces at maillist.sys.kth.se] Im Auftrag von
> Szilárd
> > Páll
> > Gesendet: Freitag, 15. März 2019 16:27
> > An: Discussion list for GROMACS users
> > Betreff: Re: [gmx-users] WG: WG: Issue with CUDA and gromacs
> >
> > Hi,
> >
> > Please share log files with an external service attachments are not
> > accepted on the list.
> >
> > Also, when checking the error with the patch supplied, please run the
> > following cases -- no long runs are needed just want to know which of
> these
> > runs and which of these doesn't:
> > - ntmpi 1 -ntomp 22 -pin on
> > - ntmpi 1 -ntomp 22 -pin off
> > - ntmpi 1 -ntomp 23 -pin off
> > - ntmpi 1 -ntomp 23 -pinstride 1 -pin on
> > - ntmpi 1 -ntomp 23 -pinstride 2 -pin on
> > - ntmpi 23 -ntomp 1 -pinstride 1 -pin on
> > - ntmpi 23 -ntomp 1 -pinstride 2 -pin on
> >
> > Thanks,
> > --
> > Szilárd
> >
> >
> > On Fri, Mar 15, 2019 at 4:04 PM Tafelmeier, Stefanie <
> > Stefanie.Tafelmeier at zae-bayern.de> wrote:
> >
> > > Hi Szilárd,
> > >
> > > thanks for the quick reply.
> > > About the first suggestion, I'll try and give feedback soon.
> > >
> > > Regarding the second, I attached the log-file for the case of
> > > mdrun -v -nt 25
> > > Which ends in the known error message.
> > >
> > > Again, thanks a lot for your information and help.
> > >
> > > Best wishes,
> > > Steffi
> > >
> > >
> > >
> > > -----Ursprüngliche Nachricht-----
> > > Von: gromacs.org_gmx-users-bounces at maillist.sys.kth.se [mailto:
> > > gromacs.org_gmx-users-bounces at maillist.sys.kth.se] Im Auftrag von
> > Szilárd
> > > Páll
> > > Gesendet: Freitag, 15. März 2019 15:30
> > > An: Discussion list for GROMACS users
> > > Betreff: Re: [gmx-users] WG: WG: Issue with CUDA and gromacs
> > >
> > > Hi Stefanie,
> > >
> > > Unless and until the error and performance-related concerns prove to be
> > > related, let's keep those separate.
> > >
> > > I'd first focus on the former. To be honest, I've never encountered
> such
> > an
> > > issue where if you use more than a certain number of threads, the run
> > > aborts with that error. To investigate further can you please apply the
> > > following patch file which hopefully give more context to the error:
> > > https://termbin.com/uhgp
> > > (e.g. you can execute the following to accomplish that:
> > > curl https://termbin.com/uhgp > devicebuffer.cuh.patch && patch -p0 <
> > > devicebuffer.cuh.patch)
> > >
> > > Regarding the performance-related questions, can you please share a
> full
> > > log file of the runs so we can see the machine config, simulation
> > > system/settings, etc. Without that it is hard to judge what's best for
> > your
> > > case. However, if you only have a single GPU (which seems to be the
> case
> > > based on the log excerpts) along those two rather beefy CPUs, than you
> > will
> > > likely not get much benefit from using all cores and it is normal that
> > you
> > > see little to no improvement from using cores of a second CPU socket.
> > >
> > > Cheers,
> > > --
> > > Szilárd
> > >
> > >
> > > On Thu, Mar 14, 2019 at 12:47 PM Tafelmeier, Stefanie <
> > > Stefanie.Tafelmeier at zae-bayern.de> wrote:
> > >
> > > > Dear all,
> > > >
> > > > I was not sure if the email before reached you, but again many thanks
> > for
> > > > your reply Szilárd.
> > > >
> > > > As written below we are still facing a problem with the performance
> of
> > > > your workstation.
> > > > I wrote before because of the error message when keeping occurring
> for
> > > > mdrun simulation:
> > > >
> > > > Assertion failed:
> > > > Condition: stat == cudaSuccess
> > > > Asynchronous H2D copy failed
> > > >
> > > > As I mentioned all Versions to install (Gormacs, Cuda, nvcc, gcc) are
> > the
> > > > newest once now.
> > > >
> > > > If I run mdrun without further settings it will lead to this error
> > > > message. If I run it and choose the thread amount directly the mdrun
> is
> > > > performing well. But only for –nt numbers between 1 – 22. Higher ones
> > > again
> > > > lead to the before mentioned error message.
> > > >
> > > > In order to investigate in more detail, I tried different versions
> for
> > > > –nt, –ntmpi – ntomp also combined with –npme:
> > > > -       The best performance in the sense of ns/day is with –nt 22
> > > > respectively –ntomp 22 alone. But then only 22 threads are involved.
> > > Which
> > > > is fine if I run more than one mdrun simultaneously, as I can
> > distribute
> > > > the other 66 threads. The GPU usage is then around 65%.
> > > > -       A similar good performance is reached with mdrun  -ntmpi 4
> > -ntomp
> > > > 18 -npme 1 -pme gpu -nb gpu. But then 44 threads are involved. The
> GPU
> > > > usage is then around 50%.
> > > >
> > > > I read the information on
> > > >
> > >
> >
> http://manual.gromacs.org/documentation/5.1/user-guide/mdrun-performance.html
> > > > which was very helpful, bur some things are still not clear now to
> me:
> > > > I was wondering if there is any other enhancement of the performance?
> > Or
> > > > what is the reason, that –nt maximum is at 22 threads? Could this be
> > > > connected to the sockets (see details below) of your workstation?
> > > > It is not clear to me how a number of thread (-nt) higher 22 can lead
> > to
> > > > the error regarding the Asynchronous H2D copy)
> > > >
> > > > Please excuse all these questions. I would appreciate a lot  if you
> > might
> > > > have a hint for this problem as well.
> > > >
> > > > Best regards,
> > > > Steffi
> > > >
> > > > -----
> > > >
> > > > The workstation details are:
> > > > Running on 1 node with total 44 cores, 88 logical cores, 1 compatible
> > GPU
> > > > Hardware detected:
> > > >
> > > >   CPU info:
> > > >     Vendor: Intel
> > > >     Brand:  Intel(R) Xeon(R) Gold 6152 CPU @ 2.10GHz
> > > >     Family: 6   Model: 85   Stepping: 4
> > > >     Features: aes apic avx avx2 avx512f avx512cd avx512bw avx512vl
> > clfsh
> > > > cmov cx8 cx16 f16c fma hle htt intel lahf mmx msr nonstop_tsc pcid
> > > pclmuldq
> > > > pdcm pdpe1gb popcnt pse rdrnd rdtscp rtm sse2 sse3 sse4.1 sse4.2
> ssse3
> > > tdt
> > > > x2apic
> > > >
> > > >     Number of AVX-512 FMA units: 2
> > > >   Hardware topology: Basic
> > > >     Sockets, cores, and logical processors:
> > > >       Socket  0: [   0  44] [   1  45] [   2  46] [   3  47] [   4
> > 48] [
> > > >  5  49] [   6  50] [   7  51] [   8  52] [   9  53] [  10  54] [  11
> > 55]
> > > > [  12  56] [  13  57] [  14  58] [  15  59] [  16  60] [  17  61] [
> 18
> > > > 62] [  19  63] [  20  64] [  21  65]
> > > >       Socket  1: [  22  66] [  23  67] [  24  68] [  25  69] [  26
> > 70] [
> > > > 27  71] [  28  72] [  29  73] [  30  74] [  31  75] [  32  76] [  33
> > 77]
> > > > [  34  78] [  35  79] [  36  80] [  37  81] [  38  82] [  39  83] [
> 40
> > > > 84] [  41  85] [  42  86] [  43  87]
> > > >   GPU info:
> > > >     Number of GPUs detected: 1
> > > >     #0: NVIDIA Quadro P6000, compute cap.: 6.1, ECC:  no, stat:
> > > compatible
> > > >
> > > > -----
> > > >
> > > >
> > > >
> > > > -----Ursprüngliche Nachricht-----
> > > > Von: gromacs.org_gmx-users-bounces at maillist.sys.kth.se [mailto:
> > > > gromacs.org_gmx-users-bounces at maillist.sys.kth.se] Im Auftrag von
> > > Szilárd
> > > > Páll
> > > > Gesendet: Donnerstag, 31. Januar 2019 17:15
> > > > An: Discussion list for GROMACS users
> > > > Betreff: Re: [gmx-users] WG: Issue with CUDA and gromacs
> > > >
> > > > On Thu, Jan 31, 2019 at 2:14 PM Szilárd Páll <pall.szilard at gmail.com
> >
> > > > wrote:
> > > > >
> > > > > On Wed, Jan 30, 2019 at 5:15 PM Tafelmeier, Stefanie
> > > > > <Stefanie.Tafelmeier at zae-bayern.de> wrote:
> > > > > >
> > > > > > Dear all,
> > > > > >
> > > > > > We are facing an issue with the CUDA toolkit.
> > > > > > We tried several combinations of gromacs versions and CUDA
> > Toolkits.
> > > > No Toolkit older than 9.2 was possible to try as there are no driver
> > for
> > > > nvidia available for a Quadro P6000.
> > > > > > Gromacs
> > > > >
> > > > > Install the latest 410.xx drivers and it will work; the NVIDIA
> driver
> > > > > download website (https://www.nvidia.com/Download/index.aspx)
> > > > > recommends 410.93.
> > > > >
> > > > > Here's a system with CUDA 10-compatible driver running o a system
> > with
> > > > > a P6000: https://termbin.com/ofzo
> > > >
> > > > Sorry, I misread that as "CUDA >=9.2 was not possible".
> > > >
> > > > Note that the driver is backward compatible, so you can use a new
> > > > driver with older CUDA versions.
> > > >
> > > > Also note that the oldest driver NVIDIA claims to have P6000 support
> > > > is 390.59 which is, as far as I know, one gen older than the 396 that
> > > > the CUDA 9.2 toolkit came with. This is however, not something I'd
> > > > recommend pursuing, use a new driver from the official site with any
> > > > CUDA version that GROMACS supports and it should be fine.
> > > >
> > > > >
> > > > > > CUDA
> > > > > >
> > > > > > Error message
> > > > > >
> > > > > > 2019
> > > > > >
> > > > > > 10.0
> > > > > >
> > > > > > gmx mdrun:
> > > > > > Assertion failed:
> > > > > > Condition: stat == cudaSuccess
> > > > > > Asynchronous H2D copy failed
> > > > > >
> > > > > > 2019
> > > > > >
> > > > > > 9.2
> > > > > >
> > > > > > gmx mdrun:
> > > > > > Assertion failed:
> > > > > > Condition: stat == cudaSuccess
> > > > > > Asynchronous H2D copy failed
> > > > > >
> > > > > > 2018.5
> > > > > >
> > > > > > 9.2
> > > > > >
> > > > > > gmx mdrun: Fatal error:
> > > > > > HtoD cudaMemcpyAsync failed: invalid argument
> > > > >
> > > > > Can we get some more details on these, please? complete log files
> > > > > would be a good start.
> > > > >
> > > > > > 5.1.5
> > > > > >
> > > > > > 9.2
> > > > > >
> > > > > > Installation make: nvcc fatal   : Unsupported gpu architecture
> > > > 'compute_20'*
> > > > > >
> > > > > > 2016.2
> > > > > >
> > > > > > 9.2
> > > > > >
> > > > > > Installation make: nvcc fatal   : Unsupported gpu architecture
> > > > 'compute_20'*
> > > > > >
> > > > > >
> > > > > > *We also tried to set the target CUDA architectures as described
> in
> > > > the installation guide (
> > > > manual.gromacs.org/documentation/2019/install-guide/index.html).
> > > > Unfortunately it didn't work.
> > > > >
> > > > > What does it mean that it didn't work? Can you share the command
> you
> > > > > used and what exactly did not work?
> > > > >
> > > > > For the P6000 which is a "compute capability 6.1" device (for
> anyone
> > > > > who needs to look it up, go here:
> > > > > https://developer.nvidia.com/cuda-gpus), you should set
> > > > > cmake ../ -DGMX_CUDA_TARGET_SM="61"
> > > > >
> > > > > --
> > > > > Szilárd
> > > > >
> > > > > > Performing simulations on CPU only always works, yet of cause are
> > > more
> > > > slowly than they could be with additionally using the GPU.
> > > > > > The issue #2761 (https://redmine.gromacs.org/issues/2762) seems
> > > > similar to our problem.
> > > > > > Even though this issue is still open, we wanted to ask if you can
> > > give
> > > > us any information about how to solve this problem?
> > > > > >
> > > > > > Many thanks in advance.
> > > > > > Best regards,
> > > > > > Stefanie Tafelmeier
> > > > > >
> > > > > >
> > > > > > Further details if necessary:
> > > > > > The workstation:
> > > > > > 2 x Xeon Gold 6152 @ 3,7Ghz (22 K, 44Th, AVX512)
> > > > > > Nvidia Quadro P6000 with 3840 Cuda-Cores
> > > > > >
> > > > > > The simulations system:
> > > > > > Long chain alkanes (previously used with gromacs 5.1.5 and CUDA
> > 7.5 -
> > > > worked perfectly)
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > ZAE Bayern
> > > > > > Stefanie Tafelmeier
> > > > > > Bereich Energiespeicherung/Division Energy Storage
> > > > > > Thermische Energiespeicher/Thermal Energy Storage
> > > > > > Walther-Meißner-Str. 6
> > > > > > 85748 Garching
> > > > > >
> > > > > > Tel.: +49 89 329442-75
> > > > > > Fax: +49 89 329442-12
> > > > > > Stefanie.tafelmeier at zae-bayern.de<mailto:
> > > > Stefanie.tafelmeier at zae-bayern.de>
> > > > > > http://www.zae-bayern.de<http://www.zae-bayern.de/>
> > > > > >
> > > > > >
> > > > > > ZAE Bayern - Bayerisches Zentrum für Angewandte Energieforschung
> e.
> > > V.
> > > > > > Vorstand/Board:
> > > > > > Prof. Dr. Hartmut Spliethoff (Vorsitzender/Chairman),
> > > > > > Prof. Dr. Vladimir Dyakonov
> > > > > > Sitz/Registered Office: Würzburg
> > > > > > Registergericht/Register Court: Amtsgericht Würzburg
> > > > > > Registernummer/Register Number: VR 1386
> > > > > >
> > > > > > Sämtliche Willenserklärungen, z. B. Angebote, Aufträge, Anträge
> und
> > > > Verträge, sind für das ZAE Bayern nur in schriftlicher und
> > ordnungsgemäß
> > > > unterschriebener Form rechtsverbindlich. Diese E-Mail ist
> > ausschließlich
> > > > zur Nutzung durch den/die vorgenannten Empfänger bestimmt. Jegliche
> > > > unbefugte Offenbarung, Nutzung oder Verbreitung, sei es insgesamt
> oder
> > > > teilweise, ist untersagt. Sollten Sie diese E-Mail irrtümlich
> erhalten
> > > > haben, benachrichtigen Sie bitte unverzüglich den Absender und
> löschen
> > > Sie
> > > > diese E-Mail.
> > > > > >
> > > > > > Any declarations of intent, such as quotations, orders,
> > applications
> > > > and contracts, are legally binding for ZAE Bayern only if expressed
> in
> > a
> > > > written and duly signed form. This e-mail is intended solely for use
> by
> > > the
> > > > recipient(s) named above. Any unauthorised disclosure, use or
> > > > dissemination, whether in whole or in part, is prohibited. If you
> have
> > > > received this e-mail in error, please notify the sender immediately
> and
> > > > delete this e-mail.
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Gromacs Users mailing list
> > > > > >
> > > > > > * Please search the archive at
> > > > http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> > > > posting!
> > > > > >
> > > > > > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> > > > > >
> > > > > > * For (un)subscribe requests visit
> > > > > >
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users
> > > or
> > > > send a mail to gmx-users-request at gromacs.org.
> > > > --
> > > > Gromacs Users mailing list
> > > >
> > > > * Please search the archive at
> > > > http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> > > > posting!
> > > >
> > > > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> > > >
> > > > * For (un)subscribe requests visit
> > > > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users
> or
> > > > send a mail to gmx-users-request at gromacs.org.
> > > > --
> > > > Gromacs Users mailing list
> > > >
> > > > * Please search the archive at
> > > > http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> > > > posting!
> > > >
> > > > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> > > >
> > > > * For (un)subscribe requests visit
> > > > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users
> or
> > > > send a mail to gmx-users-request at gromacs.org.
> > > --
> > > Gromacs Users mailing list
> > >
> > > * Please search the archive at
> > > http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> > > posting!
> > >
> > > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> > >
> > > * For (un)subscribe requests visit
> > > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> > > send a mail to gmx-users-request at gromacs.org.
> > > --
> > > Gromacs Users mailing list
> > >
> > > * Please search the archive at
> > > http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> > > posting!
> > >
> > > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> > >
> > > * For (un)subscribe requests visit
> > > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> > > send a mail to gmx-users-request at gromacs.org.
> > --
> > Gromacs Users mailing list
> >
> > * Please search the archive at
> > http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> > posting!
> >
> > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >
> > * For (un)subscribe requests visit
> > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> > send a mail to gmx-users-request at gromacs.org.
> > --
> > Gromacs Users mailing list
> >
> > * Please search the archive at
> > http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> > posting!
> >
> > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >
> > * For (un)subscribe requests visit
> > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> > send a mail to gmx-users-request at gromacs.org.
> --
> Gromacs Users mailing list
>
> * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> send a mail to gmx-users-request at gromacs.org.
> --
> Gromacs Users mailing list
>
> * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> send a mail to gmx-users-request at gromacs.org.

More information about the gromacs.org_gmx-users mailing list