[gmx-users] WG: WG: Issue with CUDA and gromacs
Szilárd Páll
pall.szilard at gmail.com
Tue Mar 26 16:57:32 CET 2019
Hi Steffi,
Thanks for running the tests; yes, the patch file was meant to be applied
to the unchanged GROMACS 2019 code.
Please also share the log files from thr failed runs, not just the
copy-paste of the fatal error -- as a result of the additional check there
might have been a note printed which I was after.
Regarding the regression tests, what I would like to have is the actual
directories of the tests that failed, i.e. as your log indicates a few of
the complex tests at least.
Cheers,
--
Szilárd
On Tue, Mar 26, 2019 at 1:44 PM Tafelmeier, Stefanie <
Stefanie.Tafelmeier at zae-bayern.de> wrote:
> Hi Szilárd,
>
> thanks again for your answer.
> Regarding the tests:
> without the new patch:
>
> -ntmpi 1 -ntomp 11 -pin on -pinstride 1: all ran
> -ntmpi 1 -ntomp 11 -pin on -pinstride 2: all ran
> -ntmpi 1 -ntomp 11 -pin on -pinstride 4: all ran
> -ntmpi 1 -ntomp 11 -pin on -pinstride 8: all ran
> and
> -ntmpi 1 -ntomp 22 -pin on -pinstride 1: none ran*
> -ntmpi 1 -ntomp 22 -pin on -pinstride 2: none ran*
> -ntmpi 1 -ntomp 22 -pin on -pinstride 4: one out of 5 ran
>
>
> With the new patch (devicebuffer.cuh had to be the original, right? The
> already patched didn't work as the lines didn't fit, as far as I
> understood.):
>
> -ntmpi 1 -ntomp 11 -pin on -pinstride 1: all ran
> -ntmpi 1 -ntomp 11 -pin on -pinstride 2: all ran
> -ntmpi 1 -ntomp 11 -pin on -pinstride 4: all ran
> -ntmpi 1 -ntomp 11 -pin on -pinstride 8: all ran
> and
> -ntmpi 1 -ntomp 22 -pin on -pinstride 1: none ran*
> -ntmpi 1 -ntomp 22 -pin on -pinstride 2: none ran*
> -ntmpi 1 -ntomp 22 -pin on -pinstride 4: none ran*
>
> * Fatal error:
> Asynchronous H2D copy failed: invalid argument
>
>
> Regarding the regressiontest:
> The LastTest.log is available here:
> https://it-service.zae-bayern.de/Team/index.php/s/3sdki7Cf2x2CEQi
> this was not given in the log:
> The following tests FAILED:
> 42 - regressiontests/complex (Timeout)
> 46 - regressiontests/essentialdynamics (Failed)
> Errors while running CTest
> CMakeFiles/run-ctest-nophys.dir/build.make:57: recipe for target
> 'CMakeFiles/run-ctest-nophys' failed
> make[3]: *** [CMakeFiles/run-ctest-nophys] Error 8
> CMakeFiles/Makefile2:1397: recipe for target
> 'CMakeFiles/run-ctest-nophys.dir/all'failed
> make[2]: *** [CMakeFiles/run-ctest-nophys.dir/all] Error 2
> CMakeFiles/Makefile2:1177: recipe for target
> 'CMakeFiles/check.dir/rule' failed
> make[1]: *** [CMakeFiles/check.dir/rule] Error 2
> Makefile:626: recipe for target 'check' failed
> make: *** [check] Error 2
>
> Many thanks again.
> Best wishes,
> Steffi
>
>
>
>
>
> -----Ursprüngliche Nachricht-----
> Von: gromacs.org_gmx-users-bounces at maillist.sys.kth.se [mailto:
> gromacs.org_gmx-users-bounces at maillist.sys.kth.se] Im Auftrag von Szilárd
> Páll
> Gesendet: Montag, 25. März 2019 20:13
> An: Discussion list for GROMACS users
> Betreff: Re: [gmx-users] WG: WG: Issue with CUDA and gromacs
>
> Hi,
>
>
>
> --
> Szilárd
>
>
> On Mon, Mar 18, 2019 at 2:34 PM Tafelmeier, Stefanie <
> Stefanie.Tafelmeier at zae-bayern.de> wrote:
>
> > Hi,
> >
> > Many thanks again.
> >
> > Regarding the tests:
> > - ntmpi 1 -ntomp 22 -pin on
> > >OK, so this suggests that your previously successful 22-thread runs did
> > not
> > turn on pinning, I assume?
> > It seems so, yet it does not run successfully each time. But if done with
> > 20-threads, which works usually without error, it does not look like the
> > pinning is turned on.
> >
>
> Pinning is only turned on if mdrun can safely assume that the cores of the
> node are not shared by multiple applications. This assumption can only be
> made if all hardware threads of the entire node are used the run itself
> (i.e. in your case 2x22 cores with HyperThreadince hence 2 threads each =
> 88 threads).
>
> -ntmpi 1 -ntomp 1 -pin on; runs
> > -ntmpi 1 -ntomp 2 -pin on; runs
> >
> > - ntmpi 24 -ntomp 1 -pinstride 1 -pin on; runs
> > - ntmpi 24 -ntomp 1 -pinstride 2 -pin on; runs
> >
> > After patch supplied:
> > - ntmpi 1 -ntomp 22 -pin on; sometime runs - sometimes doesn't* ->
> > md_run.log at :
> > https://it-service.zae-bayern.de/Team/index.php/s/ezXWnQ2pGNeFx6T
> >
> > md_norun.log at:
> > https://it-service.zae-bayern.de/Team/index.php/s/wYPY7dWEJdwmqJi
> > - ntmpi 1 -ntomp 22 -pin off; sometime runs - sometimes doesn't* (ran
> > before)
> > - ntmpi 1 -ntomp 23 -pin off; doesn't work* (ran before)
> >
> > - ntmpi 1 -ntomp 23 -pinstride 1 -pin on; doesn't work*
> >
> > - ntmpi 1 -ntomp 23 -pinstride 2 -pin on; doesn't work* (ran before)
> >
>
>
> The suspicious thing is that the patch I made only improves the verbosity
> of the error reporting, it should have no impact on whether the error is
> triggered or not. Considering the above behavior it seems that pinning (at
> least the patters tried) has no influence on whether the runs work.
>
> Can you please try:
> -ntmpi 1 -ntomp 11 -pin on -pinstride 1
> -ntmpi 1 -ntomp 11 -pin on -pinstride 2
> -ntmpi 1 -ntomp 11 -pin on -pinstride 4
> -ntmpi 1 -ntomp 11 -pin on -pinstride 8
> and
> -ntmpi 1 -ntomp 22 -pin on -pinstride 1
> -ntmpi 1 -ntomp 22 -pin on -pinstride 2
> -ntmpi 1 -ntomp 22 -pin on -pinstride 4
>
> And please run these 5 times each (-nsteps 0 is fine to make things quick).
>
> Also, please use this patch
> https://termbin.com/r8kk
> The same way as you did the one before, it adds another check that might
> shed some light on what's going on.
>
> - ntmpi 24 -ntomp 1 -pinstride 1 -pin on; runs
> > - ntmpi 24 -ntomp 1 -pinstride 2 -pin on; runs
> >
> > * Fatal error:
> > Asynchronous H2D copy failed: invalid argument
> >
> > When compiling, the make check shows that the regressiontest-complex and
> > regressiontest-essential dynamics fail.
> > I am not sure if this is correlated?
> >
>
> It might be, please share the outputs of the regressiontests.
>
> --
> Szilárd
>
>
> > Many thanks in advance.
> > Best wishes,
> > Steffi
> >
> >
> >
> >
> > -----Ursprüngliche Nachricht-----
> > Von: gromacs.org_gmx-users-bounces at maillist.sys.kth.se [mailto:
> > gromacs.org_gmx-users-bounces at maillist.sys.kth.se] Im Auftrag von
> Szilárd
> > Páll
> > Gesendet: Freitag, 15. März 2019 17:57
> > An: Discussion list for GROMACS users
> > Betreff: Re: [gmx-users] WG: WG: Issue with CUDA and gromacs
> >
> > On Fri, Mar 15, 2019 at 5:02 PM Tafelmeier, Stefanie <
> > Stefanie.Tafelmeier at zae-bayern.de> wrote:
> >
> > > Hi,
> > >
> > > about the tests:
> > > - ntmpi 1 -ntomp 22 -pin on; doesn't work*
> > >
> >
> > OK, so this suggests that your previously successful 22-thread runs did
> not
> > turn on pinning, I assume?
> > Can you please try:
> > -ntmpi 1 -ntomp 1 -pin on
> > -ntmpi 1 -ntomp 2 -pin on
> > that is to check does pinning work at all?
> > Also, please try one/both of the above (assuming they fail with) same
> > binary, but CPU-only run, i.e.
> > -ntmpi 1 -ntomp 1 -pin on -nb cpu
> >
> >
> > > - ntmpi 1 -ntomp 22 -pin off; runs
> > > - ntmpi 1 -ntomp 23 -pin off; runs
> > > - ntmpi 1 -ntomp 23 -pinstride 1 -pin on; doesn't work*
> > > - ntmpi 1 -ntomp 23 -pinstride 2 -pin on; runs
> > > - ntmpi 23 -ntomp 1 -pinstride 1 -pin on; doesn't work**
> > > - ntmpi 23 -ntomp 1 -pinstride 2 -pin on; doesn't work**
> > >
> >
> > Just to confirm, can you please run the **'s with either -ntmpi 24 (to
> > avoid the DD error).
> >
> >
> > >
> > > *Error as known.
> > >
> > > **The number of ranks you selected (23) contains a large prime factor
> 23.
> > > In
> > > most cases this will lead to bad performance. Choose a number with
> > smaller
> > > prime factors or set the decomposition (option -dd) manually.
> > >
> > > The log file is at:
> > > https://it-service.zae-bayern.de/Team/index.php/s/fypKB9iZJz8yXq8
> > >
> >
> > Will have a look and get back with more later.
> >
> >
> > >
> > > Many thanks again,
> > > Steffi
> > >
> > > -----Ursprüngliche Nachricht-----
> > > Von: gromacs.org_gmx-users-bounces at maillist.sys.kth.se [mailto:
> > > gromacs.org_gmx-users-bounces at maillist.sys.kth.se] Im Auftrag von
> > Szilárd
> > > Páll
> > > Gesendet: Freitag, 15. März 2019 16:27
> > > An: Discussion list for GROMACS users
> > > Betreff: Re: [gmx-users] WG: WG: Issue with CUDA and gromacs
> > >
> > > Hi,
> > >
> > > Please share log files with an external service attachments are not
> > > accepted on the list.
> > >
> > > Also, when checking the error with the patch supplied, please run the
> > > following cases -- no long runs are needed just want to know which of
> > these
> > > runs and which of these doesn't:
> > > - ntmpi 1 -ntomp 22 -pin on
> > > - ntmpi 1 -ntomp 22 -pin off
> > > - ntmpi 1 -ntomp 23 -pin off
> > > - ntmpi 1 -ntomp 23 -pinstride 1 -pin on
> > > - ntmpi 1 -ntomp 23 -pinstride 2 -pin on
> > > - ntmpi 23 -ntomp 1 -pinstride 1 -pin on
> > > - ntmpi 23 -ntomp 1 -pinstride 2 -pin on
> > >
> > > Thanks,
> > > --
> > > Szilárd
> > >
> > >
> > > On Fri, Mar 15, 2019 at 4:04 PM Tafelmeier, Stefanie <
> > > Stefanie.Tafelmeier at zae-bayern.de> wrote:
> > >
> > > > Hi Szilárd,
> > > >
> > > > thanks for the quick reply.
> > > > About the first suggestion, I'll try and give feedback soon.
> > > >
> > > > Regarding the second, I attached the log-file for the case of
> > > > mdrun -v -nt 25
> > > > Which ends in the known error message.
> > > >
> > > > Again, thanks a lot for your information and help.
> > > >
> > > > Best wishes,
> > > > Steffi
> > > >
> > > >
> > > >
> > > > -----Ursprüngliche Nachricht-----
> > > > Von: gromacs.org_gmx-users-bounces at maillist.sys.kth.se [mailto:
> > > > gromacs.org_gmx-users-bounces at maillist.sys.kth.se] Im Auftrag von
> > > Szilárd
> > > > Páll
> > > > Gesendet: Freitag, 15. März 2019 15:30
> > > > An: Discussion list for GROMACS users
> > > > Betreff: Re: [gmx-users] WG: WG: Issue with CUDA and gromacs
> > > >
> > > > Hi Stefanie,
> > > >
> > > > Unless and until the error and performance-related concerns prove to
> be
> > > > related, let's keep those separate.
> > > >
> > > > I'd first focus on the former. To be honest, I've never encountered
> > such
> > > an
> > > > issue where if you use more than a certain number of threads, the run
> > > > aborts with that error. To investigate further can you please apply
> the
> > > > following patch file which hopefully give more context to the error:
> > > > https://termbin.com/uhgp
> > > > (e.g. you can execute the following to accomplish that:
> > > > curl https://termbin.com/uhgp > devicebuffer.cuh.patch && patch -p0
> <
> > > > devicebuffer.cuh.patch)
> > > >
> > > > Regarding the performance-related questions, can you please share a
> > full
> > > > log file of the runs so we can see the machine config, simulation
> > > > system/settings, etc. Without that it is hard to judge what's best
> for
> > > your
> > > > case. However, if you only have a single GPU (which seems to be the
> > case
> > > > based on the log excerpts) along those two rather beefy CPUs, than
> you
> > > will
> > > > likely not get much benefit from using all cores and it is normal
> that
> > > you
> > > > see little to no improvement from using cores of a second CPU socket.
> > > >
> > > > Cheers,
> > > > --
> > > > Szilárd
> > > >
> > > >
> > > > On Thu, Mar 14, 2019 at 12:47 PM Tafelmeier, Stefanie <
> > > > Stefanie.Tafelmeier at zae-bayern.de> wrote:
> > > >
> > > > > Dear all,
> > > > >
> > > > > I was not sure if the email before reached you, but again many
> thanks
> > > for
> > > > > your reply Szilárd.
> > > > >
> > > > > As written below we are still facing a problem with the performance
> > of
> > > > > your workstation.
> > > > > I wrote before because of the error message when keeping occurring
> > for
> > > > > mdrun simulation:
> > > > >
> > > > > Assertion failed:
> > > > > Condition: stat == cudaSuccess
> > > > > Asynchronous H2D copy failed
> > > > >
> > > > > As I mentioned all Versions to install (Gormacs, Cuda, nvcc, gcc)
> are
> > > the
> > > > > newest once now.
> > > > >
> > > > > If I run mdrun without further settings it will lead to this error
> > > > > message. If I run it and choose the thread amount directly the
> mdrun
> > is
> > > > > performing well. But only for –nt numbers between 1 – 22. Higher
> ones
> > > > again
> > > > > lead to the before mentioned error message.
> > > > >
> > > > > In order to investigate in more detail, I tried different versions
> > for
> > > > > –nt, –ntmpi – ntomp also combined with –npme:
> > > > > - The best performance in the sense of ns/day is with –nt 22
> > > > > respectively –ntomp 22 alone. But then only 22 threads are
> involved.
> > > > Which
> > > > > is fine if I run more than one mdrun simultaneously, as I can
> > > distribute
> > > > > the other 66 threads. The GPU usage is then around 65%.
> > > > > - A similar good performance is reached with mdrun -ntmpi 4
> > > -ntomp
> > > > > 18 -npme 1 -pme gpu -nb gpu. But then 44 threads are involved. The
> > GPU
> > > > > usage is then around 50%.
> > > > >
> > > > > I read the information on
> > > > >
> > > >
> > >
> >
> http://manual.gromacs.org/documentation/5.1/user-guide/mdrun-performance.html
> > > > > which was very helpful, bur some things are still not clear now to
> > me:
> > > > > I was wondering if there is any other enhancement of the
> performance?
> > > Or
> > > > > what is the reason, that –nt maximum is at 22 threads? Could this
> be
> > > > > connected to the sockets (see details below) of your workstation?
> > > > > It is not clear to me how a number of thread (-nt) higher 22 can
> lead
> > > to
> > > > > the error regarding the Asynchronous H2D copy)
> > > > >
> > > > > Please excuse all these questions. I would appreciate a lot if you
> > > might
> > > > > have a hint for this problem as well.
> > > > >
> > > > > Best regards,
> > > > > Steffi
> > > > >
> > > > > -----
> > > > >
> > > > > The workstation details are:
> > > > > Running on 1 node with total 44 cores, 88 logical cores, 1
> compatible
> > > GPU
> > > > > Hardware detected:
> > > > >
> > > > > CPU info:
> > > > > Vendor: Intel
> > > > > Brand: Intel(R) Xeon(R) Gold 6152 CPU @ 2.10GHz
> > > > > Family: 6 Model: 85 Stepping: 4
> > > > > Features: aes apic avx avx2 avx512f avx512cd avx512bw avx512vl
> > > clfsh
> > > > > cmov cx8 cx16 f16c fma hle htt intel lahf mmx msr nonstop_tsc pcid
> > > > pclmuldq
> > > > > pdcm pdpe1gb popcnt pse rdrnd rdtscp rtm sse2 sse3 sse4.1 sse4.2
> > ssse3
> > > > tdt
> > > > > x2apic
> > > > >
> > > > > Number of AVX-512 FMA units: 2
> > > > > Hardware topology: Basic
> > > > > Sockets, cores, and logical processors:
> > > > > Socket 0: [ 0 44] [ 1 45] [ 2 46] [ 3 47] [ 4
> > > 48] [
> > > > > 5 49] [ 6 50] [ 7 51] [ 8 52] [ 9 53] [ 10 54] [
> 11
> > > 55]
> > > > > [ 12 56] [ 13 57] [ 14 58] [ 15 59] [ 16 60] [ 17 61] [
> > 18
> > > > > 62] [ 19 63] [ 20 64] [ 21 65]
> > > > > Socket 1: [ 22 66] [ 23 67] [ 24 68] [ 25 69] [ 26
> > > 70] [
> > > > > 27 71] [ 28 72] [ 29 73] [ 30 74] [ 31 75] [ 32 76] [
> 33
> > > 77]
> > > > > [ 34 78] [ 35 79] [ 36 80] [ 37 81] [ 38 82] [ 39 83] [
> > 40
> > > > > 84] [ 41 85] [ 42 86] [ 43 87]
> > > > > GPU info:
> > > > > Number of GPUs detected: 1
> > > > > #0: NVIDIA Quadro P6000, compute cap.: 6.1, ECC: no, stat:
> > > > compatible
> > > > >
> > > > > -----
> > > > >
> > > > >
> > > > >
> > > > > -----Ursprüngliche Nachricht-----
> > > > > Von: gromacs.org_gmx-users-bounces at maillist.sys.kth.se [mailto:
> > > > > gromacs.org_gmx-users-bounces at maillist.sys.kth.se] Im Auftrag von
> > > > Szilárd
> > > > > Páll
> > > > > Gesendet: Donnerstag, 31. Januar 2019 17:15
> > > > > An: Discussion list for GROMACS users
> > > > > Betreff: Re: [gmx-users] WG: Issue with CUDA and gromacs
> > > > >
> > > > > On Thu, Jan 31, 2019 at 2:14 PM Szilárd Páll <
> pall.szilard at gmail.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > On Wed, Jan 30, 2019 at 5:15 PM Tafelmeier, Stefanie
> > > > > > <Stefanie.Tafelmeier at zae-bayern.de> wrote:
> > > > > > >
> > > > > > > Dear all,
> > > > > > >
> > > > > > > We are facing an issue with the CUDA toolkit.
> > > > > > > We tried several combinations of gromacs versions and CUDA
> > > Toolkits.
> > > > > No Toolkit older than 9.2 was possible to try as there are no
> driver
> > > for
> > > > > nvidia available for a Quadro P6000.
> > > > > > > Gromacs
> > > > > >
> > > > > > Install the latest 410.xx drivers and it will work; the NVIDIA
> > driver
> > > > > > download website (https://www.nvidia.com/Download/index.aspx)
> > > > > > recommends 410.93.
> > > > > >
> > > > > > Here's a system with CUDA 10-compatible driver running o a system
> > > with
> > > > > > a P6000: https://termbin.com/ofzo
> > > > >
> > > > > Sorry, I misread that as "CUDA >=9.2 was not possible".
> > > > >
> > > > > Note that the driver is backward compatible, so you can use a new
> > > > > driver with older CUDA versions.
> > > > >
> > > > > Also note that the oldest driver NVIDIA claims to have P6000
> support
> > > > > is 390.59 which is, as far as I know, one gen older than the 396
> that
> > > > > the CUDA 9.2 toolkit came with. This is however, not something I'd
> > > > > recommend pursuing, use a new driver from the official site with
> any
> > > > > CUDA version that GROMACS supports and it should be fine.
> > > > >
> > > > > >
> > > > > > > CUDA
> > > > > > >
> > > > > > > Error message
> > > > > > >
> > > > > > > 2019
> > > > > > >
> > > > > > > 10.0
> > > > > > >
> > > > > > > gmx mdrun:
> > > > > > > Assertion failed:
> > > > > > > Condition: stat == cudaSuccess
> > > > > > > Asynchronous H2D copy failed
> > > > > > >
> > > > > > > 2019
> > > > > > >
> > > > > > > 9.2
> > > > > > >
> > > > > > > gmx mdrun:
> > > > > > > Assertion failed:
> > > > > > > Condition: stat == cudaSuccess
> > > > > > > Asynchronous H2D copy failed
> > > > > > >
> > > > > > > 2018.5
> > > > > > >
> > > > > > > 9.2
> > > > > > >
> > > > > > > gmx mdrun: Fatal error:
> > > > > > > HtoD cudaMemcpyAsync failed: invalid argument
> > > > > >
> > > > > > Can we get some more details on these, please? complete log files
> > > > > > would be a good start.
> > > > > >
> > > > > > > 5.1.5
> > > > > > >
> > > > > > > 9.2
> > > > > > >
> > > > > > > Installation make: nvcc fatal : Unsupported gpu architecture
> > > > > 'compute_20'*
> > > > > > >
> > > > > > > 2016.2
> > > > > > >
> > > > > > > 9.2
> > > > > > >
> > > > > > > Installation make: nvcc fatal : Unsupported gpu architecture
> > > > > 'compute_20'*
> > > > > > >
> > > > > > >
> > > > > > > *We also tried to set the target CUDA architectures as
> described
> > in
> > > > > the installation guide (
> > > > > manual.gromacs.org/documentation/2019/install-guide/index.html).
> > > > > Unfortunately it didn't work.
> > > > > >
> > > > > > What does it mean that it didn't work? Can you share the command
> > you
> > > > > > used and what exactly did not work?
> > > > > >
> > > > > > For the P6000 which is a "compute capability 6.1" device (for
> > anyone
> > > > > > who needs to look it up, go here:
> > > > > > https://developer.nvidia.com/cuda-gpus), you should set
> > > > > > cmake ../ -DGMX_CUDA_TARGET_SM="61"
> > > > > >
> > > > > > --
> > > > > > Szilárd
> > > > > >
> > > > > > > Performing simulations on CPU only always works, yet of cause
> are
> > > > more
> > > > > slowly than they could be with additionally using the GPU.
> > > > > > > The issue #2761 (https://redmine.gromacs.org/issues/2762)
> seems
> > > > > similar to our problem.
> > > > > > > Even though this issue is still open, we wanted to ask if you
> can
> > > > give
> > > > > us any information about how to solve this problem?
> > > > > > >
> > > > > > > Many thanks in advance.
> > > > > > > Best regards,
> > > > > > > Stefanie Tafelmeier
> > > > > > >
> > > > > > >
> > > > > > > Further details if necessary:
> > > > > > > The workstation:
> > > > > > > 2 x Xeon Gold 6152 @ 3,7Ghz (22 K, 44Th, AVX512)
> > > > > > > Nvidia Quadro P6000 with 3840 Cuda-Cores
> > > > > > >
> > > > > > > The simulations system:
> > > > > > > Long chain alkanes (previously used with gromacs 5.1.5 and CUDA
> > > 7.5 -
> > > > > worked perfectly)
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > ZAE Bayern
> > > > > > > Stefanie Tafelmeier
> > > > > > > Bereich Energiespeicherung/Division Energy Storage
> > > > > > > Thermische Energiespeicher/Thermal Energy Storage
> > > > > > > Walther-Meißner-Str. 6
> > > > > > > 85748 Garching
> > > > > > >
> > > > > > > Tel.: +49 89 329442-75
> > > > > > > Fax: +49 89 329442-12
> > > > > > > Stefanie.tafelmeier at zae-bayern.de<mailto:
> > > > > Stefanie.tafelmeier at zae-bayern.de>
> > > > > > > http://www.zae-bayern.de<http://www.zae-bayern.de/>
> > > > > > >
> > > > > > >
> > > > > > > ZAE Bayern - Bayerisches Zentrum für Angewandte
> Energieforschung
> > e.
> > > > V.
> > > > > > > Vorstand/Board:
> > > > > > > Prof. Dr. Hartmut Spliethoff (Vorsitzender/Chairman),
> > > > > > > Prof. Dr. Vladimir Dyakonov
> > > > > > > Sitz/Registered Office: Würzburg
> > > > > > > Registergericht/Register Court: Amtsgericht Würzburg
> > > > > > > Registernummer/Register Number: VR 1386
> > > > > > >
> > > > > > > Sämtliche Willenserklärungen, z. B. Angebote, Aufträge, Anträge
> > und
> > > > > Verträge, sind für das ZAE Bayern nur in schriftlicher und
> > > ordnungsgemäß
> > > > > unterschriebener Form rechtsverbindlich. Diese E-Mail ist
> > > ausschließlich
> > > > > zur Nutzung durch den/die vorgenannten Empfänger bestimmt. Jegliche
> > > > > unbefugte Offenbarung, Nutzung oder Verbreitung, sei es insgesamt
> > oder
> > > > > teilweise, ist untersagt. Sollten Sie diese E-Mail irrtümlich
> > erhalten
> > > > > haben, benachrichtigen Sie bitte unverzüglich den Absender und
> > löschen
> > > > Sie
> > > > > diese E-Mail.
> > > > > > >
> > > > > > > Any declarations of intent, such as quotations, orders,
> > > applications
> > > > > and contracts, are legally binding for ZAE Bayern only if expressed
> > in
> > > a
> > > > > written and duly signed form. This e-mail is intended solely for
> use
> > by
> > > > the
> > > > > recipient(s) named above. Any unauthorised disclosure, use or
> > > > > dissemination, whether in whole or in part, is prohibited. If you
> > have
> > > > > received this e-mail in error, please notify the sender immediately
> > and
> > > > > delete this e-mail.
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Gromacs Users mailing list
> > > > > > >
> > > > > > > * Please search the archive at
> > > > > http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> > > > > posting!
> > > > > > >
> > > > > > > * Can't post? Read
> http://www.gromacs.org/Support/Mailing_Lists
> > > > > > >
> > > > > > > * For (un)subscribe requests visit
> > > > > > >
> > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users
> > > > or
> > > > > send a mail to gmx-users-request at gromacs.org.
> > > > > --
> > > > > Gromacs Users mailing list
> > > > >
> > > > > * Please search the archive at
> > > > > http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> > > > > posting!
> > > > >
> > > > > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> > > > >
> > > > > * For (un)subscribe requests visit
> > > > > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users
> > or
> > > > > send a mail to gmx-users-request at gromacs.org.
> > > > > --
> > > > > Gromacs Users mailing list
> > > > >
> > > > > * Please search the archive at
> > > > > http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> > > > > posting!
> > > > >
> > > > > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> > > > >
> > > > > * For (un)subscribe requests visit
> > > > > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users
> > or
> > > > > send a mail to gmx-users-request at gromacs.org.
> > > > --
> > > > Gromacs Users mailing list
> > > >
> > > > * Please search the archive at
> > > > http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> > > > posting!
> > > >
> > > > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> > > >
> > > > * For (un)subscribe requests visit
> > > > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users
> or
> > > > send a mail to gmx-users-request at gromacs.org.
> > > > --
> > > > Gromacs Users mailing list
> > > >
> > > > * Please search the archive at
> > > > http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> > > > posting!
> > > >
> > > > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> > > >
> > > > * For (un)subscribe requests visit
> > > > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users
> or
> > > > send a mail to gmx-users-request at gromacs.org.
> > > --
> > > Gromacs Users mailing list
> > >
> > > * Please search the archive at
> > > http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> > > posting!
> > >
> > > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> > >
> > > * For (un)subscribe requests visit
> > > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> > > send a mail to gmx-users-request at gromacs.org.
> > > --
> > > Gromacs Users mailing list
> > >
> > > * Please search the archive at
> > > http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> > > posting!
> > >
> > > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> > >
> > > * For (un)subscribe requests visit
> > > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> > > send a mail to gmx-users-request at gromacs.org.
> > --
> > Gromacs Users mailing list
> >
> > * Please search the archive at
> > http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> > posting!
> >
> > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >
> > * For (un)subscribe requests visit
> > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> > send a mail to gmx-users-request at gromacs.org.
> > --
> > Gromacs Users mailing list
> >
> > * Please search the archive at
> > http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> > posting!
> >
> > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >
> > * For (un)subscribe requests visit
> > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> > send a mail to gmx-users-request at gromacs.org.
> --
> Gromacs Users mailing list
>
> * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> send a mail to gmx-users-request at gromacs.org.
> --
> Gromacs Users mailing list
>
> * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> send a mail to gmx-users-request at gromacs.org.
More information about the gromacs.org_gmx-users
mailing list