[gmx-users] Re: mdrun segmentation fault for new build of gromacs 4.6.1

Tue Jun 11 23:26:32 CEST 2013

Hi Amil,

On Mon, Jun 10, 2013 at 11:03 PM, Amil Anderson
<aanderson at wittenberg.edu> wrote:
> Szilárd,
>
> I tried the build using a PATH without the Intel compilers and NOT including the directives -DGMX_EXTERNAL_BLAS=OFF and -DGMX_EXTERNAL_LAPACK=OFF and it used the FFTW 3.2 wrappers to MKL.  Here is the output from mdrun -version:

Note that the aim of both workarounds is to stop the build system from
picking up the MKL BLAS/LAPACK libraries. Hence, I suspect that you
may have not been successful at preventing cmake to pick up the Intel
Compilers' location. I may very well be (have not looked at the code,
though) that the cmake find module looks at the standard installation
location (e.g. /opt/intel) and picks up automatically if something is
found there. Hence, removing MKL form the path may not help at all.

In any case, I suppose this should be fixed in 4.6.2, but you should
wait for the imminent bugfix release, 4.6.3.

Cheers,
Szilard

>
> Program: mdrun
> Gromacs version:    VERSION 4.6.1
> Precision:          single
> Memory model:       64 bit
> MPI library:        thread_mpi
> OpenMP support:     enabled
> GPU support:        disabled
> invsqrt routine:    gmx_software_invsqrt(x)
> CPU acceleration:   SSE4.1
> FFT library:        FFTW 3.2 wrappers to MKL
> Large file support: enabled
> RDTSCP usage:       enabled
> Built on:           Mon Jun 10 15:34:03 CDT 2013
> Built by:           softwaremgmt at warp2-login.cluster<mailto:softwaremgmt at warp2-login.cluster> [CMAKE]
> Build OS/arch:      Linux 2.6.18-308.20.1.el5 x86_64
> Build CPU vendor:   GenuineIntel
> Build CPU brand:    Intel(R) Xeon(R) CPU           E5520  @ 2.27GHz
> Build CPU family:   6   Model: 26   Stepping: 5
> Build CPU features: apic clfsh cmov cx8 cx16 htt lahf_lm mmx msr nonstop_tsc pdcm popcnt pse rdtscp sse2 sse3 sse4.1 sse4.2 ssse3
> C compiler:         /opt/centos/devtoolset-1.0/root/usr/bin/gcc GNU gcc (GCC) 4.7.0 20120507 (Red Hat 4.7.0-5)
> C compiler flags:   -msse4.1   -Wextra -Wno-missing-field-initializers -Wno-sign-compare -Wall -Wno-unused -Wunused-value   -fomit-frame-pointer -funroll-all-loops -fexcess-precision=fast  -O3 -DNDEBUG
>
>
> All the regressiontests failed.  So it appears that, at least for my system, I need to include the directives to not use the external BLAS/LAPACK.
>
> Amil
>
>
>
> On Jun 10, 2013, at 12:12 PM, Szilárd Páll [via GROMACS] wrote:
>
> Amil,
>
> It looks like there is a mixup in your software configuration and
> mdrun is linked against libguide.so, the OpenMP library part of the
> Intel compiler v11 which gets loaded early and is probably causing the
> crash. This library was probably pulled in implicitly by MKL which the
> build system detected and tried to use for BLAS/LAPACK (note that you
> have "FFT library:        FFTW 3.2 wrappers to MKL" in the log and I
> assume you intended to use FFTW). I was under the impression that this
> issue was solved in 4.6.1 (http://redmine.gromacs.org/issues/1067),
> but perhaps that's not the case.
>
> I suggest you make sure that mdrun is not linked against any Intel
> library. The easiest way to achieve this is to either:
> - set -DGMX_EXTERNAL_BLAS=OFF and -DGMX_EXTERNAL_LAPACK=OFF so that
> the internal BLAS/LAPACK is used or
> - make sure that the Intel Compilers/MKL installation is not in the
> path so that the build system does not detect it.
>
> Check the mdrun -version output, specifically the reported "FFT
> library", after you recompile mdrun.
>
> Cheers,
> --
> Szilárd
>
>
> On Mon, Jun 10, 2013 at 4:57 PM, Amil Anderson <[hidden email]<x-msg://32/user/SendEmail.jtp?type=node&node=5008989&i=0>> wrote:
>
>> Roland,
>>
>> Here is the output from the debug run with the stack trace:
>>
>> [aanderson at warp2-login gmxdemo]$ gdb mdrun
>> GNU gdb Fedora (6.8-27.el5)
>> Copyright (C) 2008 Free Software Foundation, Inc.
>> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
>> This is free software: you are free to change and redistribute it.
>> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
>> and "show warranty" for details.
>> This GDB was configured as "x86_64-redhat-linux-gnu"...
>> (gdb) run -nt 1 -s cpeptide_em -o cpeptide_em -c cpeptide_b4pr -v
>> Starting program: /shared/software/gromacs-4.6.1/bin/mdrun -nt 1 -s cpeptide_em -o cpeptide_em -c cpeptide_b4pr -v
>> [Thread debugging using libthread_db enabled]
>>                          :-)  G  R  O  M  A  C  S  (-:
>>
>>                   Gromacs Runs On Most of All Computer Systems
>>
>>                             :-)  VERSION 4.6.1  (-:
>> ...
>>
>> Back Off! I just backed up md.log to ./#md.log.18#
>> Reading file cpeptide_em.tpr, VERSION 4.6.1 (single precision)
>> Using 1 MPI thread
>> [New Thread 0x2b5fe6188ee0 (LWP 19572)]
>>
>> Program received signal SIGSEGV, Segmentation fault.
>> 0x00002b5fe3865479 in omp_set_num_threads ()
>>    from /shared/software/intel/Compiler/11.1/046/lib/intel64/libguide.so
>> (gdb) bt
>> #0  0x00002b5fe3865479 in omp_set_num_threads ()
>>    from /shared/software/intel/Compiler/11.1/046/lib/intel64/libguide.so
>> #1  0x00002b5fe514c7f8 in gmx_omp_set_num_threads (num_threads=1)
>>     at /shared/software/temp/gromacs-4.6.1/include/vec.h:885
>> #2  0x00002b5fe5199862 in gmx_omp_nthreads_init (fplog=0x17e14b70,
>>     cr=0x17e14010, nthreads_hw_avail=16, omp_nthreads_req=1,
>>     omp_nthreads_pme_req=1, bThisNodePMEOnly=0, bFullOmpSupport=0)
>>     at /shared/software/temp/gromacs-4.6.1/include/vec.h:885
>> #3  0x0000000000410a6a in mdrunner (hw_opt=0x7fff7f3d78c0, fplog=0x17e14b70,
>>     cr=0x17e14010, nfile=36, fnm=0x7fff7f3d7a50, oenv=0x17e146d0, bVerbose=1,
>>     bCompact=1, nstglobalcomm=-1, ddxyz=0x7fff7f3d7310, dd_node_order=1,
>>     rdd=0, rconstr=0, dddlb_opt=0x438449 "auto", dlb_scale=0.800000012,
>>     ddcsx=0x0, ddcsy=0x0, ddcsz=0x0, nbpu_opt=0x438449 "auto",
>>     nsteps_cmdline=-2, nstepout=100, resetstep=-1, nmultisim=0, repl_ex_nst=0,
>>     repl_ex_nex=0, repl_ex_seed=-1, pforce=-1, cpt_period=15, max_hours=-1,
>>     deviceOptions=0x43846c "", Flags=1055744)
>>     at /shared/software/temp/gromacs-4.6.1/include/vec.h:885
>> #4  0x0000000000429cd1 in cmain (argc=1, argv=0x7fff7f3d8c18)
>>     at /shared/software/temp/gromacs-4.6.1/include/vec.h:885
>> #5  0x0000000000430fd8 in main (argc=10, argv=0x7fff7f3d8c18)
>>     at /shared/software/temp/gromacs-4.6.1/src/kernel/main.c:29
>> (gdb)
>>
>>
>> I compiled with gcc-4.7.0 so I'm surprised to see a reference to the Intel compiler.
>>
>> Amil
>>
>> On Jun 9, 2013, at 6:08 PM, Roland Schulz [via GROMACS] wrote:
>>
>> Hi,
>>
>> based on Mark's idea I would have thought that the cpu detection would
>> have already failed during cmake. But it seems it detected SSE4.1
>> correctly.
>> Could you post the stack trace for the crash? (see previous mail for
>> instructions)
>>
>> Roland
>>
>> On Sun, Jun 9, 2013 at 4:42 PM, Amil Anderson <[hidden email]<x-msg://16/user/SendEmail.jtp?type=node&node=5008945&i=0>> wrote:
>>
>>> Roland,
>>>
>>> I have posted the cmake output (cmake-4.6.1) and the file CMakeError.log at
>>> the usual
>>>
>>> https://www.dropbox.com/sh/h6867f7ivl5pcl9/j9gt9CsVdP
>>>
>>> I see there are some errors but don't know what to make of them.
>>>
>>> Thanks,
>>> Amil
>>>
>>>
>>>
>>> --
>>> View this message in context: http://gromacs.5086.x6.nabble.com/mdrun-segmentation-fault-for-new-build-of-gromacs-4-6-1-tp5008873p5008944.html
>>> Sent from the GROMACS Users Forum mailing list archive at Nabble.com<http://Nabble.com<http://Nabble.com/>>.
>>> --
>>> gmx-users mailing list    [hidden email]<x-msg://16/user/SendEmail.jtp?type=node&node=5008945&i=1>
>>> http://lists.gromacs.org/mailman/listinfo/gmx-users
>>> * Please search the archive at http://www.gromacs.org/Support/Mailing_Lists/Search before posting!
>>> * Please don't post (un)subscribe requests to the list. Use the
>>> www interface or send it to [hidden email]<x-msg://16/user/SendEmail.jtp?type=node&node=5008945&i=2>.
>>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>>>
>>>
>>>
>>>
>>
>>
>>
>> --
>> ORNL/UT Center for Molecular Biophysics cmb.ornl.gov<http://cmb.ornl.gov<http://cmb.ornl.gov/>>
>> 865-241-1537, ORNL PO BOX 2008 MS6309
>> --
>> gmx-users mailing list    [hidden email]<x-msg://16/user/SendEmail.jtp?type=node&node=5008945&i=3>
>> http://lists.gromacs.org/mailman/listinfo/gmx-users
>> * Please search the archive at http://www.gromacs.org/Support/Mailing_Lists/Search before posting!
>> * Please don't post (un)subscribe requests to the list. Use the
>> www interface or send it to [hidden email]<x-msg://16/user/SendEmail.jtp?type=node&node=5008945&i=4>.
>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>>
>>
>> ________________________________
>> If you reply to this email, your message will be added to the discussion below:
>>
>> NAML<<x-msg://32/>http://gromacs.5086.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>
>>
>>
>>
>>
>> --
>> View this message in context: http://gromacs.5086.x6.nabble.com/mdrun-segmentation-fault-for-new-build-of-gromacs-4-6-1-tp5008873p5008986.html
>> Sent from the GROMACS Users Forum mailing list archive at Nabble.com<http://Nabble.com>.
>> --
>> gmx-users mailing list    [hidden email]<x-msg://32/user/SendEmail.jtp?type=node&node=5008989&i=1>
>> http://lists.gromacs.org/mailman/listinfo/gmx-users
>> * Please search the archive at http://www.gromacs.org/Support/Mailing_Lists/Search before posting!
>> * Please don't post (un)subscribe requests to the list. Use the
>> www interface or send it to [hidden email]<x-msg://32/user/SendEmail.jtp?type=node&node=5008989&i=2>.
>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> --
> gmx-users mailing list    [hidden email]<x-msg://32/user/SendEmail.jtp?type=node&node=5008989&i=3>
> http://lists.gromacs.org/mailman/listinfo/gmx-users
> * Please search the archive at http://www.gromacs.org/Support/Mailing_Lists/Search before posting!
> * Please don't post (un)subscribe requests to the list. Use the
> www interface or send it to [hidden email]<x-msg://32/user/SendEmail.jtp?type=node&node=5008989&i=4>.
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
>
> ________________________________
> If you reply to this email, your message will be added to the discussion below:
> http://gromacs.5086.x6.nabble.com/mdrun-segmentation-fault-for-new-build-of-gromacs-4-6-1-tp5008873p5008989.html
> To unsubscribe from mdrun segmentation fault for new build of gromacs 4.6.1, click here<http://gromacs.5086.x6.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=5008873&code=YWFuZGVyc29uQHdpdHRlbmJlcmcuZWR1fDUwMDg4NzN8NTI5NTk5MTIw>.
> NAML<http://gromacs.5086.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
>
>
>
>
> --
> View this message in context: http://gromacs.5086.x6.nabble.com/mdrun-segmentation-fault-for-new-build-of-gromacs-4-6-1-tp5008873p5009001.html
> Sent from the GROMACS Users Forum mailing list archive at Nabble.com.
> --
> gmx-users mailing list    gmx-users at gromacs.org
> http://lists.gromacs.org/mailman/listinfo/gmx-users
> * Please search the archive at http://www.gromacs.org/Support/Mailing_Lists/Search before posting!
> * Please don't post (un)subscribe requests to the list. Use the
> www interface or send it to gmx-users-request at gromacs.org.
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists