[gmx-users] MD workstation

Sat Oct 18 22:35:27 CEST 2014

On Fri, Oct 17, 2014 at 5:51 AM, Hadházi Ádám <hadadam at gmail.com> wrote:
> 2014-10-17 11:47 GMT+10:00 lloyd riggs <lloyd.riggs at gmx.ch>:
>
>>
>> Is there any progress in openCL versions of Gromacs, as it is listed on
>> the developer site?  Just askin.  One thing I ran across is one can get
>> integrated GPU arrays on a board if you find say Russian board designs from
>> China for about the same price with 10x the computational speed, but the
>> boards would be largly OpenCL dependent.
>>
>> Stephan Watkins
>>  *Gesendet:* Donnerstag, 16. Oktober 2014 um 20:21 Uhr
>> *Von:* "Szilárd Páll" <pall.szilard at gmail.com>
>> *An:* "Discussion list for GROMACS users" <gmx-users at gromacs.org>
>> *Betreff:* Re: [gmx-users] MD workstation
>> On Thu, Oct 16, 2014 at 3:35 PM, Hadházi Ádám <hadadam at gmail.com> wrote:
>> > May I ask why your config is better than e.g.
>> >
>> > 2x Intel Xeon E5-2620 CPUs (2x$405)
>> > 4x GTX 970(4x $330)
>> > 1x Z9PE-D8 WS ($449)
>> > 64 GB DDR3 ($600)
>> > PSU 1600W, ($250)
>> > standard 2TB 5400rpm drive, ($85)
>> > total: (~$3500)
>>
>> Mirco's suggested setup will give much higher *aggregate* simulation
>> throughput. GROMACS uses both CPUs and GPUs and requires a balanced
>> resource mix to run efficiently (less so if you don't use PME). The
>> E5-2620 is rather slow and it will be a good match for a single GTX
>> 970, perhaps even a 980, but it will be the limiting factor with two
>> GPUs per socket.
>>
>
> Maybe this is not the best forum for this question, but I also plan to use
> AMBER and Desmond for MD/FEP purposes.
> Question: Is the recommended 4x1 node config the best setup for these 2
> other software too?

What's best for those codes you should indeed ask on their forums.
However, as AMBER-GPU (and AFAIK Desmond too) doesn't rely on the CPU
except for coordinating a run so 1 core/GPU is likely enough. While
the hardware needs are quite different, I think you would get similar
performance/$ if you spread out 4-6 GPUs in multiple workstations with
fast consumer CPUs rather than one low-end server. Moreover, the
former will still work well with GROMACS, while the latter will not.

>>
>> > As for your setup...can I use that 4 nodes in parallel for 1 long
>> > simulation or 1 FEP job?
>>
>> Not without a fast network.
>>
> What would be a fast network? How should I connect them?

Infiniband is recommended, some had decent results with Ethernet too,
but cheap hardware without tweaking the drivers/OS will likely not
give good results in simulations across the network - especially not
with PME. You can still run multi-runs across machines even with cheap
Ethernet ("native" like replica exchange or "artificial" like
non-communicating independent runs).

> Is 8GB memory really enough/node?

GROMACS needs quite little memory so 8 GB should be enough for
everything except exotic things.

>>
>> > What are the weak points of my workstation?
>>
>> The CPU. Desktop IVB-E or HSW-E (e.g. i7 49XX, 59XX) will give much
>> better performance per buck.
>>
>> Also note:
>> * your smaller 25k MD setup will not scale across multiple GPUs;
>> * in FEP runs you, by sharing a GPU between multiple runs you can
>> increase the aggregate throughput by quite a lot!
>>
>> Do you mean, by sharing 1 GPU between multiple FEP runs, or  by sharing
> more GPUs between multiple FEP runs?

Both can work, but sharing one GPU between 2-3 FEP runs is more
straightforward and more efficient too (no domain-decomposition
needed). The GPU does only force compute (concurrently with the CPU),
so it will be idle during constraints, integration, neighbor search,
etc. If multiple independent simulations share a GPU, hardware can be
utilized better and therefore the aggregate simulation performance
will increase.

The gain can be especially impressive on large multi-core CPUs with
small systems; e.g. see the second plot here: http://goo.gl/2xH52y

> We have an access to a server having multiple nodes... both of them
> possesses 4x Intel X5675 (3.07GHz) CPU (each 6 cores) and 6x NVIDIA Tesla
> M2070 GPU). Total: 2 x (24 cores and 6 GPU).
> Question: Is this a good construction for MD/FEP?

Do you mean six M2070's per node or across two nodes?

Depending on the setup, your larger systems will likely scale OK to
3-4 GPUs, but you'll have to try. The smaller one should run quite
well with FEP + sharing GPUs between runs. Comparing the log files of
a few short runs with different setups is the best way to tell.

Cheers,
--
Szilárd

> Regards,
> Ádám
>
>> Cheers,
>> --
>> Szilárd
>>
>> > Best,
>> > Adam
>> >
>> >
>> > 2014-10-16 23:00 GMT+10:00 Mirco Wahab <
>> mirco.wahab at chemie.tu-freiberg.de>:
>> >
>> >> On 16.10.2014 14:38, Hadházi Ádám wrote:
>> >>
>> >>> Dear GMX Stuff and Users,
>> >>>>> I am planning to buy a new MD workstation with 4 GPU (GTX 780 or 970)
>> >>>>> or 3
>> >>>>> GPU (GTX 980) for 4000$.
>> >>>>> Could you recommend me a setup for this machine?
>> >>>>> 1 or 2 CPU is necessary? 32/64 GB memory? Cooling? Power?
>> >>>>>
>> >>>>
>> >>>> - What system (size, type, natoms) do you plan to simulate?
>> >>>>
>> >>>> - Do you have to run *only one single simulation* over long time
>> >>>> or *some similar simulations* with similar parameters?
>> >>>>
>> >>>
>> >>> The systems are kind of mix:
>> >>> MD:
>> >>> smallest system: 25k atoms, spc/tip3p, 2fs/4fs, NPT, simulation time:
>> >>> 500-1000ns
>> >>> biggest system: 150k atoms, spc/tip3p, 2fs/4fs, NPT, simulation time:
>> >>> 100-1000ns
>> >>> FEP (free energy perturbation): ligand functional group mutation
>> >>> 25k-150k atoms, in complex and in water simulations, production
>> >>> simulation:
>> >>> 5ns for each lambda window (number of windows: 12)
>> >>>
>> >>
>> >> In this situation, I'd probably use 4 machines for $1000 each,
>> >> putting in each:
>> >> - consumer i7/4790(K), $300
>> >> - any 8GB DDR3, $75-$80
>> >> - standard Z97 board, $100
>> >> - standard PSU 450W, $40
>> >> - standard 2TB 5400rpm drive, $85
>> >>
>> >> the rest of the money (4 x $395), I'd use for 4 graphics
>> >> cards, probably 3 GTX-970 ($330) and one GTX-980 ($550) -
>> >> depending on availability, the actual prices, and
>> >> your detailed budget.
>> >>
>> >> YMMV,
>> >>
>> >>
>> >> Regards
>> >>
>> >> M.
>> >>
>> >> --
>> >> Gromacs Users mailing list
>> >>
>> >> * Please search the archive at http://www.gromacs.org/
>> >> Support/Mailing_Lists/GMX-Users_List before posting!
>> >>
>> >> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>> >>
>> >> * For (un)subscribe requests visit
>> >> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
>> >> send a mail to gmx-users-request at gromacs.org.
>> >>
>> > --
>> > Gromacs Users mailing list
>> >
>> > * Please search the archive at
>> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
>> posting!
>> >
>> > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>> >
>> > * For (un)subscribe requests visit
>> > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
>> send a mail to gmx-users-request at gromacs.org.
>> --
>> Gromacs Users mailing list
>>
>> * Please search the archive at
>> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
>> posting!
>>
>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>>
>> * For (un)subscribe requests visit
>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
>> send a mail to gmx-users-request at gromacs.org.
>>
>> --
>> Gromacs Users mailing list
>>
>> * Please search the archive at
>> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
>> posting!
>>
>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>>
>> * For (un)subscribe requests visit
>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
>> send a mail to gmx-users-request at gromacs.org.
>>
>>
> --
> Gromacs Users mailing list
>
> * Please search the archive at http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a mail to gmx-users-request at gromacs.org.