[gmx-users] Specs for GPU box

Wed Jan 18 14:25:40 CET 2017

On Thu, Jan 12, 2017 at 2:19 AM, Alex <nedomacho at gmail.com> wrote:
> Szilárd, I think I am starting to understand and appreciate your argument.
> In essence, you are warning me against building a beast of stupidity. : )

Indeed. Note however, that the kind of multi-socket & multi-GPU beast
you are considering is not too common in the classical HPC world, so
take my word with a grain of salt. Perhaps ask your favorite vendor
for technical advice and tell them the same thing you told me (clock
speed + many cores / sockets + cache, and also many GPUs).

>
>> Sounds like you're in a situation that's not too bad. Just for the
>> $5-10k extra you'd get decent MD server! ;)
>> (Have you considered donating the leftover $ e.g. to GROMACS? ;)
>
> Trust me when I say that my employer's donation to GMX would bring tears to
> both your eyes and mine. Those would not be tears of joy, unfortunately. : (

Hmmm, does that mean I should stop typing, press "discard mail" and move on?

>>
>> Seems like you want to have the cake and eat it. Especially clock rate
>> and cache are somewhat orthogonal in all but the crazy-priced E7 chips
>> (like E7-8890/91). Even if you'd get the E7-8890 in a 4-socket setup,
>> I you'll be able to plug in at most 4 GPUs, probably no GeForce.
>> Supermicro has some large 5U up to 8S servers, no idea what other
>> vendors have as I'm mostly used to 2S GPU.
>
> Well, I'd like to have the cake and eat it, yes. However, now that I went
> into more detail looking at the differences between E5 and E7, your point is
> quite clear.
>
>> Insisting on a single E7 + GPU server will most probably not give the
>> best of both worlds. You will likely end up with a machine that's
>> mediocre for many if not most tasks. Why not split the
>> responsibilities and get two separate machines e.g. a large memory
>> 4-socket CPU-only and a separate 2S (or 4S) with GPUs?
>>
>> BTW, E5-4xxx does 4S, do you really _need_ E7?
>
> Not much of an insistence. We were leaning against multiple low core count
> nodes to avoid dealing with interconnects.

I see your point, but note that complexity is often not only in the
interconnect. Node sharing among (by jobs / users) will bring up many
of the management/scheduling issues that arise with (simpler)
multi-node setups. Secondly, depending on the planned job concurrency,
you may need to care little about the (hw/sw) hurdles of scaling
across machines.

The important question is what is the job distribution, size, resource
and time to completion requirement? E.g. If you'll always have a dozen
small runs in the pipeline, you might as well optimize for job
throughput rather than latency (e.g. always run 1-2 jobs/GPU).

> However, your point about two
> E5-based machines (one with GPUs, one without) is perfectly attractive.
> If you were to build an E5-based GPU-accelerated _high-core_ count MD server
> with a budget of $20-25K, what would you go with?

Something like 2xE5 269x + 2-4x TITAN X-P; what's best will depend on
the kind of simulations you want to run, so that question really is
best addressed first! Small simulation systems will anyway not scale
across GPUs, so you might want more of the less power-hungry GPUs. If
you want (and can) run across the entire node, I'd go for fewer of the
fastest GPUs (e.g. 2-3 TITAN X-P).

Note that GROMACS uses offload-based acceleration, so hardware balance
is important! Have you checked out doi:10.1002/jcc.24030?

>>
>> GeForce TITAN X. (Money no issue? Get Tesla P40 or P100. Both will be
>> slower, tho).
>
> What do you mean "both will be slower?"

Both Pascal Teslas are slower than the consumer TITAN X-P for the
compute-bound GROMACS kernels (and in fact for many if not most SP
compute-bound codes). The TITAN X-P is slower on paper than the P40,
but it has a crazy sustained clock rate (~1850 MHz).

Cheers,
--
Szilárd

>>
>> Sounds good. We'll definitely be interested if you end up with
>> something really dense and packing a good punch.
>>
> I will be happy to share the results, once I have something to share.
> Especially if it's something good!
>
> Thanks,
>
> Alex
>
>
>>
>>> Thank you!
>>>
>>> Alex
>>>
>>>
>>>
>>> On 1/6/2017 6:05 AM, Szilárd Páll wrote:
>>>>
>>>> Hi Alex,
>>>>
>>>> Benchmarks of quad-socket Intel machines are rare because AFAIK such
>>>> systems are mighty expensive and you will not get good bang for buck
>>>> with
>>>> them, especially if you combine these pricey nodes/CPUs with the old and
>>>> slow K80s.
>>>>
>>>> The only reason to get E7 is if >=4 sockets or >1.5 TB memory per node
>>>> is
>>>> a
>>>> must. Furthermore, the only reason to buy K80s today (for GROMACS) if
>>>> they
>>>> are dirt-cheap (e.g. free :).
>>>>
>>>> You'll be much better off with:
>>>> - 1-2-socket Broadwell nodes
>>>> - P100 if you need Tesla, GeForce 1070/1080
>>>>
>>>> However, more importantly, what kind of simulations you want to run? For
>>>> 50K you might be able to get multiple nodes with optimal price/perf.
>>>>
>>>> Cheers,
>>>> --
>>>> Szilárd
>>>>
>>>> On Tue, Dec 27, 2016 at 9:24 PM, Alex <nedomacho at gmail.com> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> We've got some dedicated funding (~50K) for a computing box. GMX will
>>>>> be
>>>>> one of the applications used there (the other MD package would be
>>>>> LAMMPS,
>>>>> which has similar requirements). Other applications would be ab initio
>>>>> and
>>>>> DFT packages, so, aside from a ton of RAM and possibly a fast SSD for
>>>>> scratch, there aren't too many requirements.
>>>>>
>>>>> My question is about an "optimal" CPU-GPU combination. Initially, we
>>>>> wanted
>>>>> something like a quad-Xeon (something relatively senior in the E7
>>>>> family,
>>>>> ~48-64 cores total) with two K80 cards, but I can't find anything like
>>>>> this
>>>>> in your benchmark documents.
>>>>>
>>>>> Can someone help spec this thing?
>>>>>
>>>>> Thanks a lot,
>>>>>
>>>>> Alex
>>>>> --
>>>>> Gromacs Users mailing list
>>>>>
>>>>> * Please search the archive at http://www.gromacs.org/Support
>>>>> /Mailing_Lists/GMX-Users_List before posting!
>>>>>
>>>>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>>>>>
>>>>> * For (un)subscribe requests visit
>>>>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
>>>>> send a mail to gmx-users-request at gromacs.org.
>>>>>
>>> --
>>> Gromacs Users mailing list
>>>
>>> * Please search the archive at
>>> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
>>> posting!
>>>
>>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>>>
>>> * For (un)subscribe requests visit
>>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
>>> send a
>>> mail to gmx-users-request at gromacs.org.
>
>
> --
> Gromacs Users mailing list
>
> * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a
> mail to gmx-users-request at gromacs.org.