[gmx-users] Number of Xeon cores per GTX 1080Ti

Fri Apr 6 17:39:02 CEST 2018

I went on to refresh my memory a bit and I thought I'd share a summary
for those curious.

The actual clocks CPU will be running at are determined by a number of
factors: base and boost clocks (by spec), actual boost clocks
achievable (depending on cores used, workload, TDP) and this is not
hugely different across architectures.

When it comes to Intel, things are somewhat complicated. Since Haswell
the base clocks depend on the vector instructions used: there is a
"non-AVX" base which is what the official specs show, and a lower one
for AVX, generally a few hundred MHz less (and only in-depth searching
will reveal). More recently on Skylake further throttle is applied
when AVX512 code is executed; more info on Wikichip [0].

Intel has been (sadly getting away with) omitting this data on their
specs for quite long e.g. [1] and details are hard to find. For
Broadwell CPUs, here's the data sheet that shows "turbo bin" vs cores
used: [2], but that's missing the AVX clocks and I have not come
across any official info on those; there's however a decent guide that
has a lot of data nicely visualized [3]. With Skylake they've been
somewhat more open about the specs and similar data is available on
spec sheets [4], or on the previously linked site [5].

[0] https://en.wikichip.org/wiki/intel/frequency_behavior
[1] https://ark.intel.com/products/64593/Intel-Xeon-Processor-E5-2630-15M-Cache-2_30-GHz-7_20-GTs-Intel-QPI
[2] https://www.intel.com/content/dam/www/public/us/en/documents/specification-updates/xeon-e5-v4-spec-update.pdf
[3] https://www.microway.com/knowledge-center-articles/detailed-specifications-of-the-intel-xeon-e5-2600v4-broadwell-ep-processors/
[4] https://www.intel.com/content/www/us/en/processors/xeon/scalable/xeon-scalable-spec-update.html
[5] https://www.microway.com/knowledge-center-articles/detailed-specifications-of-the-skylake-sp-intel-xeon-processor-scalable-family-cpus/
--
Szilárd

On Thu, Apr 5, 2018 at 10:22 AM, Jochen Hub <jhub at gwdg.de> wrote:
>
>
> Am 03.04.18 um 19:03 schrieb Szilárd Páll:
>
>> On Tue, Apr 3, 2018 at 5:10 PM, Jochen Hub <jhub at gwdg.de> wrote:
>>>
>>>
>>>
>>> Am 03.04.18 um 16:26 schrieb Szilárd Páll:
>>>
>>>> On Tue, Apr 3, 2018 at 3:41 PM, Jochen Hub <jhub at gwdg.de> wrote:
>>>>>
>>>>>
>>>>> benchmar
>>>>>
>>>>> Am 29.03.18 um 20:57 schrieb Szilárd Páll:
>>>>>
>>>>>> Hi Jochen,
>>>>>>
>>>>>> For that particular benchmark I only measured performance with
>>>>>> 1,2,4,8,16 cores with a few different kinds of GPUs. It would be easy
>>>>>> to do the runs on all possible core counts with increments of 1, but
>>>>>> that won't tell a whole lot more than what the performance is of a run
>>>>>> using a E5-2620 v4 CPU (with some GPUs) on a certain core count. Even
>>>>>> extrapolating from that 2620 to a E5-2630 v4 and expecting to get a
>>>>>> good estimate is tricky (given that the latter has 25% more cores for
>>>>>> the same TDP!), let alone to any 26xxv4 CPU or the current-gen Skylake
>>>>>> chips which have different performance characteristics.
>>>>>>
>>>>>> As Mark notes, there are some mdp option as well as some system
>>>>>> charateristics that will have a strong influence on performance -- if
>>>>>> tens of % is something you consider "strong" (some users are fine to
>>>>>> be within a 2x ballpark :).
>>>>>>
>>>>>> What's worth considering is to try to avoid ending up strongly CPU or
>>>>>> GPU bound from start. That may admittedly could be a difficult task
>>>>>> you would run e.g. both biased MD with large pull groups and all-bonds
>>>>>> constraints with Amber FF on large-ish (>100k) systems as well as
>>>>>> vanilla MD with CHARMM FF with small-ish (<25k) systems. On the same
>>>>>> hardware the former will be more prone to be CPU-bound while the
>>>>>> latter will have relatively more GPU-heavy workload.
>>>>>>
>>>>>> There are many factors that influence the performance of a run and
>>>>>> therefore giving a the one right answer to your question is not really
>>>>>> possible. What can say is that 7-10 "core-GHz" per fast Pascal GPU is
>>>>>> generally sufficient for "typical" protein simulations to run at >=85%
>>>>>> of peak.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Hi Szilárd,
>>>>>
>>>>> many thanks, this alrady helps me a lot. Just to get it 100% clear what
>>>>> you
>>>>> mean with core-GHz: A 10-core E5-2630v4 with 2.2 GHz would have 22
>>>>> core-GHz,
>>>>> right?
>>>>
>>>>
>>>>
>>>> Yes, that's what I was referring to; note that a 2630v4 won't be
>>>> running at a 2.2 GHz base clock if you run AVX code ;)
>>>
>>>
>>>
>>> Okay, I didn't know this. What would be the base clock instead with AVX
>>> code?
>>
>>
>>
>> Short version: It's not easy to out details as Intel conveniently
>> omits it from the specs, but it's AFAIK 3-400 MHz lower; also note
>> that "turbo bins" change as a function of cores used (so you can't
>> just benchmark on a few cores leaving the rest idle). Also, the actual
>> clock speed (and overall performance) depend on other factors too so
>> benchmarking and extrapolation might require consider other factors
>> too.
>>
>> Let me know if you are interested in more details.
>
>
> Hi Szilárd,
>
> many thanks, that helps a lot!
>
> Best,
> Jochen
>
>
>>
>> --
>> Szilárd
>>
>>>
>>>>
>>>>> Thanks,
>>>>> Jochen
>>>>>
>>>>>
>>>>>>
>>>>>> Cheers,
>>>>>> --
>>>>>> Szilárd
>>>>>>
>>>>>>
>>>>>> On Wed, Mar 28, 2018 at 4:31 AM, Mark Abraham
>>>>>> <mark.j.abraham at gmail.com>
>>>>>> wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> On Tue, Mar 27, 2018 at 6:43 PM Jochen Hub <jhub at gwdg.de> wrote:
>>>>>>>
>>>>>>>> Dear Gromacs community, dear Mark,
>>>>>>>>
>>>>>>>> Mark showed in the webinar today that having more than 8 Xeon
>>>>>>>> E5-26XXv4
>>>>>>>> cores does not help when using a GTX 1080Ti and PME on the GPU.
>>>>>>>>
>>>>>>>
>>>>>>> ... for that particular simulation system.
>>>>>>>
>>>>>>>
>>>>>>>> Unfortunately, there were no data points between 4 and 8 CPU cores,
>>>>>>>> hence it was not clear at which #cores the performance actually
>>>>>>>> levels
>>>>>>>> off. With a GTX 1080 (not Ti) I once found that having more than 5
>>>>>>>> Xeon
>>>>>>>> cores does not help, if not having UB potentials, but I don't have a
>>>>>>>> 1080Ti at hand to test for that.
>>>>>>>>
>>>>>>>
>>>>>>> Those data points may not have been run. Szilard might have the data
>>>>>>> -
>>>>>>> this
>>>>>>> was GLIC 2fs comparing 1080 with 1080Ti from the recent plots he
>>>>>>> shared.
>>>>>>>
>>>>>>>
>>>>>>>> So my questions are:
>>>>>>>>
>>>>>>>> - At which number of E5-26XXv4 cores does the performance for common
>>>>>>>> systems level off with a 1080Ti for your test system (GLIC)?
>>>>>>>>
>>>>>>>> - Does the answer depend strongly on the mdp settings (in particular
>>>>>>>> on
>>>>>>>> the LJ cutoff)?
>>>>>>>>
>>>>>>>
>>>>>>> Longer LJ cutoff (e.g. from different forcefields) will certainly
>>>>>>> require
>>>>>>> more non-bonded work, so then fewer CPU cores would be needed to do
>>>>>>> the
>>>>>>> remaining non-offloaded work. However any sweet spot for a particular
>>>>>>> .tpr
>>>>>>> would be highly dependent on other effects, such as the ratio of
>>>>>>> solvent
>>>>>>> (which typically has less LJ and simpler update) to solute, or the
>>>>>>> density
>>>>>>> of dihedral or U-B interactions. And doing pulling or FEP is very
>>>>>>> different
>>>>>>> again. The sweet spot for the next project will be elsewhere, sadly.
>>>>>>>
>>>>>>> This would help us a lot when choosing the appropriate CPU for a
>>>>>>> 1080Ti.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Many thanks for any suggestions,
>>>>>>>> Jochen
>>>>>>>>
>>>>>>>> --
>>>>>>>> ---------------------------------------------------
>>>>>>>> Dr. Jochen Hub
>>>>>>>> Computational Molecular Biophysics Group
>>>>>>>> Institute for Microbiology and Genetics
>>>>>>>> Georg-August-University of Göttingen
>>>>>>>> Justus-von-Liebig-Weg 11, 37077 Göttingen, Germany
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> <https://maps.google.com/?q=Justus-von-Liebig-Weg+11,+37077+G%C3%B6ttingen,+Germany&entry=gmail&source=g>
>>>>>>>> .
>>>>>>>> Phone: +49-551-39-14189 <+49%20551%203914189>
>>>>>>>> http://cmb.bio.uni-goettingen.de/
>>>>>>>> ---------------------------------------------------
>>>>>>>> --
>>>>>>>> Gromacs Users mailing list
>>>>>>>>
>>>>>>>> * Please search the archive at
>>>>>>>> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
>>>>>>>> posting!
>>>>>>>>
>>>>>>>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>>>>>>>>
>>>>>>>> * For (un)subscribe requests visit
>>>>>>>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users
>>>>>>>> or
>>>>>>>> send a mail to gmx-users-request at gromacs.org.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Gromacs Users mailing list
>>>>>>>
>>>>>>> * Please search the archive at
>>>>>>> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
>>>>>>> posting!
>>>>>>>
>>>>>>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>>>>>>>
>>>>>>> * For (un)subscribe requests visit
>>>>>>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
>>>>>>> send a mail to gmx-users-request at gromacs.org.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> ---------------------------------------------------
>>>>> Dr. Jochen Hub
>>>>> Computational Molecular Biophysics Group
>>>>> Institute for Microbiology and Genetics
>>>>> Georg-August-University of Göttingen
>>>>> Justus-von-Liebig-Weg 11, 37077 Göttingen, Germany.
>>>>> Phone: +49-551-39-14189
>>>>>
>>>>> http://cmb.bio.uni-goettingen.de/
>>>>> ---------------------------------------------------
>>>>> --
>>>>> Gromacs Users mailing list
>>>>>
>>>>> * Please search the archive at
>>>>> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
>>>>> posting!
>>>>>
>>>>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>>>>>
>>>>> * For (un)subscribe requests visit
>>>>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
>>>>> send a
>>>>> mail to gmx-users-request at gromacs.org.
>>>
>>>
>>>
>>> --
>>> ---------------------------------------------------
>>> Dr. Jochen Hub
>>> Computational Molecular Biophysics Group
>>> Institute for Microbiology and Genetics
>>> Georg-August-University of Göttingen
>>> Justus-von-Liebig-Weg 11, 37077 Göttingen, Germany.
>>> Phone: +49-551-39-14189
>>> http://cmb.bio.uni-goettingen.de/
>>> ---------------------------------------------------
>>> --
>>> Gromacs Users mailing list
>>>
>>> * Please search the archive at
>>> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
>>> posting!
>>>
>>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>>>
>>> * For (un)subscribe requests visit
>>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
>>> send a
>>> mail to gmx-users-request at gromacs.org.
>
>
> --
> ---------------------------------------------------
> Dr. Jochen Hub
> Computational Molecular Biophysics Group
> Institute for Microbiology and Genetics
> Georg-August-University of Göttingen
> Justus-von-Liebig-Weg 11, 37077 Göttingen, Germany.
> Phone: +49-551-39-14189
> http://cmb.bio.uni-goettingen.de/
> ---------------------------------------------------
> --
> Gromacs Users mailing list
>
> * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a
> mail to gmx-users-request at gromacs.org.