[gmx-users] NVIDIA GTX cards in Rackable servers, how do you do it ?

Tue Feb 24 13:17:54 CET 2015

BS"D

Dear David,

  We did some tests with Gromacs and other programs on CPU's with core counts up to 16 per socket, and found that after about 12 cores jobs/threads begin to interfere with each other.  In other words there was a performace penalty when using core counts above 12.  I don't have the details in front of me, but you should  at the very least get a test machine and try running your simulations for short periods with 10, 12, 14, 16 and 18 cores in use to see how Gromacs behaves with these processors (unless someone has done these tests, and can confirm that Gromacs has no issues with 16 or 18 core cpu's).

Harry

On Feb 24, 2015, at 1:32 PM, David McGiven wrote:

Hi Szilard,

Thank you very much for your great advice.

2015-02-20 19:03 GMT+01:00 Szilárd Páll <pall.szilard at gmail.com<mailto:pall.szilard at gmail.com>>:

On Fri, Feb 20, 2015 at 2:17 PM, David McGiven <davidmcgivenn at gmail.com<mailto:davidmcgivenn at gmail.com>>
wrote:
Dear Gromacs users and developers,

We are thinking about buying a new cluster of ten or twelve 1U/2U
machines
with 2 Intel Xeon CPU's 8-12 cores each. Some of the 2600v2 or v3 series.
Not yet clear the details, we'll see.

If you can afford them get the 14/16 or 18 core v3 Haswells, those are
*really* fast, but a pair can cost as much as a decent car.

Get IVB (v2) if it saves you a decent amount of money compared to v3.
The AVX2 with FMA of the Haswell chips is great, but if you run
GROMACS with GPUs on them my guess is that a higher frequency v2 will
be more advantageous than the v3's AVX2 support. Won't swear on this
as I have not tested thoroughly.

According to an email exchange I had with Carsten Kutzner, for the kind of
simulations we would like to run (see below), lower frequency v2's give
better performance-to-price ratio.

For instance, we can get from a national reseller :

2U server (supermicro rebranded I guess)
2 x E5-2699V3 18c 2,3Ghz
64 GB DDR4
2 x GTX980 (certified for the server)
-
13.400 EUR (sans VAT)

2U server (supermicro rebranded I guess)
2 x E5-2695V2 12c 2,4 Ghz
64 GB DDR3
2 x GTX980 (certified for the server)
-
9.140 EUR (sans VAT)

Does that qualify as "saving a decent amount of money" to go for the V2 ? I
don't think so, also because we care about rack space. Less servers but
potent ones. The latests haswells are way too overpriced for us.

We want to run molecular dynamics simulations of transmembrane proteins
inside a POPC lipid bilayer, in a system with ~100000 atoms, from which
almost 1/3 correspond to water molecules and employing usual conditions
with PME for electorstatics and cutoffs for LJ interactions.

I think we'll go for the V3 version.

I've been told in this list that NVIDIA GTX offer the best
performance/price ratio for gromacs 5.0.

Yes, that is the case.

However, I am wondering ... How do you guys use the GTX cards in rackable
servers ?

GTX cards are consummer grade, for personal workstations, gaming, and so
on
and it's nearly impossible to find any servers manufacturer like HP,
Dell,
SuperMicro, etc. to certify that those cards will function properly on
their servers.

Certification can be an issue - unless you buy many and you can cut a
deal with a company. There are some companies that do certify servers,
but AFAIK most/all are US-based. I won't do public a long
advertisement here, but you can find many names if you browse NVIDIA's
GPU computing site (and as a matter of fact the AMBER GPU site is
quite helpful in this respect too).

You can consider getting vanilla server nodes and plug the GTX cards
in yourself. In general, I can recommend Supermicro, they have pretty
good value servers from 1 to 4U. The easiest is to use the latter
because GTX cards will just fit vertically, but it will be a serious
waste of rack-space.

With a bit of tinkering you may be able to get
GTX cards into 3U, but you'll either need cards with connectors on the
back or 90 deg angled 4-pin PCIE power cables. Otherwise you can only
fit the cards with PCIE raisers and I have no experience with that
setup, but I know some build denser machines with GTX cards.

Cheer,

--
Szilárd

What are your views about this ?

Thanks.

Best Regards
--
Gromacs Users mailing list

* Please search the archive at
http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
posting!

* Can't post? Read http://www.gromacs.org/Support/Mailing_Lists

* For (un)subscribe requests visit
https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
send a mail to gmx-users-request at gromacs.org<mailto:gmx-users-request at gromacs.org>.
--
Gromacs Users mailing list

* Please search the archive at
http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
posting!

* Can't post? Read http://www.gromacs.org/Support/Mailing_Lists

* For (un)subscribe requests visit
https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
send a mail to gmx-users-request at gromacs.org<mailto:gmx-users-request at gromacs.org>.

--
Gromacs Users mailing list

* Please search the archive at http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting!

* Can't post? Read http://www.gromacs.org/Support/Mailing_Lists

* For (un)subscribe requests visit
https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a mail to gmx-users-request at gromacs.org<mailto:gmx-users-request at gromacs.org>.

-------------------------------------------------------------------------

Harry M. Greenblatt

Associate Staff Scientist

Dept of Structural Biology

Weizmann Institute of Science        Phone:  972-8-934-3625

234 Herzl St.                        Facsimile:   972-8-934-4159

Rehovot, 76100

Israel

Harry.Greenblatt at weizmann.ac.il<mailto:Harry.Greenblatt at weizmann.ac.il>