One of the issues we encountered designing a small virtualization farm was that as you add more DIMMs to the system, the clock speed of the RAM had to go down - addressing more addresses over a limited bandwidth meant ou traded of speed for capacity...

we chose capacity - and lower speed, lower cost RAM - which worked out for our use case (a few large high capacity VM's). If we had been planning one of the bioinformatics grid compute modes, it would have been a different story...

On 20 May 2018 at 01:17, Russell Coker via luv-main <luv-main@luv.asn.au> wrote:
https://en.wikipedia.org/wiki/DDR4_SDRAM

At the LUV meeting today someone asked me how servers can have so much RAM. 
Firstly the above page is worth reading, DDR3 (which is in most desktop PCs in
use now) can have a maximum of 16G per DIMM, some PCs only have 2 slots, 4 is
very common, and 6 is reasonably common for high end systems (I own a couple
of those).  6*16G=96G as the theoretical maximum in a high-end DDR3 desktop
system and 6*64G=384G as the theoretical maximum of a DDR4 desktop, of course
the BIOS or chipset might not support so much.

https://en.wikipedia.org/wiki/List_of_Intel_chipsets

The above page lists Intel chipsets and gives the maximum RAM supported.  A
lot of the Core 2 family were limited to 4G of address space, NB that's NOT 4G
of RAM, that's 4G including address space for video cards etc - so 3.25G was a
common amount of usable RAM on such systems.  The Wikipedia page doesn't give
information on the RAM limits of the i3/i5/i7 systems.

The DDR4 Wikipedia page says that one of the benefits of DDR4 is a maximum
module size of 64G.  A quick check of my favorite PC parts store showed that
they don't sell DDR3 larger than 8G modules or DDR4 larger than 16G (preumably
I could get larger via mail order).  If I got a motherboard that took 6*DDR4
DIMMs then I could have 96G of RAM, but that would cost me 6*$295=$1770 so I'm
not about to do it.

http://www.dell.com/en-au/work/shop/povw/poweredge-t640

The Dell page for the PowerEdge T640 says that it has 24*DDR4 slots for up to
3TB of RAM with a caveat that the 128G modules aren't available yet (as an
aside it seems like the DDR4 Wikipedia page needs an update in this regard).

https://en.wikipedia.org/wiki/Fully_Buffered_DIMM

Threre are significant engineering issues related to supporting large numbers
of DIMM sockets.  The above Wikipedia page is a good place to start if you
want to learn about how server RAM is different from (and incompatible to)
desktop RAM because of such issues.

https://en.wikipedia.org/wiki/ECC_memory
https://en.wikipedia.org/wiki/Hamming_code

Another difference with server RAM is the use of Error Correcting Codes (see
the above Wikipedia pages).  Server RAM has extra bits in each word to store
codes that allow single bit errors to be detected and corrected and double bit
errors to be detected.  It's a really good feature to have if you don't want
your data to be corrupted.  It's also supported in high-end workstation
systems and some systems have support for both ECC and non-ECC RAM.  ECC RAM
is usually "buffered" and won't work in systems that take "unbuffered" RAM (IE
all the desktop systems that don't use ECC RAM).  There is unbuffered ECC RAM
for low-end server systems.  Apart from buffered vs unbuffered there's
apparently no reason why ECC RAM shouldn't work in non-ECC systems, although
when I tried this back in the P4 days (before buffered RAM was invented) it
didn't work.

One of the more dedicated members of this list got a free server system from
LUV and uses it as his personal workstation.  It has something like 96G of RAM
but makes more noise than most people want in the same building they are in. 
There's no reason why you couldn't design a system with 24 DIMM sockets that
doesn't sound like an aircraft taking off, but most people who want so much
RAM have a soundproofed server room.

--
My Main Blog         http://etbe.coker.com.au/
My Documents Blog    http://doc.coker.com.au/



_______________________________________________
luv-main mailing list
luv-main@luv.asn.au
https://lists.luv.asn.au/cgi-bin/mailman/listinfo/luv-main



--
Dr Paul van den Bergen