3 ms·
The big headache is not 1TB but 64TB which is the maximum physical RAM limit of the Linux kernel / x86-64 architecture. Big NUMA systems could go higher but the
by chx 10y ago
The big headache is not 1TB but 64TB which is the maximum physical RAM limit of the Linux kernel / x86-64 architecture. Big NUMA systems could go higher but they don't. Look here: https://www.sgi.com/products/servers/uv/ https://www.sgi.com/products/servers/uv/ "SGI UV 300 scales from 4 to 64 sockets with up to 64TB of shared memory in a single system." vs "SGI UV 3000 scales from 4 to 256 CPU sockets with up to 64TB of shared memory as a single system." see how 256 and 64 sockets both only support 64TB?
More ordinary servers typically stop at 6TB since one Xeon can support 1.5TB so an ordinary quad socket board typically with 96 DIMM slots can go up to 6TB with 64GB DIMMs. You can configure a machine like this at http://www.thinkmate.com/system/superserver-4048b-tr4ft http://www.thinkmate.com/system/superserver-4048b-tr4ft and see that for the relatively low price of $110K you can get a machine with 6TB of RAM.
- AaronFriel 10y ago> The big headache is not 1TB but 64TB which is the maximum physical RAM limit of the Linux kernel / x86-64 architecture. It sounds like you know what you're talking about, so I'm sure it was inadvertent that you wrote "physical RAM limit of the Linux kernel". It's primarily the x86-64 architecture and 5-level paging is coming which extends the linear address space to 57 bits (128 PiB) and the physical address space up to 52 bits (4 PiB). Still, one wonders how long it will take for that to be inadequate. https://software.intel.com/sites/default/files/managed/2b/80/5-level_paging_white_paper.pdf https://software.intel.com/sites/default/files/managed/2b/80...
- qb45 10y agoAnd Linux support for 5-level is being actively worked on since December, a quick git search shows this in 4.11: Merge 5-level page table prep from Kirill Shutemov: "Here's relatively low-risk part of 5-level paging patchset. Merging it now will make x86 5-level paging enabling in v4.12 easier.
- dom0 10y agoThere are still 8-socket systems, like the Supermicro 7088B (192 DIMMs), which support a theoretical 24 TB (with 128 GB modules). (It would be interesting to know whether this backplane-based system actually manages to achieve 4.8 GHz QPI speed (9.6 GHz symbol rate), or whether the physical aspects limit it to lower speeds. Given that processors only have four QPI links this would further increase communication overhead -- in a 8-way system some sockets are separated by two hops).
- chx 10y agoSo 128GB modules exist and available http://www.memory4less.com/crucial-128gb-ddr4-pc19200-m386aak40b40-cuc http://www.memory4less.com/crucial-128gb-ddr4-pc19200-m386aa... but at least 30% more expensive than 2*64GB: http://www.memory4less.com/samsung-64gb-ddr4-pc17000-m386a8k40bm1-cpb http://www.memory4less.com/samsung-64gb-ddr4-pc17000-m386a8k... and of course this is the cheapest vendor. Brand names will fleece http://www.memory4less.com/hp-128gb-ddr4-pc19200-809208-b21 http://www.memory4less.com/hp-128gb-ddr4-pc19200-809208-b21 you for 128GB modules while http://www.memory4less.com/hp-64gb-ddr4-pc17000-726724-b21 http://www.memory4less.com/hp-64gb-ddr4-pc17000-726724-b21 64GB is just a little more expensive.