Minor nitpick: given their huge focus on memory size, they also have to care a lot about memory bandwidth. That means they want Xeons with fewer cores per CPU,

Minor nitpick: given their huge focus on memory size, they also have to care a lot about memory bandwidth. That means they want Xeons with fewer cores per CPU, since each CPU (socket) has just 4 memory channels. Furthermore, if they can get away with just 4 DIMMs per CPU, they avoid the ~20% memory throughput performance penalty from memory channel interleaving. Crunching the numbers, the thing that makes most sense is 128 GB per socket, with 10 core CPUs in that, giving 2500 CPUs, probably dual socket nodes so 1250 nodes. (I'm assuming 2,500 cores above is a typo.) So each CPU would talk to 4x 32 GB DDR4 DIMMs, which are indeed available. This is also just speculation, but I couldn't see any other options giving as good performance. Given that this speculation is correct, it's hard to see how this is much different from existing HPC hardware tailored to high-memory applications. As for what your musings on interconnect latency: I believe you are correct that you can't really run with the multithreaded paradigm once you start computing at significant physical scales. Perhaps they want to move to something MPI-like for their applications?