5 ms·
"128GB is now starting to become a limitation. With 16 cores that is only 8GB/ core." This comment gave me pause, I feel like I'm falling out of pace with the
by mey 11y ago
"128GB is now starting to become a limitation. With 16 cores that is only 8GB/ core."
This comment gave me pause, I feel like I'm falling out of pace with the evolution of commodity server systems. We've gone from being network (10/40gbe) and disk io (fixed with SSD, PCIe NVRAM and Fiberchannel) bound in the the near past to swing towards memory capacity bound?
- xxpor 11y agoI think that's more of a reaction to the "fixes" to disk IO still not being fast enough. The more you can keep in memory, the better off you're going to be, even with the fastest SSDs in the world.
- jakub_h 11y agoIsn't 8 GB per core still reasonable? Especially if you can share indices and buffer caches across cores, for example. There must still be diminishing returns for caching.
- zymhan 11y ago"reasonable" is entirely context-dependent. Some workloads are CPU or network bound. Others are memory bound.
- Sanddancer 11y agoDepends on your dataset. Systems like Databases perform much, much better the less you have to touch a storage controller. One of the things that's going into Intel's 2017/2018 platform is the support for 6 channels of memory, and not all of that needs to be DRAM. There's a new standard, NVDIMM, that lets you use non-volatile memory as RAM, and take advantage of its much higher densities. So, if you have things like a database, you can map a much larger dataset into actual memory, instead of needing to just have the indexes in there.
- mnw21cam 11y agoI do a load of work with high throughput genetics sequencing. One of the programs that processes the data for a sample is single-threaded, but requires about 40GB of RAM. I currently have a server with 384GB RAM and 64 CPU threads, so 6GB/thread. RAM capacity is a definite bottleneck, since I can only run 9 at a time, instead of 64. It all depends.
- zymhan 11y agoSee this article: https://queue.acm.org/detail.cfm?id=2874238 https://queue.acm.org/detail.cfm?id=2874238 In a high-end server, with PCIe flash or NVRAM, you're increasingly unlikely to be IO bound.
- xxpor 11y agoThat was a great read, thanks for sharing!
- frozenport 11y agoMy takeaway was that the single JBOD approach won't work in next generation systems as a JBOD of SSD/NVME/PCI-SSDs will be greatly limited by the controller, and that infrastructure complexity will increase presumably by having many JBODs with smaller number of disks. Which is silly, because controllers can simply be designed to handle more data, and likewise they can be made more power efficient. So, far the only real limitation has been the interface, which off-course can be changed.
- zymhan 11y agoI'm not sure why that is your main take away from that article. The point is that relatively slow disk controllers designed for rotating magnetic storage are a bottleneck when you connect fast enough Flash storage. Connecting the storage via PCIe alleviates this bottleneck, to the point that you are no longer bound by the traditional constraints of small caches and large, slow permanent storage.
- Keyframe 11y agoI wonder how soon we will see the inevitable convergence of RAM and "storage". XPoint is still slower than RAM, but is a path towards unification.
- ihsw 11y agoThat is correct. NVMe-backed storage, 10 GigE-backed network IO, and 16-core CPU gets you pretty far nowadays for ~$5K USD, and RAM starts to be the limiting factor. Disk IO and network IO can be saturated pretty easily, and that kind of work usually involves a lot of RAM so here we are.
- iheartmemcache 11y agoKeep in mind, Phi offers an additional 8 GDDR controllers at 2 lanes a piece. If you really ran into that 128GB wall, you could basically use your Phi as an off-loader and make those pages addressable. Profile your program and find the fetch patterns, then chuck in a cheap Phi as a memmapper for the LRU and/or older generation stuff. (I'd imagine you can borrow a lot of the last 20 years of GC research.) It's a non-trivial (but certainly doable) task.
- thrownaway2424 11y agoWhat's going on with this 128GB number? What's the actual limit here? I thought we had 40 bits of physical memory to work with.
- toast0 11y agoThe board has 4 ram slots, and 32gb is generally available (64gb is still in 'soon' territory, I think). Xeon e5 v3 has four ram channels instead of two for the processors in the article, and can handle three dimms per channel (or at least the board in the article only haa two slots per channel). I assume there's some market segmentation happening here, but it's also a way to reduce board size and pin count.
- thrownaway2424 11y agoThat makes sense. I didn't realize these had so few channels. What I was concerned about was possibly baseless market segmentation engineering, in the same way that they elide ECC from the "Core" line of CPUs.
- ksec 11y agoSamsung has announced they had a 128GB Stacked DRAM ready soon. So 4 DIMM could fit 512GB Memory. So i guess the problem is with address space as mentioned above.
- Sanddancer 11y agoThe Xeon Ds have had a max of 128 gigs, or 37 bits of addressing, since they came out last year. For Skylake, they have bumped up the addressibility to 36 bits from 35 bits, which is where it had been for a number of processor generations. For the E5 Xeons, the amount of physically addressable RAM is at 46 bits, and has been for a couple generations. Intel's used addressible RAM for a while as one of their segmentation methods to push people towards their higher end if their usage is RAM heavy.
- baq 11y agoit's market segmentation, sure, but also every bit of address line that you don't use translates directly into watts wasted.
- fweespeech 11y agoHonestly, our production systems are 256GB RAM and 16 GB/Core. So no, I don't think so.
- oofabz 11y agoThat quote from the article makes sense if the author is thinking about virtualization. He is imagining running 16 VMs on it.