3 ms·
As Greg points out this has come up multiple times and in multiple strategies for implementation. In the early 2000's I lead the design of a memory based Ether
by ChuckMcM 9y ago
As Greg points out this has come up multiple times and in multiple strategies for implementation.
In the early 2000's I lead the design of a memory based Ethernet appliance which we called "Network Accessible Memory". The basic idea combined the fact that FPGAs were pretty cheap, you could easily build a gigabit network interface on one side and a DDR2/3 interface on the other, and with a very simple ethernet protocol have terabytes of memory basically a couple of microseconds away from a CPU as opposed to 'swap' which was milliseconds away at best. (we also simulated the architecture using a machine just exporting memory to the network)
The two biggest wins are that the memory is already 'live' when machines that might use it boot up so they don't spend time reading in state, and in a multi-processor storage subsystem you can put the buffer cache into memory without tying up any individual processor's main memory which gives you faster throughput and lower latency as you're not turning around the disk interfaces for meta data reads and mixed r/w workloads don't blow up your buffer cache.
All very cool. But all very "specialized" too.
- greglindahl 9y agoFun -- Texas Memory Systems was already shipping a product in that era that was FPGA-based. It did a variety of complicated network protocols, like Fibre Channel and eventually InfiniBand, with a pure FPGA implementation. The guy who ran the company was also an FPGA wizard. I was on a telecon with him once. https://en.wikipedia.org/wiki/Texas_Memory_Systems https://en.wikipedia.org/wiki/Texas_Memory_Systems
- ChuckMcM 9y agoYeah, I think they are cited in the patent we did. The key differences were a) no non-volatile storage at all, and b) we took 16 memory 'blades' (1GBit each) and combined them into a single 10gbit backed 'controller'. It leveraged a lot of ideas from the disk shelves at the time. It did give you 1GB/s read/write fully raid protected memory (with equivalent FEC bits in the packets to catch network corruption) to the network. I expect TMS would have gotten there eventually once they moved on from 'storage' on the network/fabric to 'memory' on the network/fabric.
- akoster 9y agoThis is super neat! Comments like this keep me coming back to HN. Thanks for sharing!
- makmanalp 9y agoSingle System Image is some cool tech in this area, where basically you're running a single operating system instance across the network, which sees each computer as just more available computing resources, and you can for example migrate processes from one computer to another. There are also database systems that use a ring of nodes connected with RDMA and optimize for that data layout, creating a sort of merry-go-round of data: http://homepages.cwi.nl/~mk/datacyclotron.pdf http://homepages.cwi.nl/~mk/datacyclotron.pdf
- ChuckMcM 9y agoThat is pretty cool, all the dates are place holders, when was it published? Tom Lyon had a startup in this space that was trying to build network accessible memory systems to solve the 'process migration' problem. That is the problem where moving a process on a Linux system takes too long relative to the benefit. Google looked at some proposals that were kinda sorta like this when they were soliciting ideas for their 'energy proportional computing' effort. Basically they were attacking the problem that the difference between an idle machine and a fully loaded machine was about 25% power usage. That isn't held against your PUE (after all the machine is 'running') but it does mean you could theoretically increase your energy efficiency by 50% if you could leave idle machines powered off and effectively bring them on line quickly enough to meet your design goals.
- makmanalp 9y agoIt seems the paper was published around 2010-2011. This is Martin Kersten, same guy who did MonetDB. I'm not sure if the idea went anywhere since it was very radical and wacky, but perhaps it's waiting for its time to come :-) The neat thing about it isn't that it /can/ work in such an architecture, it's more that it's optimized to leverage it. Their idea is that you can spin data through the ring one way, and spin requests the other way, and you can have a multi-node distributed data system while avoiding a ton of global coordination. > you could theoretically increase your energy efficiency by 50% if you could leave idle machines powered off and effectively bring them on line quickly enough to meet your design goals It's a cool pattern that I keep seeing of people trying similar ideas at different scales! Multicore processors for mobile computers also turn cores on and off to save power, and migrate and balance processes across cores, I believe. Similarly, the process migration thing could be solved at the language level rather than the OS level, which I think is what some erlang stuff does - as long as you have global coordination and can serialize process state and code, why not?
- eeks 9y agoI also worked in that space several years ago. RDMA-based paging is a very tempting (and sexy) concept. There are however several challenges that make it very impractical in commercial scenarios: 1. Availability: what happens when a node goes down ? 2. Page sharing: how should / can multiple writers be supported ? 3. Kernel overhead: how do you hook in the paging subsystem without significant overhead ? 4. Cost/performance: rare are the companies that would blindly jump into an IB network investment; and RoCE is not as fast as IB (although it may be catching up with 25/40/100 Gb cards). In the end we ended up designing a RDMA-based DHT. Besides, this kind of approach becomes less relevant nowadays with solutions like Intel Optane.
- ChuckMcM 9y agoAwesome, did write ups appear anywhere of what you learned? I would enjoy reading them. > 1. Availability: what happens when a node goes down ? Pretty typically you fail. One of the things we noted was that making a "shelf" that could get 5 9's (99.999% uptime) was a lot easier than a system that got five 9's. In the NAM design the RAM was ECC (bit protection), the individual cards were RAIDed across (card failure protection), dual power supplies and controller (no state that isn't in the memory so you could have active passive controllers with no synchronization protocol) and a very basic optimistic locking protocol (cost free on success, resync on collision which for this use case was pretty reasonable (not a lot of write contention)). > 2. Page sharing: how should / can multiple writers be supported ? The architecture was single writer, multiple readers. It was not designed (although could probably be adapted) to prevent other network nodes from reading data or snooping data reads across the network. We talked briefly about using multi-cast in a shoot down protocol to allow machines to invalidate their lock state. The idea was that nobody 'cached' these blocks so invalidating caches wasn't an issue. That said, looking at the way AMD and Intel manage the Layer1 (per core)/Layer2 (across cores) cache layers for their chips had me wondering if that would have been useful here. Generally if you write and your nonce is invalid you re-read the current nonce and re-write with the updated nonce. Also important to note that in our case it was a 'data' cache not an execution cache. > 3. Kernel overhead: how do you hook in the paging subsystem without significant overhead ? We were using Ethernet, which had lower latency and overhead than using the (SATA/IDE) layer in Linux. Not IP, Ethernet. So to fetch a 'block' (8K bytes) you'd drop the address of that block on the network pipe and the packet would go out, and the response would come back pretty much instantly. (1 - 2uS). > 4. Cost/performance: rare are the companies that would blindly jump into an IB network investment; and RoCE is not as fast as IB (although it may be catching up with 25/40/100 Gb cards). Our thought was you got a benefit just using the existing Ethernet network you had. (after all you were mostly getting rid of disk seeks which were milliseconds for Ethernet packets (microseconds) who doesn't like a 1000:1 reduction in latency :-). In our case it was 'add this box to your storage stack and get a xx% boost in throughput or yy% decrease in latency.' Where the numbers could be calculated based on your workload and your read/write mix. If the sale price had 55 points of margin then we had successfully done our job :-).