4 ms·
No, single system image is really expensive (see Superdome Flex) and most workloads can't justify that cost.
by wmf 5y ago
No, single system image is really expensive (see Superdome Flex) and most workloads can't justify that cost.
- zozbot234 5y agoCould one reimplement SSI at the OS layer, similar to existing distributed OS's? Distributed shared memory is usually dismissed because of the overhead involved in typical scenarios, but this kind of hardware platform might make it feasible.
- wmf 5y agoThere was Mosix at the OS layer in the 1990s and Virtual Iron at the hypervisor layer in the aughts. I think the cost and performance of software SSI just doesn't intersect with demand anywhere.
- e12e 5y agoPlan9 on an oxide rack might get you close to an SSI work-a-like - but I think it's unlikely it'd be a practical use of the hw?
- zozbot234 5y agoAIUI, Plan9 is not quite fully SSI. It is a distributed OS, and gets quite close, but it's missing support for distributed memory (i.e. software-implemented shared memory exposing a single global address space); it also does not do process checkpointing and auto-migration, without which you don't really have a "global" system image.
- rbanffy 5y agoMosix and VirtualIron worked at a time 1GBps ethernet was in its infancy. Today 10GBps are consumer grade and 40GBps can go over Cat 8 copper, roughly equivalent to a DDR4-4000 channel. Not great, but this is near COTS hardware. They can do significantly better than that.
- wmf 5y agoUh no, DDR4-4000 (which servers can't use BTW) is ~256 Gigabits per second. Latency is also a killer; optimized Infiniband is ~1 us which is 10x slower than local RAM at ~100 ns.
- rbanffy 5y agoSorry. I wrote GBps when I should Gbps and I got myself fooled in the process. We can somewhat mitigate the latencies with good caches. The overall machine will suffer from a bad case of NUMA ,but would still behave better than a cluster.
- e12e 5y agoAftermarket oxide rack running DragonFlyBSD, anyone ? ;)
- rbanffy 5y agoThe OS can still do cost-based memory allocation considering the latencies of going between nodes. These Milan chips have tons of memory controllers for local memory and compute nodes can allocate all those PCIe channels to talk to a shared memory module (IBM's OMI goes in that direction - a little bit of extra latency, but lots of bandwidth and ability to go a little bit further than DDR4/5 can go). I think the bigger POWER9 boxes do this kind of thing. Migrating processes to off-board cores is silly in this case, but core/socket/drawer pinning can go a long way towards making this seamless while enabling applications that wouldn't be feasible in more mundane boxes.
- zozbot234 5y ago> The OS can still do cost-based memory allocation considering the latencies of going between nodes. That's a rather seamless extension of what OS's have to do already in order to deal with NUMA. Pinning can definitely be worthwhile since the default pointless shuttling of workloads across cores is already killing performance on existing NUMA platforms. But that could be addressed more elegantly by adjusting some tuning knobs and still allowing for migration in rare cases.