8 ms·
A “really high-capacity x86 machine” is really just a machine with a lot of CPUs and memory. It is not a machine that can feed each one of those CPUs, or DMA to
by derefr 7y ago
A “really high-capacity x86 machine” is really just a machine with a lot of CPUs and memory. It is not a machine that can feed each one of those CPUs, or DMA to each one of those memories, from separate network or disk IO sources, all at once. Basically, there’s only one PCI-e bus, and it’s a bottleneck. That’s why, for IO-intensive operations like search indexing, you don’t scale x86 vertically, but rather horizontally, with map-reduce strategies: you need each core and each DIMM of memory to have a commensurate IO bandwidth available to it. And that’s also why a 64-vCPU VM on shared hosting will always underperform a 64-core dedicated instance from the same provider: with the dedicated single-tenant host machine, you’re not fighting over an “oversubscribed” PCI-e bus.
Mainframes are built differently, such that each CPU and NUMA zone of memory has its own (virtual) private bus to its own (dynamically) dedicated IO peripherals—it’s like a little circuit-switched network with CPUs “placing calls” to peripherals. And, because of this, mainframes do let you just scale your problem vertically, without IO bandwidth limitations ever getting in the way. They function less like like one computer, and more like a Mosix cluster that all happens to be running on a common backplane. But, unlike Mosix and its ilk, the DC only sees one SNMP node with one set of traps, and the whole thing provides hotswap robustness against itself rather than that robustness being partitioned per machine.
- rbanffy 7y agoThe other day I described a modern mainframe as something that resembles a cluster where CPUs are connected through very high-speed low-latency buses (to the point they can behave like a single-image NUMA machine or be partitioned into multiple smaller ones) connected to multiple networks of specialized computers each running their own tiny specialized OSs that manage IO over different buses. It's common for them to have hundreds of PCI-e buses (and cards connected to storage, communications, other mainframes - for failover or clustering - etc). An x86 server is a computer built around its CPU. A mainframe is a computer built around its buses.
- 0815test 7y ago"Single system image" is a real approach to clustering, if a bit of an experimental one. There used to be a variety of toolsets attempting to provide this functionality under Linux (e.g. OpenMosix, OpenSSI) but they all seem to have bitrotted by now.
- rbanffy 7y agoI almost forgot about those. I'll have to re-read a couple papers.