6 ms·

This is a fascinating bit of nerdery. Would someone be willing to explain what mainframes are used for in a modern context, and how they compare to a really hig

This is a fascinating bit of nerdery. Would someone be willing to explain what mainframes are used for in a modern context, and how they compare to a really high-capacity x86 machine? I know that one difference is that you can hot-swap pretty much any component on a mainframe, but would be very curious to learn more about these beasts.

vbezhenar 7y ago
AFAIK they have incredible redundancy. Think about RAID-1 but for all components, so one CPU might fail and your software won't even notice that.
- mikehollinger 7y ago
  It goes a bit deeper than that. I work on POWER which is a sibling to the z platform. They share some common architecture and microarchitectural elements. To get a sense, have a look at the E980’s Reliability, Availability, and Serviceability Manual. [1] that systems NOT a mainframe - but it borrows ideas from it. Even in that system all levels of it are designed for failure, from Customer replaceable units like fans or power supplies to certain busses have extra lanes or wires for parity or spares depending on the design. Internal to the processor logic there’s a large number of internal checkers that watch the architected state of the machine and notice when something abnormal happens. For several things the processor core will unwind its state and retry an op, and if it proceeds, flag itself as having had a “recoverable” error, which usually triggers a service action. If the system can’t recover it halts either the core or the entire platform, and a dump happens from the service processor which then offloads it to the management console for diagnosis. Z boxes do all of the above - and more. There’s pictures floating around the office of a system that went through and earthquake, fell over (someone didn’t buy the optional earthquake bracing for the rack) and as it fell, it domino’d over along with several other racks. It still running when the techs went to inspect the data center, though it did turn on its “service me” attention light. ;) [1] https://www.ibm.com/downloads/cas/2RJYYJML https://www.ibm.com/downloads/cas/2RJYYJML
ofrzeta 7y ago
"As late as 2017, IBM reported 92 of the top 100 banks still used mainframes for their core businesses." https://www.tpr.org/post/how-cobol-still-powers-global-economy-60-years-old https://www.tpr.org/post/how-cobol-still-powers-global-econo...
- bubblethink 7y ago
  But why ? Other than inertia and legacy reasons that is. Is there some non partisan resource on this ? A paper or a neutral blog ? Whenever you try to research something like this, you end up on either IBM pages or pages by mainframe enthusiasts. I want to know exactly what you are buying compared to a regular high availability database like setup in the cloud.
  - PebblesHD 7y ago
    I work fairly regularly with our mainframe team, and the general reasoning boils down to massive parallelism and consistency when processing critical data. The number of rows our mainframe can process and write with perfect accuracy is leagues ahead of what similarly priced commodity hardware could achieve. There’s a reason the world still pretty much runs on COBOL/Natural and a hardware design from the 70s, it worked well then and has worked well ever since. Slightly related but I’d describe most of my role being getting data from our micro-services down into the mainframe in a format it understands so we can do ‘something’ with it.
    - bubblethink 7y ago
      Interesting. So something like google bigquery (or any of the various saas offerings by others) would not be a good match for this problem space ? For massive parallelism, we have GPUs, accelerators, and database products built on these things (admittedly, still somewhat new). By consistency, do you mean tolerance to transient errors and the like ? i.e. Beyond whatever you get with ECC ? Or are you talking about memory consistency models ? I am just constantly surprised that on the one hand FAANG+ which run so much of the internet economy do not use mainframes, and yet banks do.
lboc 7y ago
A couple of links here: https://www.ibm.com/support/knowledgecenter/zosbasics/com.ibm.zos.zcourses/zcourses_MFHWinternals.pdf https://www.ibm.com/support/knowledgecenter/zosbasics/com.ib... http://www.redbooks.ibm.com/redpapers/pdfs/redp5346.pdf http://www.redbooks.ibm.com/redpapers/pdfs/redp5346.pdf Plus one that I seem to be posting on a monthly basis these days: https://www.redbooks.ibm.com/redbooks/pdfs/sg246366.pdf https://www.redbooks.ibm.com/redbooks/pdfs/sg246366.pdf Here's one I just found, and will be reading tonight I think: http://www.redbooks.ibm.com/redbooks/pdfs/sg248550.pdf http://www.redbooks.ibm.com/redbooks/pdfs/sg248550.pdf Enjoy!
- idlewords 7y ago
  Thank you!
  - lboc 7y ago
    No problem :) If you're really keen, the base z/OS docs can be found here when you've got a spare couple of decades: https://www-01.ibm.com/servers/resourcelink/svc00100.nsf/pages/zOSV2R3Library?OpenDocument https://www-01.ibm.com/servers/resourcelink/svc00100.nsf/pag...
    - rbanffy 7y ago
      As someone who approached this problem before, the document called "Principles of Operation" IS NOT the correct entry point.
      - lboc 7y ago
        Ha, yeah should probably have mentioned that :)
znpy 7y ago
I have interviewed at one of the biggest retailers in Italy (Esselunga, a retailer in groceries and pretty much all the related things). While they had they own datacenters (two of them, hosting about ~2k physical x86 server machines and many more virtual machines) all of their retail shop inventory and sales processing ran on IBM SystemZ mainframes. They had more than one of them, and were in the process of buying a new one. What they told me is basically they have a 24/7 operation with no downtime accepted. They also need to process a huge number of orders (check-out at the grocery stores, basically) in real time. Their inventory systems were not very different from the original ones written in the late 70ies except for some evolutions along the way. They also told me about that time when they switched from an old mainframe to a new one, during the business day, in the middle of the operations, with no downtime or service interruption. So basically what I got from that is that the benefits are: - write once, run forever code - 100% uptime (not "five nines", literally 100% uptime) - a reference environment that you can rely on (no runtime change, everything that works today will keep workin in 30 years) - value added consulting from IBM - a complete set of tools (runtime, database, primitives, libraries etc)
- 0815test 7y ago
  > - 100% uptime (not "five nines", literally 100% uptime) There's literally no such thing as 100% uptime. You can add more "nines", meaning that disruptions become exponentially less likely, but that's still very different from "100%".
  - opless 7y ago
    Oh you've probably not heard of Non-Stop environments for example: https://en.wikipedia.org/wiki/Tandem_Computers#Tandem_NonStop_(TNS)_stack_machines https://en.wikipedia.org/wiki/Tandem_Computers#Tandem_NonSto...
    - dfox 7y ago
      I remember that in late 90's, NonStop was marketed as having some ludicrous (7?) number of nines of uptime, but not 100%.
derefr 7y ago
A “really high-capacity x86 machine” is really just a machine with a lot of CPUs and memory. It is not a machine that can feed each one of those CPUs, or DMA to each one of those memories, from separate network or disk IO sources, all at once. Basically, there’s only one PCI-e bus, and it’s a bottleneck. That’s why, for IO-intensive operations like search indexing, you don’t scale x86 vertically, but rather horizontally, with map-reduce strategies: you need each core and each DIMM of memory to have a commensurate IO bandwidth available to it. And that’s also why a 64-vCPU VM on shared hosting will always underperform a 64-core dedicated instance from the same provider: with the dedicated single-tenant host machine, you’re not fighting over an “oversubscribed” PCI-e bus. Mainframes are built differently, such that each CPU and NUMA zone of memory has its own (virtual) private bus to its own (dynamically) dedicated IO peripherals—it’s like a little circuit-switched network with CPUs “placing calls” to peripherals. And, because of this, mainframes do let you just scale your problem vertically, without IO bandwidth limitations ever getting in the way. They function less like like one computer, and more like a Mosix cluster that all happens to be running on a common backplane. But, unlike Mosix and its ilk, the DC only sees one SNMP node with one set of traps, and the whole thing provides hotswap robustness against itself rather than that robustness being partitioned per machine.
- rbanffy 7y ago
  The other day I described a modern mainframe as something that resembles a cluster where CPUs are connected through very high-speed low-latency buses (to the point they can behave like a single-image NUMA machine or be partitioned into multiple smaller ones) connected to multiple networks of specialized computers each running their own tiny specialized OSs that manage IO over different buses. It's common for them to have hundreds of PCI-e buses (and cards connected to storage, communications, other mainframes - for failover or clustering - etc). An x86 server is a computer built around its CPU. A mainframe is a computer built around its buses.
  - 0815test 7y ago
    "Single system image" is a real approach to clustering, if a bit of an experimental one. There used to be a variety of toolsets attempting to provide this functionality under Linux (e.g. OpenMosix, OpenSSI) but they all seem to have bitrotted by now.
nickpsecurity 7y ago
My favorite advantage is Channel I/O: https://en.wikipedia.org/wiki/Channel_I/O https://en.wikipedia.org/wiki/Channel_I/O That difference a big part of why our PC's and servers have low utilization while mainframes have massive utilization/throughput. There's designed in embedded space copying it, too, where they have a main processor and a cheaper one for I/O.

vbezhenar 7y ago

AFAIK they have incredible redundancy. Think about RAID-1 but for all components, so one CPU might fail and your software won't even notice that.

mikehollinger 7y ago
It goes a bit deeper than that. I work on POWER which is a sibling to the z platform. They share some common architecture and microarchitectural elements. To get a sense, have a look at the E980’s Reliability, Availability, and Serviceability Manual. [1] that systems NOT a mainframe - but it borrows ideas from it. Even in that system all levels of it are designed for failure, from Customer replaceable units like fans or power supplies to certain busses have extra lanes or wires for parity or spares depending on the design. Internal to the processor logic there’s a large number of internal checkers that watch the architected state of the machine and notice when something abnormal happens. For several things the processor core will unwind its state and retry an op, and if it proceeds, flag itself as having had a “recoverable” error, which usually triggers a service action. If the system can’t recover it halts either the core or the entire platform, and a dump happens from the service processor which then offloads it to the management console for diagnosis. Z boxes do all of the above - and more. There’s pictures floating around the office of a system that went through and earthquake, fell over (someone didn’t buy the optional earthquake bracing for the rack) and as it fell, it domino’d over along with several other racks. It still running when the techs went to inspect the data center, though it did turn on its “service me” attention light. ;) [1] https://www.ibm.com/downloads/cas/2RJYYJML https://www.ibm.com/downloads/cas/2RJYYJML

ofrzeta 7y ago

"As late as 2017, IBM reported 92 of the top 100 banks still used mainframes for their core businesses." https://www.tpr.org/post/how-cobol-still-powers-global-economy-60-years-old https://www.tpr.org/post/how-cobol-still-powers-global-econo...

bubblethink 7y ago
But why ? Other than inertia and legacy reasons that is. Is there some non partisan resource on this ? A paper or a neutral blog ? Whenever you try to research something like this, you end up on either IBM pages or pages by mainframe enthusiasts. I want to know exactly what you are buying compared to a regular high availability database like setup in the cloud.
- PebblesHD 7y ago
  I work fairly regularly with our mainframe team, and the general reasoning boils down to massive parallelism and consistency when processing critical data. The number of rows our mainframe can process and write with perfect accuracy is leagues ahead of what similarly priced commodity hardware could achieve. There’s a reason the world still pretty much runs on COBOL/Natural and a hardware design from the 70s, it worked well then and has worked well ever since. Slightly related but I’d describe most of my role being getting data from our micro-services down into the mainframe in a format it understands so we can do ‘something’ with it.
  - bubblethink 7y ago
    Interesting. So something like google bigquery (or any of the various saas offerings by others) would not be a good match for this problem space ? For massive parallelism, we have GPUs, accelerators, and database products built on these things (admittedly, still somewhat new). By consistency, do you mean tolerance to transient errors and the like ? i.e. Beyond whatever you get with ECC ? Or are you talking about memory consistency models ? I am just constantly surprised that on the one hand FAANG+ which run so much of the internet economy do not use mainframes, and yet banks do.

lboc 7y ago

A couple of links here: https://www.ibm.com/support/knowledgecenter/zosbasics/com.ibm.zos.zcourses/zcourses_MFHWinternals.pdf https://www.ibm.com/support/knowledgecenter/zosbasics/com.ib... http://www.redbooks.ibm.com/redpapers/pdfs/redp5346.pdf http://www.redbooks.ibm.com/redpapers/pdfs/redp5346.pdf Plus one that I seem to be posting on a monthly basis these days: https://www.redbooks.ibm.com/redbooks/pdfs/sg246366.pdf https://www.redbooks.ibm.com/redbooks/pdfs/sg246366.pdf Here's one I just found, and will be reading tonight I think: http://www.redbooks.ibm.com/redbooks/pdfs/sg248550.pdf http://www.redbooks.ibm.com/redbooks/pdfs/sg248550.pdf Enjoy!

idlewords 7y ago
Thank you!
- lboc 7y ago
  No problem :) If you're really keen, the base z/OS docs can be found here when you've got a spare couple of decades: https://www-01.ibm.com/servers/resourcelink/svc00100.nsf/pages/zOSV2R3Library?OpenDocument https://www-01.ibm.com/servers/resourcelink/svc00100.nsf/pag...
  - rbanffy 7y ago
    As someone who approached this problem before, the document called "Principles of Operation" IS NOT the correct entry point.
    - lboc 7y ago
      Ha, yeah should probably have mentioned that :)

znpy 7y ago

I have interviewed at one of the biggest retailers in Italy (Esselunga, a retailer in groceries and pretty much all the related things). While they had they own datacenters (two of them, hosting about ~2k physical x86 server machines and many more virtual machines) all of their retail shop inventory and sales processing ran on IBM SystemZ mainframes. They had more than one of them, and were in the process of buying a new one. What they told me is basically they have a 24/7 operation with no downtime accepted. They also need to process a huge number of orders (check-out at the grocery stores, basically) in real time. Their inventory systems were not very different from the original ones written in the late 70ies except for some evolutions along the way. They also told me about that time when they switched from an old mainframe to a new one, during the business day, in the middle of the operations, with no downtime or service interruption. So basically what I got from that is that the benefits are: - write once, run forever code - 100% uptime (not "five nines", literally 100% uptime) - a reference environment that you can rely on (no runtime change, everything that works today will keep workin in 30 years) - value added consulting from IBM - a complete set of tools (runtime, database, primitives, libraries etc)

0815test 7y ago
> - 100% uptime (not "five nines", literally 100% uptime) There's literally no such thing as 100% uptime. You can add more "nines", meaning that disruptions become exponentially less likely, but that's still very different from "100%".
- opless 7y ago
  Oh you've probably not heard of Non-Stop environments for example: https://en.wikipedia.org/wiki/Tandem_Computers#Tandem_NonStop_(TNS)_stack_machines https://en.wikipedia.org/wiki/Tandem_Computers#Tandem_NonSto...
  - dfox 7y ago
    I remember that in late 90's, NonStop was marketed as having some ludicrous (7?) number of nines of uptime, but not 100%.

derefr 7y ago

A “really high-capacity x86 machine” is really just a machine with a lot of CPUs and memory. It is not a machine that can feed each one of those CPUs, or DMA to each one of those memories, from separate network or disk IO sources, all at once. Basically, there’s only one PCI-e bus, and it’s a bottleneck. That’s why, for IO-intensive operations like search indexing, you don’t scale x86 vertically, but rather horizontally, with map-reduce strategies: you need each core and each DIMM of memory to have a commensurate IO bandwidth available to it. And that’s also why a 64-vCPU VM on shared hosting will always underperform a 64-core dedicated instance from the same provider: with the dedicated single-tenant host machine, you’re not fighting over an “oversubscribed” PCI-e bus. Mainframes are built differently, such that each CPU and NUMA zone of memory has its own (virtual) private bus to its own (dynamically) dedicated IO peripherals—it’s like a little circuit-switched network with CPUs “placing calls” to peripherals. And, because of this, mainframes do let you just scale your problem vertically, without IO bandwidth limitations ever getting in the way. They function less like like one computer, and more like a Mosix cluster that all happens to be running on a common backplane. But, unlike Mosix and its ilk, the DC only sees one SNMP node with one set of traps, and the whole thing provides hotswap robustness against itself rather than that robustness being partitioned per machine.

rbanffy 7y ago
The other day I described a modern mainframe as something that resembles a cluster where CPUs are connected through very high-speed low-latency buses (to the point they can behave like a single-image NUMA machine or be partitioned into multiple smaller ones) connected to multiple networks of specialized computers each running their own tiny specialized OSs that manage IO over different buses. It's common for them to have hundreds of PCI-e buses (and cards connected to storage, communications, other mainframes - for failover or clustering - etc). An x86 server is a computer built around its CPU. A mainframe is a computer built around its buses.
- 0815test 7y ago
  "Single system image" is a real approach to clustering, if a bit of an experimental one. There used to be a variety of toolsets attempting to provide this functionality under Linux (e.g. OpenMosix, OpenSSI) but they all seem to have bitrotted by now.

nickpsecurity 7y ago

My favorite advantage is Channel I/O: https://en.wikipedia.org/wiki/Channel_I/O https://en.wikipedia.org/wiki/Channel_I/O That difference a big part of why our PC's and servers have low utilization while mainframes have massive utilization/throughput. There's designed in embedded space copying it, too, where they have a main processor and a cheaper one for I/O.