4 ms·

There is something like live patching. One reason mainframes and micros are still around us, is that you can change almost everything between hardware and soft

by pjmlp 6mo ago

There is something like live patching. One reason mainframes and micros are still around us, is that you can change almost everything between hardware and software without downtime. It is also available in commercial surviving UNIXes, and as paid for feature in some Linux distros, although not to the extent that those grandparent systems are capable of.

Scramblejams 6mo ago
I’ve long wanted that amazing uptime and virtualization and huge I/O and all that cool stuff mainframes offered, but on the desktop or in the closet, with modern CPUs. I think I’m gonna hafta keep waiting...
silvestrov 6mo ago
A Danish bank found out that this can bite you in the ass. When you hotpatch the system for years then you have no idea if the system can boot up or it will fail somewhere in the booting process. i.e. you can only trust what you regularly test.
- pjmlp 6mo ago
  Interesting, it there any public info on the case? Not doubting it, only curious about some kind of postmorten.
  - silvestrov 6mo ago
    In Danish: https://danskebank.com/da/news-og-insights/nyhedsarkiv/press-releases/2003/pm03042003 https://danskebank.com/da/news-og-insights/nyhedsarkiv/press... or translated: https://danskebank-com.translate.goog/da/news-og-insights/nyhedsarkiv/press-releases/2003/pm03042003?_x_tr_sl=da&_x_tr_tl=en&_x_tr_hl=en-US&_x_tr_pto=wapp https://danskebank-com.translate.goog/da/news-og-insights/ny... TLDR: power supply failed completely and DB2 failed running recovery operations due to multiple old/existing software bugs.
    - pjmlp 6mo ago
      Thanks for hunting it down.
- Suzuran 6mo ago
  Mainframes can LPAR dynamically. When you want to test if your production system will IPL cleanly, you clone your production environment to an isolated LPAR and IPL it. No impact to production and you get your test.
- dredmorbius 6mo ago
  US telcos as well. There were several switch failures in the 1980s / 1990s in which systems which had been upgraded in place without a full restart failed. (IIRC, one burnt down, literally.) Engineers were uncertain as to whether or not a cold-boot restart was even possible. Account concerning an AT&T system upgrade sourcing Risks Digest (Vol 9, Issue 62, February 26, 1990) by the recently deceased Peter G. Neumann: <https://telephoneworld.org/landline-telephone-history/the-crash-of-the-att-network-in-1990/ https://telephoneworld.org/landline-telephone-history/the-cr...>.
ErroneousBosh 6mo ago
> One reason mainframes and micros are still around us, is that you can change almost everything between hardware and software without downtime. We have some Sun V880s at work and I'm fairly sure the only part you cannot change with the power on and system running is the motherboard itself. And I would not be surprised if some ex-Sun Gandalf Beard "well akshully"s this comment.
- linksnapzz 6mo ago
  Hot swapping the failed half of a bonded NIC pair on a v880 was a treat…
  - ErroneousBosh 6mo ago
    I come from an era when unplugging the RAM pack could blow every chip on the ZX80's board, so hot-swappable PCIe cards are just absolute fucking black magic to me.
    - linksnapzz 6mo ago
      Yeah, I almost had a heart attack the first time I saw someone do a 'cfgadm unconfigure' && 'cfgadm disconnect'; then pop open the side of a prod box, press a button and pull a card out. "See, oracle's still running!" Things like that used to be how one distinguished enterprise hw & sw vs. PCs w/ delusions of grandeur.
da_chicken 6mo ago
The problem with live patching is twofold. First, you might not reload everything in memory, so it will be patched on disk but not in process. Second, you have not tested that the system can boot to a functional system. Say you have done live patching for 5 years and never rebooted, and then you have a power loss or hardware failure/upgrade that takes the system down. When you try to bring it back up, it doesn't work. Which configuration change in the past 5 years caused that? Which backup do you use? And, yeah, everything is hot swappable on VAX. Those machines also cost 6+ figures, and often require a service contract that includes a permanent on site tech.
- coldtea 6mo ago
  >First, you might not reload everything in memory, so it will be patched on disk but not in process. You design for this with generational tagged objects or something similar.
- pjmlp 6mo ago
  Yes, some things actually cost money, especially if they aren't easy to implement.
- mx7zysuj4xew 6mo ago
  Which is moot, because of the system is important enough you'll have an automatic failover to another system running on standby All this "we must reboot to test" is bullshit excuses by unqualified workers
  - z3t4 6mo ago
    Had an accidental reboot, and it could not boot. Had redundancy, but the other server had failed silently days prior. Solved it with three way redundancy and extra monitoring. Systems fail in many ways at the same time. If you do not test it, there is a chance it wont work. Controlled failure is preferred over unknowns, like rebooting once in a while just to make sure it works.
  - X0Refraction 6mo ago
    Not sure I'm following honestly. Your primary goes down and it fails over to the secondary (which becomes the primary), but if you can't boot how do you then get another secondary ready to fail over to again when the new primary inevitably fails?
mesrik 6mo ago
You should't need mainframe for 100% (or five nines if that's fine) service uptime. You can build that way cheaper with 2-3 proper clustered load balancer units, 2-3 application servers behind those and those using persistent storage (databases,ldap, files) which allow writing multiple nodes simultaneously. I used to work uni that we had few services from 2012 to 2025 my retirement with zero downtime. One time my manager with tech background tried to add PBR in hurry using WebUI and did not understand cli syntax and caused close to require reboot, but I was able to fix it from cli rolling back previous config and rebooting one unit at time. Upgrading software major version up to each unit supported level wasn't hard, upgrade node it joins back cluster, upgrade another node and it joins cluster, all done. Few times I had to fix manually config for some less important test backend servers that I had forgotten to change before upgrade. No big deal. No major outages during all that 13 years time happened. Some redirecting policy and action syntax was first hard to understand and learn like GeoIP, but I was very surprised how darn reliable and nice they to use and maintain. The LB's were (Citrix) Netscalers in clustering mode (all nodes process traffic concurrently), which allowed live update one node at time without losing any connectivity through them. That wouldn't have been possible devices in just HA mode. We had just 2 beefy units which worked very well for us, but you can have 2-32 of them in cluster and managing thousands of servers behind them if you need that. Netscalers are FreeBSD derived where quite a bit of the TCP/IP stack was rewritten adding support many some quite odd features std FreeBSD doesn't have. Much of that is IP/ethernet multicast features, PBR's, Traffic Domains (VRF's) and of many service and monitoring processes which sync cluster (or HA) and if node fails another can continue straight from there without any loss of traffic to clients being proxied. Though I think most people in this forum are familiar with with haproxy, pound and web-server software provided reverse proxying. A car analogy if previous were your fancy sport sedan Netscaler and F5 BigIP are formula F1 class cars ie. quite different beasts altogether. e: And proper LB's are not just for HTTPS etc. but very nice proxying many other protocols were they TCP, UDP or something else. We did done VPN's and something like Cisco AP'S CAPWAP (DTLS ie SSL over UDP). e: typo.
- pjmlp 6mo ago
  > You should't need mainframe for 100% (or five nines if that's fine) service uptime. Hence my second paragraph. Thanks for sharing the story.

by pjmlp 6mo ago

Scramblejams 6mo ago

I’ve long wanted that amazing uptime and virtualization and huge I/O and all that cool stuff mainframes offered, but on the desktop or in the closet, with modern CPUs. I think I’m gonna hafta keep waiting...

silvestrov 6mo ago

A Danish bank found out that this can bite you in the ass. When you hotpatch the system for years then you have no idea if the system can boot up or it will fail somewhere in the booting process. i.e. you can only trust what you regularly test.

pjmlp 6mo ago
Interesting, it there any public info on the case? Not doubting it, only curious about some kind of postmorten.
- silvestrov 6mo ago
  In Danish: https://danskebank.com/da/news-og-insights/nyhedsarkiv/press-releases/2003/pm03042003 https://danskebank.com/da/news-og-insights/nyhedsarkiv/press... or translated: https://danskebank-com.translate.goog/da/news-og-insights/nyhedsarkiv/press-releases/2003/pm03042003?_x_tr_sl=da&_x_tr_tl=en&_x_tr_hl=en-US&_x_tr_pto=wapp https://danskebank-com.translate.goog/da/news-og-insights/ny... TLDR: power supply failed completely and DB2 failed running recovery operations due to multiple old/existing software bugs.
  - pjmlp 6mo ago
    Thanks for hunting it down.
Suzuran 6mo ago
Mainframes can LPAR dynamically. When you want to test if your production system will IPL cleanly, you clone your production environment to an isolated LPAR and IPL it. No impact to production and you get your test.
dredmorbius 6mo ago
US telcos as well. There were several switch failures in the 1980s / 1990s in which systems which had been upgraded in place without a full restart failed. (IIRC, one burnt down, literally.) Engineers were uncertain as to whether or not a cold-boot restart was even possible. Account concerning an AT&T system upgrade sourcing Risks Digest (Vol 9, Issue 62, February 26, 1990) by the recently deceased Peter G. Neumann: <https://telephoneworld.org/landline-telephone-history/the-crash-of-the-att-network-in-1990/ https://telephoneworld.org/landline-telephone-history/the-cr...>.

ErroneousBosh 6mo ago

> One reason mainframes and micros are still around us, is that you can change almost everything between hardware and software without downtime. We have some Sun V880s at work and I'm fairly sure the only part you cannot change with the power on and system running is the motherboard itself. And I would not be surprised if some ex-Sun Gandalf Beard "well akshully"s this comment.

linksnapzz 6mo ago
Hot swapping the failed half of a bonded NIC pair on a v880 was a treat…
- ErroneousBosh 6mo ago
  I come from an era when unplugging the RAM pack could blow every chip on the ZX80's board, so hot-swappable PCIe cards are just absolute fucking black magic to me.
  - linksnapzz 6mo ago
    Yeah, I almost had a heart attack the first time I saw someone do a 'cfgadm unconfigure' && 'cfgadm disconnect'; then pop open the side of a prod box, press a button and pull a card out. "See, oracle's still running!" Things like that used to be how one distinguished enterprise hw & sw vs. PCs w/ delusions of grandeur.

da_chicken 6mo ago

The problem with live patching is twofold. First, you might not reload everything in memory, so it will be patched on disk but not in process. Second, you have not tested that the system can boot to a functional system. Say you have done live patching for 5 years and never rebooted, and then you have a power loss or hardware failure/upgrade that takes the system down. When you try to bring it back up, it doesn't work. Which configuration change in the past 5 years caused that? Which backup do you use? And, yeah, everything is hot swappable on VAX. Those machines also cost 6+ figures, and often require a service contract that includes a permanent on site tech.

coldtea 6mo ago
>First, you might not reload everything in memory, so it will be patched on disk but not in process. You design for this with generational tagged objects or something similar.
pjmlp 6mo ago
Yes, some things actually cost money, especially if they aren't easy to implement.
mx7zysuj4xew 6mo ago
Which is moot, because of the system is important enough you'll have an automatic failover to another system running on standby All this "we must reboot to test" is bullshit excuses by unqualified workers
- z3t4 6mo ago
  Had an accidental reboot, and it could not boot. Had redundancy, but the other server had failed silently days prior. Solved it with three way redundancy and extra monitoring. Systems fail in many ways at the same time. If you do not test it, there is a chance it wont work. Controlled failure is preferred over unknowns, like rebooting once in a while just to make sure it works.
- X0Refraction 6mo ago
  Not sure I'm following honestly. Your primary goes down and it fails over to the secondary (which becomes the primary), but if you can't boot how do you then get another secondary ready to fail over to again when the new primary inevitably fails?

mesrik 6mo ago

You should't need mainframe for 100% (or five nines if that's fine) service uptime. You can build that way cheaper with 2-3 proper clustered load balancer units, 2-3 application servers behind those and those using persistent storage (databases,ldap, files) which allow writing multiple nodes simultaneously. I used to work uni that we had few services from 2012 to 2025 my retirement with zero downtime. One time my manager with tech background tried to add PBR in hurry using WebUI and did not understand cli syntax and caused close to require reboot, but I was able to fix it from cli rolling back previous config and rebooting one unit at time. Upgrading software major version up to each unit supported level wasn't hard, upgrade node it joins back cluster, upgrade another node and it joins cluster, all done. Few times I had to fix manually config for some less important test backend servers that I had forgotten to change before upgrade. No big deal. No major outages during all that 13 years time happened. Some redirecting policy and action syntax was first hard to understand and learn like GeoIP, but I was very surprised how darn reliable and nice they to use and maintain. The LB's were (Citrix) Netscalers in clustering mode (all nodes process traffic concurrently), which allowed live update one node at time without losing any connectivity through them. That wouldn't have been possible devices in just HA mode. We had just 2 beefy units which worked very well for us, but you can have 2-32 of them in cluster and managing thousands of servers behind them if you need that. Netscalers are FreeBSD derived where quite a bit of the TCP/IP stack was rewritten adding support many some quite odd features std FreeBSD doesn't have. Much of that is IP/ethernet multicast features, PBR's, Traffic Domains (VRF's) and of many service and monitoring processes which sync cluster (or HA) and if node fails another can continue straight from there without any loss of traffic to clients being proxied. Though I think most people in this forum are familiar with with haproxy, pound and web-server software provided reverse proxying. A car analogy if previous were your fancy sport sedan Netscaler and F5 BigIP are formula F1 class cars ie. quite different beasts altogether. e: And proper LB's are not just for HTTPS etc. but very nice proxying many other protocols were they TCP, UDP or something else. We did done VPN's and something like Cisco AP'S CAPWAP (DTLS ie SSL over UDP). e: typo.

pjmlp 6mo ago
> You should't need mainframe for 100% (or five nines if that's fine) service uptime. Hence my second paragraph. Thanks for sharing the story.