6 ms·

One of my older coworkers told me that IBM used to have a travelling mainframe demo in the back of a truck. As part of this demo, they would start a big comput

One of my older coworkers told me that IBM used to have a travelling mainframe demo in the back of a truck. As part of this demo, they would start a big computation, and then reach into the mainframe and start pulling out RAM and entire CPUs straight from the sockets, while the thing kept right on computing.

bunderbunder 8y ago
FWIW, you could do the same thing with a Spark cluster. It might not handle it quite as gracefully, since you'd be taking down entire compute nodes, and probably some of the data storage along with them, but the computation would also be able to continue.
- dralley 8y ago
  Yes, but I'm guessing this predates Spark by decades. Not sure of the exact timeframe, though. In any case, it helps explain why a purse-string-holding exec would be convinced to shell out money to IBM.
- fs111 8y ago
  spark clusters still have a spof as far as I understand the architecture.
  - bunderbunder 8y ago
    Two of them, I believe - the master for the cluster as a whole, and, for any individual job, its driver. I wasn't really meaning to say that a Spark cluster is as robust as a mainframe (though maybe someone's figured out some tricks), more that I could totally see some sales folks conducting the same demo using something like Spark.
    - shub 8y ago
      If you're using Spark's built-in scheduler then the cluster manager is a SPOF. Hadoop docs say you can get active/standy ResourceManager, not that I've tried it. Spark can also use k8s and nomad to schedule executors, and those have HA modes as well. I assume Mesos does HA. You're still boned if the driver dies. I am pretty sure that the driver keeps some important state in RAM so if the node hosting it goes down you have to restart from the beginning, even if the cluster manager restarts the driver.
floatboth 8y ago
ZFS developers like to do this with hard drives :) But how did IBM do that with RAM? Do mainframes have… redundant RAM? Like literally sticks mirroring others?
- dralley 8y ago
  Apparently, they can. https://en.wikipedia.org/wiki/Redundant_array_of_independent_memory https://en.wikipedia.org/wiki/Redundant_array_of_independent... But that was only introduced in 2010, so either the timeframe I was guessing at is wrong, or I'm misremembering what component it was. Could have been disk drives maybe.
- AnIdiotOnTheNet 8y ago
  See [0] for an example of how people who are serious about fault tolerance design things. [0] https://en.wikipedia.org/wiki/Tandem_Computers https://en.wikipedia.org/wiki/Tandem_Computers
- krylon 8y ago
  I have seen one PC-based server that had redundant RAM. I never tried pulling out RAM modules, but I guess there is not much point in having a RAID1-like arrangement for RAM if the machine cannot handle random RAM modules dying without crashing the OS. And it wasn't even a super high-end system.
  - forapurpose 8y ago
    > I have seen one PC-based server that had redundant RAM Yes, I've seen this too, and on lowish-end servers.
    - snuxoll 8y ago
      Memory mirroring is a pretty common feature on basically every x86 server I've used (going back to the Core 2-based Xeon's, no experience with older platforms), though I've never tried just yanking DIMM's out and wouldn't recommend trying it.
- 8y ago
tCfD 8y ago
Were they pulling out compute resources they knew in advance were not being used or part of the active partition for that particular job? Or did they let potential customers do the pulling themselves?
kitd 8y ago
The current "funfact" is that the on/off button on a Z-series machine is only designed to be used twice.
- justinjlynn 8y ago
  IPLing a Z-Series machine is some serious dark magick.
- shub 8y ago
  Reminds me of an IBM technical bulletin I found, somehow, about 10 years ago. It advised that if you were moving from this z9 hardware configuration to that one, you would have 30 minutes of downtime. Implying that other changes required no downtime, which I inferred wasn't stated because it wasn't insanely awesome to mainframe guys, adding CPUs and RAM to a running system was just how things worked. I was in college at the time and looked at job postings for mainframe programmers because I wanted to work on that. Never found one that required less than 5-10 years experience, not then and not when I've checked every couple years after that. Too bad; I quite liked the idea of working in an ecosystem that starts from "let's make this work every single time" instead of "let's make this work well enough to keep customer complaints to a dull roar".
southern_cross 8y ago
And there are stories of IBM hardware surviving fires and floods and coming back to life after not much more than a minor cleanup and a reboot. I once personally dealt with an IBM system that got hit by lightning. It crashed and took its own sweet time coming back up but it did come back up, with no special intervention, having only lost a few RS-232 comm cards to the lightning. Meanwhile, these days you have folks running "start of the art" systems which may just roll over and die for no discernible reason. These may not easily come back up, either, if at all. That's why they have to have so many of them!

bunderbunder 8y ago

FWIW, you could do the same thing with a Spark cluster. It might not handle it quite as gracefully, since you'd be taking down entire compute nodes, and probably some of the data storage along with them, but the computation would also be able to continue.

dralley 8y ago
Yes, but I'm guessing this predates Spark by decades. Not sure of the exact timeframe, though. In any case, it helps explain why a purse-string-holding exec would be convinced to shell out money to IBM.
fs111 8y ago
spark clusters still have a spof as far as I understand the architecture.
- bunderbunder 8y ago
  Two of them, I believe - the master for the cluster as a whole, and, for any individual job, its driver. I wasn't really meaning to say that a Spark cluster is as robust as a mainframe (though maybe someone's figured out some tricks), more that I could totally see some sales folks conducting the same demo using something like Spark.
  - shub 8y ago
    If you're using Spark's built-in scheduler then the cluster manager is a SPOF. Hadoop docs say you can get active/standy ResourceManager, not that I've tried it. Spark can also use k8s and nomad to schedule executors, and those have HA modes as well. I assume Mesos does HA. You're still boned if the driver dies. I am pretty sure that the driver keeps some important state in RAM so if the node hosting it goes down you have to restart from the beginning, even if the cluster manager restarts the driver.

floatboth 8y ago

ZFS developers like to do this with hard drives :) But how did IBM do that with RAM? Do mainframes have… redundant RAM? Like literally sticks mirroring others?

dralley 8y ago
Apparently, they can. https://en.wikipedia.org/wiki/Redundant_array_of_independent_memory https://en.wikipedia.org/wiki/Redundant_array_of_independent... But that was only introduced in 2010, so either the timeframe I was guessing at is wrong, or I'm misremembering what component it was. Could have been disk drives maybe.
AnIdiotOnTheNet 8y ago
See [0] for an example of how people who are serious about fault tolerance design things. [0] https://en.wikipedia.org/wiki/Tandem_Computers https://en.wikipedia.org/wiki/Tandem_Computers
krylon 8y ago
I have seen one PC-based server that had redundant RAM. I never tried pulling out RAM modules, but I guess there is not much point in having a RAID1-like arrangement for RAM if the machine cannot handle random RAM modules dying without crashing the OS. And it wasn't even a super high-end system.
- forapurpose 8y ago
  > I have seen one PC-based server that had redundant RAM Yes, I've seen this too, and on lowish-end servers.
  - snuxoll 8y ago
    Memory mirroring is a pretty common feature on basically every x86 server I've used (going back to the Core 2-based Xeon's, no experience with older platforms), though I've never tried just yanking DIMM's out and wouldn't recommend trying it.
8y ago

tCfD 8y ago

Were they pulling out compute resources they knew in advance were not being used or part of the active partition for that particular job? Or did they let potential customers do the pulling themselves?

kitd 8y ago

The current "funfact" is that the on/off button on a Z-series machine is only designed to be used twice.

justinjlynn 8y ago
IPLing a Z-Series machine is some serious dark magick.
shub 8y ago
Reminds me of an IBM technical bulletin I found, somehow, about 10 years ago. It advised that if you were moving from this z9 hardware configuration to that one, you would have 30 minutes of downtime. Implying that other changes required no downtime, which I inferred wasn't stated because it wasn't insanely awesome to mainframe guys, adding CPUs and RAM to a running system was just how things worked. I was in college at the time and looked at job postings for mainframe programmers because I wanted to work on that. Never found one that required less than 5-10 years experience, not then and not when I've checked every couple years after that. Too bad; I quite liked the idea of working in an ecosystem that starts from "let's make this work every single time" instead of "let's make this work well enough to keep customer complaints to a dull roar".

southern_cross 8y ago

And there are stories of IBM hardware surviving fires and floods and coming back to life after not much more than a minor cleanup and a reboot. I once personally dealt with an IBM system that got hit by lightning. It crashed and took its own sweet time coming back up but it did come back up, with no special intervention, having only lost a few RS-232 comm cards to the lightning. Meanwhile, these days you have folks running "start of the art" systems which may just roll over and die for no discernible reason. These may not easily come back up, either, if at all. That's why they have to have so many of them!