7 ms·

The mainframe infrastructure is both the core feature and showstopper bug of the platform. On the plus side, it has empowered countless core business applicati

The mainframe infrastructure is both the core feature and showstopper bug of the platform. On the plus side, it has empowered countless core business applications to scale years, even decades, beyond their original development lifecycle. On the negative side, it has created a stagnant cult of mainframe “priests” who oversee a platform that most businesses would rather get rid of, if they could only figure out how. My startup was acquired by BMC software, one of the largest third party providers of mainframe software, and I also worked for EMC for a time early in my career, so I spent a lot of time hanging around the mainframe space. Two quick anecdotes: - working on a giant data center move for a big bank, there was a break room filled on a Saturday with operations people from the “open systems” (non-mainframe) and mainframe teams. I was struck by the contrast: the Unix/windows operations folks were tattooed, pierced, jeans and t-shirts, late 20s. The mainframe folks were short hair, polos and jeans, mid-50s. The two groups self segregated into clusters on opposite ends of the room, not speaking to each other. - at a BMC leadership offsite, I found myself st the dinner table with a couple of mainframe engineering and product directors. After a few drinks, we got on the subject of why more tech companies weren’t using mainframes. “They’re so reliable! Why wouldn’t you just go out and rent one from IBM, and then you don’t have to worry about uptime or stability?”. I explained that the fashion had become to build the reliability into the application and assume that the hardware was not reliable. This confused them. “That sounds like a huge pain in the ass, why would you bother dealing with that?” So I explained that the cost per compute cycle was so much cheaper that it made it worth it, plus open source, etc etc One of them kept pressing: “but Facebook could just take all the engineers they’re devoting to building reliable infrastructure and shift those people to writing customer facing code!” The other director stopped him and said, “don’t you see? They’ve already done it. There’s no reason for them to go back now. People have figured out that they don’t need mainframes to get mainframe reliability”. The other director just kept shaking his head and we moved to other subjects. This conversation happened in 2014.

dralley 8y ago
One of my older coworkers told me that IBM used to have a travelling mainframe demo in the back of a truck. As part of this demo, they would start a big computation, and then reach into the mainframe and start pulling out RAM and entire CPUs straight from the sockets, while the thing kept right on computing.
- bunderbunder 8y ago
  FWIW, you could do the same thing with a Spark cluster. It might not handle it quite as gracefully, since you'd be taking down entire compute nodes, and probably some of the data storage along with them, but the computation would also be able to continue.
  - dralley 8y ago
    Yes, but I'm guessing this predates Spark by decades. Not sure of the exact timeframe, though. In any case, it helps explain why a purse-string-holding exec would be convinced to shell out money to IBM.
  - fs111 8y ago
    spark clusters still have a spof as far as I understand the architecture.
    - bunderbunder 8y ago
      Two of them, I believe - the master for the cluster as a whole, and, for any individual job, its driver. I wasn't really meaning to say that a Spark cluster is as robust as a mainframe (though maybe someone's figured out some tricks), more that I could totally see some sales folks conducting the same demo using something like Spark.
      - shub 8y ago
        If you're using Spark's built-in scheduler then the cluster manager is a SPOF. Hadoop docs say you can get active/standy ResourceManager, not that I've tried it. Spark can also use k8s and nomad to schedule executors, and those have HA modes as well. I assume Mesos does HA. You're still boned if the driver dies. I am pretty sure that the driver keeps some important state in RAM so if the node hosting it goes down you have to restart from the beginning, even if the cluster manager restarts the driver.
- floatboth 8y ago
  ZFS developers like to do this with hard drives :) But how did IBM do that with RAM? Do mainframes have… redundant RAM? Like literally sticks mirroring others?
bunderbunder 8y ago
I think that, within the context of that story, both directors are missing the deeper issue: A company's technology decisions aren't just about the final state, they're also about getting the project bootstrapped, and all the other technology decisions between that point and the final state. So it's not just that Facebook had already done it their way, it's that, along the way, they never had any realistic alternatives that led toward a mainframe solution. Having never worked directly with a mainframe, I can't really speak to the reliability question. What I do know, though, is that I've spent my entire life living around PC hardware, working on PC hardware, etc, because that's the hardware that's designed to fit into my day-to-day life. So, if I have a new idea, and want to try it out, I'm going to reach for the hardware I already have, because it's basically free. Just like Mark Zuckerberg did. And at that point, I'm already on a path that leads inexorably toward building the reliability into the software. There's simply never going to be any point at which it makes sense for me to scrap everything I have so far and do a complete rewrite for a different platform.
- xoa 8y ago
  >I think that, within the context of that story, both directors are missing the deeper issue: A company's technology decisions aren't just about the final state, they're also about getting the project bootstrapped, and all the other technology decisions between that point and the final state. So it's not just that Facebook had already done it their way, it's that, along the way, they never had any realistic alternatives that led toward a mainframe solution. Absolutely agreed, "path dependency" is a big issue that comes up in a lot of different contexts both in the natural world and human endeavors. Just because from a high level or after the fact someone can identify a much more optimal final result doesn't mean that result was actually realistic to get to, or would be worth moving to from whatever local minima a project ended up in. I think path dependency plays a major part in what makes "disruption" possible, and in how businesses can sometimes be eaten from the bottom. A lot of companies end up in a state where they have a bunch of end point products but they haven't considered the paths necessary for users to reach that end point. Without a ramp their pipe can slowly empty.
- southern_cross 8y ago
  > There's simply never going to be any point at which it makes sense for me to scrap everything I have so far and do a complete rewrite for a different platform. Yet people still often claim that it makes sense to go to all the time and trouble and expense of moving off of reliable legacy platforms onto other platforms which aren't naturally anywhere near as reliable. Imagine that!
rodgerd 8y ago
The sales job Amazon et al did in convincing the world resilience was the job of app developers, not the system vendor, was quite impressive. And while it's true that commodity folks could stand to learn from mainframe folks - serverless looks quite a lot like a forty years on re-invention of CICS if you squint a bit - your story reminds me how profoundly true the opposite is, as well. I've been part of a team that ran Linux on s390x, and it was a great exercise in demonstrating that however reliable your mainframe, it's still a SPOF. Even if you don't have my bad luck of four catastrophic hardware failures in as many years, your one reliable box still depends on the power, network fabric, storage fabric, etc at a single site. If you want to avoid losing your business when a sparky screws up, you need... several mainframes. At which point you're spending an absolutely astronomical amount to either sysplex, or you're building resilience into your applications, just like a bitty box. And yes, so many mainframe folks are so very out of touch with the broader world. In around 2016 I had the supposed regional tech expert on z-Series system lecture me (rather snidely) on how: 1. s390 processors were 20 - 40 times more performant that Intel processors (demonstrably not true for any workload I cared about). 2. You could not do virtualisation on x86_64. It was impossible. With a straight face, in the year 2016, this guy told me, and clearly believed, that it was not possible to run heterogenous virtualised workloads on an Intel processor. Apparently tens of billions of Jeff Bezos' net worth literally did not exist for him. There was more, but it was like talking to someone who had been frozen in a block of ice since 1996.
- southern_cross 8y ago
  Given what is currently known today (there may be more bombshells yet to come) about x86 system- and chip-level vulnerabilities, both unintentional and intentional, I'm not sure that any sane CxO would allow their data to come anywhere near the platform if they were starting all over today again from scratch. Not that I would necessarily trust any other vendor either (especially not Microsoft), but still ... Always remember (and don't ever forget) that the only reason most of us have even heard of Intel and Microsoft is because of their relationship to IBM back in the day. They weren't chosen, nor did they ascend to great heights, based the quality of their products. Rather IBM chose them mostly for the sake of its own convenience, and then they leveraged that relationship to the hilt.
southern_cross 8y ago
If you listen to the folks over at Terminal Talk (http://terminaltalk.net http://terminaltalk.net), they routinely question the wisdom of all those server folks having to build out and maintain infrastructure that just comes built-in to the mainframe. And from what I gather a lot of those server folks (or at least their bosses) may be starting to question that now, too. As for Facebook and Google and Netflix and so on, you have to remember that for all their claims of reliability and such, their stuff really only needs to be "good enough". But good enough generally just doesn't cut it when dealing with things like financials and such. BTW, I have a colleague who works for a massive corporation - one that is still recovering from a ransomware attack which has so far cost them at least $300 million. They have thousands of servers, and she says that it has now become IT's full-time job just to keep those updated and patched. (There's apparently little or no time these days for silly little things like development and testing and code loads and such.) I didn't ask her if they are still planning to replace their few remaining mainframes (I'm guessing they're in no big hurry now), but she did tell me that upper management currently thinks that "The Cloud" may yet be their road to salvation.

dralley 8y ago

One of my older coworkers told me that IBM used to have a travelling mainframe demo in the back of a truck. As part of this demo, they would start a big computation, and then reach into the mainframe and start pulling out RAM and entire CPUs straight from the sockets, while the thing kept right on computing.

bunderbunder 8y ago
FWIW, you could do the same thing with a Spark cluster. It might not handle it quite as gracefully, since you'd be taking down entire compute nodes, and probably some of the data storage along with them, but the computation would also be able to continue.
- dralley 8y ago
  Yes, but I'm guessing this predates Spark by decades. Not sure of the exact timeframe, though. In any case, it helps explain why a purse-string-holding exec would be convinced to shell out money to IBM.
- fs111 8y ago
  spark clusters still have a spof as far as I understand the architecture.
  - bunderbunder 8y ago
    Two of them, I believe - the master for the cluster as a whole, and, for any individual job, its driver. I wasn't really meaning to say that a Spark cluster is as robust as a mainframe (though maybe someone's figured out some tricks), more that I could totally see some sales folks conducting the same demo using something like Spark.
    - shub 8y ago
      If you're using Spark's built-in scheduler then the cluster manager is a SPOF. Hadoop docs say you can get active/standy ResourceManager, not that I've tried it. Spark can also use k8s and nomad to schedule executors, and those have HA modes as well. I assume Mesos does HA. You're still boned if the driver dies. I am pretty sure that the driver keeps some important state in RAM so if the node hosting it goes down you have to restart from the beginning, even if the cluster manager restarts the driver.
floatboth 8y ago
ZFS developers like to do this with hard drives :) But how did IBM do that with RAM? Do mainframes have… redundant RAM? Like literally sticks mirroring others?

bunderbunder 8y ago

I think that, within the context of that story, both directors are missing the deeper issue: A company's technology decisions aren't just about the final state, they're also about getting the project bootstrapped, and all the other technology decisions between that point and the final state. So it's not just that Facebook had already done it their way, it's that, along the way, they never had any realistic alternatives that led toward a mainframe solution. Having never worked directly with a mainframe, I can't really speak to the reliability question. What I do know, though, is that I've spent my entire life living around PC hardware, working on PC hardware, etc, because that's the hardware that's designed to fit into my day-to-day life. So, if I have a new idea, and want to try it out, I'm going to reach for the hardware I already have, because it's basically free. Just like Mark Zuckerberg did. And at that point, I'm already on a path that leads inexorably toward building the reliability into the software. There's simply never going to be any point at which it makes sense for me to scrap everything I have so far and do a complete rewrite for a different platform.

xoa 8y ago
>I think that, within the context of that story, both directors are missing the deeper issue: A company's technology decisions aren't just about the final state, they're also about getting the project bootstrapped, and all the other technology decisions between that point and the final state. So it's not just that Facebook had already done it their way, it's that, along the way, they never had any realistic alternatives that led toward a mainframe solution. Absolutely agreed, "path dependency" is a big issue that comes up in a lot of different contexts both in the natural world and human endeavors. Just because from a high level or after the fact someone can identify a much more optimal final result doesn't mean that result was actually realistic to get to, or would be worth moving to from whatever local minima a project ended up in. I think path dependency plays a major part in what makes "disruption" possible, and in how businesses can sometimes be eaten from the bottom. A lot of companies end up in a state where they have a bunch of end point products but they haven't considered the paths necessary for users to reach that end point. Without a ramp their pipe can slowly empty.
southern_cross 8y ago
> There's simply never going to be any point at which it makes sense for me to scrap everything I have so far and do a complete rewrite for a different platform. Yet people still often claim that it makes sense to go to all the time and trouble and expense of moving off of reliable legacy platforms onto other platforms which aren't naturally anywhere near as reliable. Imagine that!

rodgerd 8y ago

The sales job Amazon et al did in convincing the world resilience was the job of app developers, not the system vendor, was quite impressive. And while it's true that commodity folks could stand to learn from mainframe folks - serverless looks quite a lot like a forty years on re-invention of CICS if you squint a bit - your story reminds me how profoundly true the opposite is, as well. I've been part of a team that ran Linux on s390x, and it was a great exercise in demonstrating that however reliable your mainframe, it's still a SPOF. Even if you don't have my bad luck of four catastrophic hardware failures in as many years, your one reliable box still depends on the power, network fabric, storage fabric, etc at a single site. If you want to avoid losing your business when a sparky screws up, you need... several mainframes. At which point you're spending an absolutely astronomical amount to either sysplex, or you're building resilience into your applications, just like a bitty box. And yes, so many mainframe folks are so very out of touch with the broader world. In around 2016 I had the supposed regional tech expert on z-Series system lecture me (rather snidely) on how: 1. s390 processors were 20 - 40 times more performant that Intel processors (demonstrably not true for any workload I cared about). 2. You could not do virtualisation on x86_64. It was impossible. With a straight face, in the year 2016, this guy told me, and clearly believed, that it was not possible to run heterogenous virtualised workloads on an Intel processor. Apparently tens of billions of Jeff Bezos' net worth literally did not exist for him. There was more, but it was like talking to someone who had been frozen in a block of ice since 1996.

southern_cross 8y ago
Given what is currently known today (there may be more bombshells yet to come) about x86 system- and chip-level vulnerabilities, both unintentional and intentional, I'm not sure that any sane CxO would allow their data to come anywhere near the platform if they were starting all over today again from scratch. Not that I would necessarily trust any other vendor either (especially not Microsoft), but still ... Always remember (and don't ever forget) that the only reason most of us have even heard of Intel and Microsoft is because of their relationship to IBM back in the day. They weren't chosen, nor did they ascend to great heights, based the quality of their products. Rather IBM chose them mostly for the sake of its own convenience, and then they leveraged that relationship to the hilt.

southern_cross 8y ago

If you listen to the folks over at Terminal Talk (http://terminaltalk.net http://terminaltalk.net), they routinely question the wisdom of all those server folks having to build out and maintain infrastructure that just comes built-in to the mainframe. And from what I gather a lot of those server folks (or at least their bosses) may be starting to question that now, too. As for Facebook and Google and Netflix and so on, you have to remember that for all their claims of reliability and such, their stuff really only needs to be "good enough". But good enough generally just doesn't cut it when dealing with things like financials and such. BTW, I have a colleague who works for a massive corporation - one that is still recovering from a ransomware attack which has so far cost them at least $300 million. They have thousands of servers, and she says that it has now become IT's full-time job just to keep those updated and patched. (There's apparently little or no time these days for silly little things like development and testing and code loads and such.) I didn't ask her if they are still planning to replace their few remaining mainframes (I'm guessing they're in no big hurry now), but she did tell me that upper management currently thinks that "The Cloud" may yet be their road to salvation.