4 ms·

StackOverflow, I feel, inherently ties into the "Release early. Release often." philosophy: if you ship broken code , oh well , there'll be a release soon enoug

by chx 4y ago

StackOverflow, I feel, inherently ties into the "Release early. Release often." philosophy: if you ship broken code , oh well , there'll be a release soon enough to fix it. This can not be borne in a mainframe environment. This is not to say there aren't bugs because of course there are but the expectation is completely different. This kind of "by your bootstraps" education is no bueno. You will get lots of training and excellent documentation. We used to measure the amount of documentation IBM ships with the mainframe not in pages but in yards it takes on the shelves. StackOverflow in itself is the answer to the question "who you gonna call" -- because you might be alone, or you might be working on a small team where no one has better expertise, and usually you are working with open source and so no one is obligated to pick up the phone at the software "manufacturer". This scenario does not play in a mainframe environment: you are going to be part of a larger team, full of literal graybeards and when push comes to shove then you do have a contact number: your employer is paying absolutely ridiculous amounts of money to the software company and they will have a contact person who will pick up the phone.

bombcar 4y ago
Yeah, mainframe coding is much closer to "code NASA runs on spacecraft" than what most people consider coding to be today. And the level of hardware support (mainframes are basically redundant on everything) and software support is unheard of. Consider that these things are so integrated into major banks, etc, that even minutes of downtime would cost millions upon millions, and think of all the things you can spend millions on to reduce the chance of it happening.
- chx 4y ago
  > mainframes are basically redundant on everything Not only that but that redundancy is hidden from everything. Already decades ago the Hungarian electricity distributor had a "RAID" storage unit where one bunch of disks were at the capitol, the other bunch connected with fiber at another city -- and you talked to this thing as if it were an ordinary disk. Short of an invading army nothing could cause data loss on that thing -- simple aerial bombing wouldn't because the unit in the capitol was tens of meters underground... that's not concerns you have on your average web app.
  - bombcar 4y ago
    Yeah, one thing that I've been really disappointed with is how UNRELIABLE the "cloud" is - unless you engineer all the reliability in the software itself. We have the technology and the capability to do things like run a VM and have it move between hardware when that hardware fails or needs replacement. but that would cost money.
    - ghaff 4y ago
      VMs can often be configured to do "live migration" but that doesn't protect a running instance if the hardware it is running on fails completely. There is such a thing as fault tolerant hardware/software stacks but, yes, most modern architectures shift application resiliency higher up the stack.
    - runako 4y ago
      > the capability to do things like run a VM and have it move between hardware when that hardware fails This is one of my favorite features of Google Cloud: https://cloud.google.com/compute/docs/instances/live-migration-process https://cloud.google.com/compute/docs/instances/live-migrati... Unfortunately, Google continues to be the third-string provider in the US, so I don't often get the opportunity to work with Google Cloud.
      - nzentzis 4y ago
        I want to be able to justify using Google Cloud for projects, but the fact that it’s a Google product is the main obstacle to that. If history is anything to go by, relying on Google products and services seems like a great way to set yourself up for major problems down the line. Either the product you’re using is shut down on short notice or some automation flags a false-positive and you’re left screaming into the void trying to get yourself unbanned.
    - huslage 4y ago
      It’s not a live migration. It’s running an a VM in a sysplex which is in lockstep with its peer. There’s not a failover in the tradition sense..the I/O just goes from one side to the other.
derefr 4y ago
> This can not be borne in a mainframe environment. I mean, maybe not in an old-school 1970s batch-SIMD mainframe environment with limited storage memory. But in a modern mainframe environment, where 1. every tenant being processed is hitting their own virtualized environment locked into their own legacy versions of everything (think: separately-SCMed "Enterprise" deployments of SaaS software — just all living together on one machine cluster rather than on-prem per customer); and 2. you've got the spare storage capacity to do everything via async-queued CQRS/ES + writing secondary representations and cubes to a local data lake, rather than ever updating anything in place... ...then why not do "agile" blue-green deploys of your ACH reconciliation logic, starting with the smallest tenants first? If you ever produce a bug in such an environment, you can always just: stop the job; emit tombstone records for all the CQRS Commands it produced so they'll be ignored on "outbox processing" rather than pushed to consumers; fix the bug; up the version number for OLAP report generation, invalidating + requiring recomputation for any reports produced from the data in the reporting period; and then start the job up again. No external system will ever have to know, because your own system should be set up such that no other external system is synchronously dependent on its outputs. (This is half the reason things like ACH are designed with the async "flex" they have.) And, in fact, assuming you've got the spare CPU power (which you do), you can even run entire reporting / integration cycles twice — once under the old logic, once under the new logic — and examine the resulting queued commands + reports to see that they all come out the same, before allowing anything to go through. You can do this not just in production, but experimentally, against a corpus of all old input data for each tenant (which you've almost certainly kept around.) And, as such, you don't even have to stick with those virtualized legacy environments that mainframe architectures like z/OS were designed to enable — you can actively migrate clients away from them (even automatically, with CI/CD-triggered graduated rollouts et al) as soon as you can prove that their workloads (both present and historical!) won't be affected by the change. Each tenant can be re-pinned over time to the newest release that can be proven to be equivalent to the original release provided to them on all known data. One of the key insights about mainframe programming, is that the software architectures of the kinds of systems deployed on mainframes, are designed to be resilient against individual bugs; or, one might say, against individual programmers who don't know what they're doing. Systems engineering in e.g. finance, is to bad individual-component / individual-release logic, as NASA's Mars-rover systems engineering is to bad hardware: designed under the assumption of the potential for failure, and designed with built-in remediation strategies for that failure.
markus_zhang 4y ago
Thanks for sharing. TBH I envy those good trainings if they are that good. The trainings we get (e.g. GCP) are OK but definitely not exceptionally good. I actually wonder what training IBM gives.

by chx 4y ago

bombcar 4y ago

Yeah, mainframe coding is much closer to "code NASA runs on spacecraft" than what most people consider coding to be today. And the level of hardware support (mainframes are basically redundant on everything) and software support is unheard of. Consider that these things are so integrated into major banks, etc, that even minutes of downtime would cost millions upon millions, and think of all the things you can spend millions on to reduce the chance of it happening.

chx 4y ago
> mainframes are basically redundant on everything Not only that but that redundancy is hidden from everything. Already decades ago the Hungarian electricity distributor had a "RAID" storage unit where one bunch of disks were at the capitol, the other bunch connected with fiber at another city -- and you talked to this thing as if it were an ordinary disk. Short of an invading army nothing could cause data loss on that thing -- simple aerial bombing wouldn't because the unit in the capitol was tens of meters underground... that's not concerns you have on your average web app.
- bombcar 4y ago
  Yeah, one thing that I've been really disappointed with is how UNRELIABLE the "cloud" is - unless you engineer all the reliability in the software itself. We have the technology and the capability to do things like run a VM and have it move between hardware when that hardware fails or needs replacement. but that would cost money.
  - ghaff 4y ago
    VMs can often be configured to do "live migration" but that doesn't protect a running instance if the hardware it is running on fails completely. There is such a thing as fault tolerant hardware/software stacks but, yes, most modern architectures shift application resiliency higher up the stack.
  - runako 4y ago
    > the capability to do things like run a VM and have it move between hardware when that hardware fails This is one of my favorite features of Google Cloud: https://cloud.google.com/compute/docs/instances/live-migration-process https://cloud.google.com/compute/docs/instances/live-migrati... Unfortunately, Google continues to be the third-string provider in the US, so I don't often get the opportunity to work with Google Cloud.
    - nzentzis 4y ago
      I want to be able to justify using Google Cloud for projects, but the fact that it’s a Google product is the main obstacle to that. If history is anything to go by, relying on Google products and services seems like a great way to set yourself up for major problems down the line. Either the product you’re using is shut down on short notice or some automation flags a false-positive and you’re left screaming into the void trying to get yourself unbanned.
  - huslage 4y ago
    It’s not a live migration. It’s running an a VM in a sysplex which is in lockstep with its peer. There’s not a failover in the tradition sense..the I/O just goes from one side to the other.

derefr 4y ago

> This can not be borne in a mainframe environment. I mean, maybe not in an old-school 1970s batch-SIMD mainframe environment with limited storage memory. But in a modern mainframe environment, where 1. every tenant being processed is hitting their own virtualized environment locked into their own legacy versions of everything (think: separately-SCMed "Enterprise" deployments of SaaS software — just all living together on one machine cluster rather than on-prem per customer); and 2. you've got the spare storage capacity to do everything via async-queued CQRS/ES + writing secondary representations and cubes to a local data lake, rather than ever updating anything in place... ...then why not do "agile" blue-green deploys of your ACH reconciliation logic, starting with the smallest tenants first? If you ever produce a bug in such an environment, you can always just: stop the job; emit tombstone records for all the CQRS Commands it produced so they'll be ignored on "outbox processing" rather than pushed to consumers; fix the bug; up the version number for OLAP report generation, invalidating + requiring recomputation for any reports produced from the data in the reporting period; and then start the job up again. No external system will ever have to know, because your own system should be set up such that no other external system is synchronously dependent on its outputs. (This is half the reason things like ACH are designed with the async "flex" they have.) And, in fact, assuming you've got the spare CPU power (which you do), you can even run entire reporting / integration cycles twice — once under the old logic, once under the new logic — and examine the resulting queued commands + reports to see that they all come out the same, before allowing anything to go through. You can do this not just in production, but experimentally, against a corpus of all old input data for each tenant (which you've almost certainly kept around.) And, as such, you don't even have to stick with those virtualized legacy environments that mainframe architectures like z/OS were designed to enable — you can actively migrate clients away from them (even automatically, with CI/CD-triggered graduated rollouts et al) as soon as you can prove that their workloads (both present and historical!) won't be affected by the change. Each tenant can be re-pinned over time to the newest release that can be proven to be equivalent to the original release provided to them on all known data. One of the key insights about mainframe programming, is that the software architectures of the kinds of systems deployed on mainframes, are designed to be resilient against individual bugs; or, one might say, against individual programmers who don't know what they're doing. Systems engineering in e.g. finance, is to bad individual-component / individual-release logic, as NASA's Mars-rover systems engineering is to bad hardware: designed under the assumption of the potential for failure, and designed with built-in remediation strategies for that failure.

markus_zhang 4y ago

Thanks for sharing. TBH I envy those good trainings if they are that good. The trainings we get (e.g. GCP) are OK but definitely not exceptionally good. I actually wonder what training IBM gives.