3 ms·

The sentiment ITT seems to be overly dismissive of the value of a system that can reliably run for decades. How many of the new Kafka systems being stood up tod

The sentiment ITT seems to be overly dismissive of the value of a system that can reliably run for decades. How many of the new Kafka systems being stood up today would you realistically expect to still be running in 2061? How reliably do you expect those systems to perform over that 40 year period? The trade offs are quite apparent, but the value these systems provide is perfectly obvious too. I’m not sure I buy the extensibility argument either. In my experience the COBOL typically runs on the core processing system, the bit of the bank that runs the central ledger. This is usually a small portion of the banks technology footprint, and the rest of their integrating systems can run anything they like. I’ve worked in quite a few banks, and in my experience .NET is a lot more ubiquitous than COBOL is.

zeckalpha 5y ago
Confluent managed Kafka offers an SLA with a downtime of 20 minutes a month… for mission critical real time services (rather than async background processing) that won’t cut it.
lmm 5y ago
> The sentiment ITT seems to be overly dismissive of the value of a system that can reliably run for decades. In my experience it's a very narrow kind of reliability, where no-one cares whether the calculations are right as long as they're the same ones you've always run, and no-one's ever validated what or why is being calculated or sanity-checked it. Probably one of the vital constants in the system is 10x what it should be because someone made a typo it in 1982, but that's still considered "reliable". > How many of the new Kafka systems being stood up today would you realistically expect to still be running in 2061? How reliably do you expect those systems to perform over that 40 year period? If they were babied as much as the COBOL systems (no changes without months of testing, if data comes in in an incompatible format then we preprocess it...) then I'd expect them to last longer. They probably won't be, but if you decided to run the Kafka systems in a reliability-above-all-else way, I bet they'd do it better than the COBOL ones.
- hakfoo 5y ago
  > If they were babied as much as the COBOL systems (no changes without months of testing, if data comes in in an incompatible format then we preprocess it...) then I'd expect them to last longer. I'm skeptical on a pure hardware basis. Your classic mainframe was a very expensive piece of kit and outfitted as such. Everything redundant, self-monitoring, and hot swappable. The hardware support and backwards compatibility runs into the decades. Today we're renting a cloud service, that's made of pig-in-a-poke hardware. Reliable/high availability hardware is simulated with a bunch of disposable instances, at the cost of needing to design systems at the software level to sync and preserve data rather than having it built into the hardware. I'd be pleasantly surprised if I can snapshot an AWS/Azure/GCP image today, and fire it up in 2041 with no changes. (Hell, we have no assurances they'll be offering native x86-64 hardware in 20 years)
  - lmm 5y ago
    > Today we're renting a cloud service, that's made of pig-in-a-poke hardware. Reliable/high availability hardware is simulated with a bunch of disposable instances, at the cost of needing to design systems at the software level to sync and preserve data rather than having it built into the hardware. Right, but Kafka's already set up to do that. I've worked on Zookeeper clusters that have been continuously up for 10 years - not the individual hardware, but the overall system was continuously available for all that time. > I'd be pleasantly surprised if I can snapshot an AWS/Azure/GCP image today, and fire it up in 2041 with no changes. (Hell, we have no assurances they'll be offering native x86-64 hardware in 20 years) We'll be able to emulate them, and even if not, the Kafka stack is all JVM-based, so you don't need x86 to run it.
- dzdt 5y ago
  The Cobol code is mostly handling transactions that result in money going from one account to another. The people who own that money tend to scrutinize this very carefully! So after 40 years, pretty much any bug that occurs in practice has been squashed out of existence. This doesn't mean that every function in the code works correctly for arbitrary inputs that appear to be allowed by the function interface (formally a bug!) but it does the right thing for the inputs that have historically been provided (in practice very stable and reliable). That kind of reliability makes it hard to rewrite the code. There are cases for historically unused input combinations where you have no particular reason to believe the old code is bug free. So when your rewrite produces a different result in an obscure case, it is a long and painful business logic task to decide which version is right.

zeckalpha 5y ago

Confluent managed Kafka offers an SLA with a downtime of 20 minutes a month… for mission critical real time services (rather than async background processing) that won’t cut it.

lmm 5y ago

> The sentiment ITT seems to be overly dismissive of the value of a system that can reliably run for decades. In my experience it's a very narrow kind of reliability, where no-one cares whether the calculations are right as long as they're the same ones you've always run, and no-one's ever validated what or why is being calculated or sanity-checked it. Probably one of the vital constants in the system is 10x what it should be because someone made a typo it in 1982, but that's still considered "reliable". > How many of the new Kafka systems being stood up today would you realistically expect to still be running in 2061? How reliably do you expect those systems to perform over that 40 year period? If they were babied as much as the COBOL systems (no changes without months of testing, if data comes in in an incompatible format then we preprocess it...) then I'd expect them to last longer. They probably won't be, but if you decided to run the Kafka systems in a reliability-above-all-else way, I bet they'd do it better than the COBOL ones.

hakfoo 5y ago
> If they were babied as much as the COBOL systems (no changes without months of testing, if data comes in in an incompatible format then we preprocess it...) then I'd expect them to last longer. I'm skeptical on a pure hardware basis. Your classic mainframe was a very expensive piece of kit and outfitted as such. Everything redundant, self-monitoring, and hot swappable. The hardware support and backwards compatibility runs into the decades. Today we're renting a cloud service, that's made of pig-in-a-poke hardware. Reliable/high availability hardware is simulated with a bunch of disposable instances, at the cost of needing to design systems at the software level to sync and preserve data rather than having it built into the hardware. I'd be pleasantly surprised if I can snapshot an AWS/Azure/GCP image today, and fire it up in 2041 with no changes. (Hell, we have no assurances they'll be offering native x86-64 hardware in 20 years)
- lmm 5y ago
  > Today we're renting a cloud service, that's made of pig-in-a-poke hardware. Reliable/high availability hardware is simulated with a bunch of disposable instances, at the cost of needing to design systems at the software level to sync and preserve data rather than having it built into the hardware. Right, but Kafka's already set up to do that. I've worked on Zookeeper clusters that have been continuously up for 10 years - not the individual hardware, but the overall system was continuously available for all that time. > I'd be pleasantly surprised if I can snapshot an AWS/Azure/GCP image today, and fire it up in 2041 with no changes. (Hell, we have no assurances they'll be offering native x86-64 hardware in 20 years) We'll be able to emulate them, and even if not, the Kafka stack is all JVM-based, so you don't need x86 to run it.
dzdt 5y ago
The Cobol code is mostly handling transactions that result in money going from one account to another. The people who own that money tend to scrutinize this very carefully! So after 40 years, pretty much any bug that occurs in practice has been squashed out of existence. This doesn't mean that every function in the code works correctly for arbitrary inputs that appear to be allowed by the function interface (formally a bug!) but it does the right thing for the inputs that have historically been provided (in practice very stable and reliable). That kind of reliability makes it hard to rewrite the code. There are cases for historically unused input combinations where you have no particular reason to believe the old code is bug free. So when your rewrite produces a different result in an obscure case, it is a long and painful business logic task to decide which version is right.