8 ms·

How did this scale without kubernetes, docker, and microservices? How does this work without transpiling from TypeScript to another language? This wont scale wi

How did this scale without kubernetes, docker, and microservices? How does this work without transpiling from TypeScript to another language? This wont scale without type safety!! Does it even have a JIRA backlog and dedicated product manager? No wonder this system is so mediocre. It probably wasnt even made following Agile guidelines!

bpodgursky 4y ago
The NY metro is infamously unable to scale, in large part because of extraordinary tech debt! The cost of each new line is stratospheric! They use century-old manual track routing cables! You might think this is a clever take but it's really not far from the truth. They NEED a revamp on a modern tech stack so they can use the trains, electronics, and technology trains in the rest of the world run on.
- woodruffw 4y ago
  The subway’s issues are mostly physical, not in software: the city (really, the state) contracts virtually everything around fare collection out to Cubic, and they do an okay enough job. See for example the OMNY card, which rolled out surprisingly fast. In other words: the things that are currently falling apart (the physical relays, for example) are almost completely disconnected from fare collection (which is what the article’s about). The latter runs pretty well; replacing it with a modern tech stack rather than the current incremental approach wouldn’t have many benefits.
- devwastaken 4y ago
  In 20 years all current software stacks will be legacy and equally if not more unusable. We rely significantly more now on tooling to just "do it for us". That tooling relies heavily on the OS it runs on and makes assumptions.
  - chinabot 4y ago
    If you diddnt do updates I seriously doubt the stack would survive 2 years. The joys of modern interconnected systems
- nickbauman 4y ago
  I believe your comment might be a bit too broad for the claims you're making. Arguably, the most advanced "tech stack" for moving trains around rail systems today (at scale, with the highest safety rating) is based around the Sicas ECC electronic interlocking by Siemens. However, this article is merely about the technology used for collection of fares for the NYC metro rail system.
- bontaq 4y ago
  A modern tech stack won't help that much, there are other issues. > Most subway services cannot significantly increase their frequencies during rush hours, except for the 1, G, J/Z, L, and M trains (the L service already is automated with CBTC). > However, even without CBTC, the system is currently retrofitted to operate at frequencies of up to 60 trains per hour (tph) on the IND Queens Boulevard Line (30 tph on each of the local and express pairs of tracks made possible by the Jamaica–179th Street terminal, which has four sidings past the terminal for each set of tracks) and 33 tph on the IRT Flushing Line. https://en.wikipedia.org/wiki/Signaling_of_the_New_York_City_Subway https://en.wikipedia.org/wiki/Signaling_of_the_New_York_City...
yieldcrv 4y ago
It didn't scale, thats the problem and why we have all those other things now. Imagine if we required $ billions in public resources to run other mundane software projects people came to rely on.
- ok_dad 4y ago
  It scaled fine, because it’s been used for 30 years without many issues. All that modern garbage would have been out of date several times by now. The only reason to modernize is because the old hardware is failing.
  - ncmncm 4y ago
    Not even for that: OS/2 runs fine in a VM.
- woodruffw 4y ago
  What didn't scale? I've lived in NYC for over 20 years, and I've never had a single availability issue with the MetroCard system. It's pretty rock solid, despite the decrepit stack it runs on, and probably uses a tiny fraction of the compute resources that a rewrite in a "modern" stack would demand.
wslh 4y ago
Beyond the irony, commenting on this thread here: OS/2 supports CORBA for distributing objects on different computers [1]. [1] https://en.wikipedia.org/wiki/IBM_System_Object_Model https://en.wikipedia.org/wiki/IBM_System_Object_Model
- retrocryptid 4y ago
  This system used IBM's DAE (Distributed Automation Edition) which was sort of like CORBA before CORBA. Imagine CORBA except no OO features, tightly integrated with SNA, no interceptors or IDL compilers and only worked with C, REXX and COBOL. And it was all implemented by interns and contractors. IIRC, they eventually dumped DAE, but had to deliver something before CORBA (or DSOM) was a thing.
1vuio0pswjnm7 4y ago
It was possible to (cross)compile binaries on OS/2 that would run on both OS/2 and DOS. For example, it is possible to run the same Word 5.5 executable on both OS/2 and DOS. Everything was textmode but there is also a "pseudo-textmode" that allows, e.g, italics. https://web.archive.org/web/20071012190431/http://pages.prodigy.net/michaln/history/os213/index.html https://web.archive.org/web/20071012190431/http://pages.prod... The power and energy efficiency of today's computers is wasted on commercial surveillance and draining advertising budgets. Let us pray that someone discovers a more effective means of advertising than the computer. The 2018 Netflix series "Maniac" seemed to hint at this alternate future. https://en.wikipedia.org/wiki/Maniac_(miniseries) https://en.wikipedia.org/wiki/Maniac_(miniseries)
necubi 4y ago
The article says that the peak daily traffic for the subway was 5M. This is not exactly huge scale today. I've worked on systems that handle that many users every second (and yes, they were built on Kubernetes). I think it's likely that the whole system could run (from a resources perspective, if not from a reliability one) from a single commodity server today.
- throwawaylinux 4y ago
  In that case it shouldn't take more than a few weeks for a couple fullstack devops rockstars to craft a reimagined blockchain based system holistically using rust as a service with agility at scale.
- woodruffw 4y ago
  I think you're right about the single commodity server, but just to refine the numbers a bit: the MetroCard is used on the subway, all NYCT and MTA buses, all PATH trains, the Roosevelt Island tram, the AirTrain, and Westchester's Bee-Line. Put together, the daily weekday ridership is probably closer to ~14M, pre-COVID. And that's not including some of the knock-on computational costs, like the radio network that links the MTA's buses or the in-system transfer rules.
Test0129 4y ago
I love this comment. The sarcasm is so thick. It turns out, in my opinion, we solved most problems in software engineering decades ago. All this high flying non-sense, web3 petabyte scale cloud powered type safe microservices, is pointless navel gazing. Agile is pointless navel gazing at least in it's current incarnation. Our entire industry has been mired in bullshit that hamstrings us. There's also a tremendous difference in developers. Developers even two decades ago were near top of their game because they had to be. They didn't have kubernetes-docker-CI/CD-SRE-devops to protect them. The result? Far better, far more reliable code. Any idiot can write software. Changing languages won't save you. A good programmer can make any language into a good language - and as it turns out 30 years ago we did it right. If they're only encountering scaling issues now that isn't a knock against them. It goes to show just how well they took "primitive" tools and made a system that simply works. Asking a modern dev team to produce a system that simply works often requires several months of planning, two full time product managers, a tech lead, several weeks of architecting, and approval (in blood) from the CTO who somehow has to involve themselves in literally anything and everything. Let's not forget we have to get it spun up with terraform so it's in the cloud and that'll take a request to SRE and possibly even dev-ops to get done. By the time the developers even get to write code, the deadline already arrived! We are comical facsimiles of our fore-developers and frankly should feel ashamed.
- jboy55 4y ago
  You have a lot of survivorship bias in your post here. Having lived, albeit briefly in the software industry of the 90s, the world was full of massively overbudgeted, multi-million dollar messes. 22 years ago, I was working on software to run a single state's fish and game licensing. We had 4 product managers (backend, admin, retail-front-end, consumer-web), 6-8 architects, at least 10 project managers and probably 70+ engineers, 30+ QA engineers. It had its specs written by a separate group for 6 months, that was then thrown away, and then rebuilt using Rational Unified Process. It was 10x over budget money-wise and 3x time wise. It took around 3 years to release to the public. Those four products, back-end team wrote in pure Oracle stored procedures. the retail-front-end team wrote in 95% JSPs, with a small library of java functions. When logic needed to split amoungst 4 types of licenses, they copied pasted an entire 2000 line JSP over and changed minor code, then just forwarded to the right JSP depending on the type. The consumer-web code was written in Perl, which needed to call functions in the retail-front-end, so the solution was to wrap the JSP code, call them with, what we would call now, mocked request/response objects. The PERL software called this mess using SOAP. So care had to be strip down every interface to use primitive objects so PERL and Java could talk. The admin-code, my team, wrote in our own Java MVC framework, Struts was just created. What started as a web-project turned into a 'recreate the mainframe interface using html' though. So yeah, 30 years ago, we did things right, or perhaps you meant 50 years ago?
busterarm 4y ago
If you think that's something, this whole system ran on a Token Ring network and extremely limited bandwidth.
funstuff007 4y ago
It also probably doesn't spend $500K a month on AWS--most of which would be allocated to egress costs and servers that are monitoring and logging the servers that actually provide the services.

bpodgursky 4y ago

The NY metro is infamously unable to scale, in large part because of extraordinary tech debt! The cost of each new line is stratospheric! They use century-old manual track routing cables! You might think this is a clever take but it's really not far from the truth. They NEED a revamp on a modern tech stack so they can use the trains, electronics, and technology trains in the rest of the world run on.

woodruffw 4y ago
The subway’s issues are mostly physical, not in software: the city (really, the state) contracts virtually everything around fare collection out to Cubic, and they do an okay enough job. See for example the OMNY card, which rolled out surprisingly fast. In other words: the things that are currently falling apart (the physical relays, for example) are almost completely disconnected from fare collection (which is what the article’s about). The latter runs pretty well; replacing it with a modern tech stack rather than the current incremental approach wouldn’t have many benefits.
devwastaken 4y ago
In 20 years all current software stacks will be legacy and equally if not more unusable. We rely significantly more now on tooling to just "do it for us". That tooling relies heavily on the OS it runs on and makes assumptions.
- chinabot 4y ago
  If you diddnt do updates I seriously doubt the stack would survive 2 years. The joys of modern interconnected systems
nickbauman 4y ago
I believe your comment might be a bit too broad for the claims you're making. Arguably, the most advanced "tech stack" for moving trains around rail systems today (at scale, with the highest safety rating) is based around the Sicas ECC electronic interlocking by Siemens. However, this article is merely about the technology used for collection of fares for the NYC metro rail system.
bontaq 4y ago
A modern tech stack won't help that much, there are other issues. > Most subway services cannot significantly increase their frequencies during rush hours, except for the 1, G, J/Z, L, and M trains (the L service already is automated with CBTC). > However, even without CBTC, the system is currently retrofitted to operate at frequencies of up to 60 trains per hour (tph) on the IND Queens Boulevard Line (30 tph on each of the local and express pairs of tracks made possible by the Jamaica–179th Street terminal, which has four sidings past the terminal for each set of tracks) and 33 tph on the IRT Flushing Line. https://en.wikipedia.org/wiki/Signaling_of_the_New_York_City_Subway https://en.wikipedia.org/wiki/Signaling_of_the_New_York_City...

yieldcrv 4y ago

It didn't scale, thats the problem and why we have all those other things now. Imagine if we required $ billions in public resources to run other mundane software projects people came to rely on.

ok_dad 4y ago
It scaled fine, because it’s been used for 30 years without many issues. All that modern garbage would have been out of date several times by now. The only reason to modernize is because the old hardware is failing.
- ncmncm 4y ago
  Not even for that: OS/2 runs fine in a VM.
woodruffw 4y ago
What didn't scale? I've lived in NYC for over 20 years, and I've never had a single availability issue with the MetroCard system. It's pretty rock solid, despite the decrepit stack it runs on, and probably uses a tiny fraction of the compute resources that a rewrite in a "modern" stack would demand.

wslh 4y ago

Beyond the irony, commenting on this thread here: OS/2 supports CORBA for distributing objects on different computers [1]. [1] https://en.wikipedia.org/wiki/IBM_System_Object_Model https://en.wikipedia.org/wiki/IBM_System_Object_Model

retrocryptid 4y ago
This system used IBM's DAE (Distributed Automation Edition) which was sort of like CORBA before CORBA. Imagine CORBA except no OO features, tightly integrated with SNA, no interceptors or IDL compilers and only worked with C, REXX and COBOL. And it was all implemented by interns and contractors. IIRC, they eventually dumped DAE, but had to deliver something before CORBA (or DSOM) was a thing.

1vuio0pswjnm7 4y ago

It was possible to (cross)compile binaries on OS/2 that would run on both OS/2 and DOS. For example, it is possible to run the same Word 5.5 executable on both OS/2 and DOS. Everything was textmode but there is also a "pseudo-textmode" that allows, e.g, italics. https://web.archive.org/web/20071012190431/http://pages.prodigy.net/michaln/history/os213/index.html https://web.archive.org/web/20071012190431/http://pages.prod... The power and energy efficiency of today's computers is wasted on commercial surveillance and draining advertising budgets. Let us pray that someone discovers a more effective means of advertising than the computer. The 2018 Netflix series "Maniac" seemed to hint at this alternate future. https://en.wikipedia.org/wiki/Maniac_(miniseries) https://en.wikipedia.org/wiki/Maniac_(miniseries)

necubi 4y ago

The article says that the peak daily traffic for the subway was 5M. This is not exactly huge scale today. I've worked on systems that handle that many users every second (and yes, they were built on Kubernetes). I think it's likely that the whole system could run (from a resources perspective, if not from a reliability one) from a single commodity server today.

throwawaylinux 4y ago
In that case it shouldn't take more than a few weeks for a couple fullstack devops rockstars to craft a reimagined blockchain based system holistically using rust as a service with agility at scale.
woodruffw 4y ago
I think you're right about the single commodity server, but just to refine the numbers a bit: the MetroCard is used on the subway, all NYCT and MTA buses, all PATH trains, the Roosevelt Island tram, the AirTrain, and Westchester's Bee-Line. Put together, the daily weekday ridership is probably closer to ~14M, pre-COVID. And that's not including some of the knock-on computational costs, like the radio network that links the MTA's buses or the in-system transfer rules.

Test0129 4y ago

I love this comment. The sarcasm is so thick. It turns out, in my opinion, we solved most problems in software engineering decades ago. All this high flying non-sense, web3 petabyte scale cloud powered type safe microservices, is pointless navel gazing. Agile is pointless navel gazing at least in it's current incarnation. Our entire industry has been mired in bullshit that hamstrings us. There's also a tremendous difference in developers. Developers even two decades ago were near top of their game because they had to be. They didn't have kubernetes-docker-CI/CD-SRE-devops to protect them. The result? Far better, far more reliable code. Any idiot can write software. Changing languages won't save you. A good programmer can make any language into a good language - and as it turns out 30 years ago we did it right. If they're only encountering scaling issues now that isn't a knock against them. It goes to show just how well they took "primitive" tools and made a system that simply works. Asking a modern dev team to produce a system that simply works often requires several months of planning, two full time product managers, a tech lead, several weeks of architecting, and approval (in blood) from the CTO who somehow has to involve themselves in literally anything and everything. Let's not forget we have to get it spun up with terraform so it's in the cloud and that'll take a request to SRE and possibly even dev-ops to get done. By the time the developers even get to write code, the deadline already arrived! We are comical facsimiles of our fore-developers and frankly should feel ashamed.

jboy55 4y ago
You have a lot of survivorship bias in your post here. Having lived, albeit briefly in the software industry of the 90s, the world was full of massively overbudgeted, multi-million dollar messes. 22 years ago, I was working on software to run a single state's fish and game licensing. We had 4 product managers (backend, admin, retail-front-end, consumer-web), 6-8 architects, at least 10 project managers and probably 70+ engineers, 30+ QA engineers. It had its specs written by a separate group for 6 months, that was then thrown away, and then rebuilt using Rational Unified Process. It was 10x over budget money-wise and 3x time wise. It took around 3 years to release to the public. Those four products, back-end team wrote in pure Oracle stored procedures. the retail-front-end team wrote in 95% JSPs, with a small library of java functions. When logic needed to split amoungst 4 types of licenses, they copied pasted an entire 2000 line JSP over and changed minor code, then just forwarded to the right JSP depending on the type. The consumer-web code was written in Perl, which needed to call functions in the retail-front-end, so the solution was to wrap the JSP code, call them with, what we would call now, mocked request/response objects. The PERL software called this mess using SOAP. So care had to be strip down every interface to use primitive objects so PERL and Java could talk. The admin-code, my team, wrote in our own Java MVC framework, Struts was just created. What started as a web-project turned into a 'recreate the mainframe interface using html' though. So yeah, 30 years ago, we did things right, or perhaps you meant 50 years ago?

busterarm 4y ago

If you think that's something, this whole system ran on a Token Ring network and extremely limited bandwidth.

funstuff007 4y ago

It also probably doesn't spend $500K a month on AWS--most of which would be allocated to egress costs and servers that are monitoring and logging the servers that actually provide the services.