16 ms·

The Forgotten Operating System That Keeps the NYC Subway System Alive (2019)

170 points by lproven 4y ago

noodlesUK 4y ago
How is it possible that the magnetic stripe encoding is a secret? Surely you can just run the card through a reader before and after taking some journeys and reverse engineer from there? Is the balance actually stored on the card, or is it just basically a primary key in a database somewhere that gets queried on each ride?
- crazygringo 4y ago
  It's both, and you can find the answer on Wikipedia. [1] > During a swipe, the MetroCard is read, re-written to, then check-read to verify correct encoding... Each MetroCard stored value card is assigned a unique, permanent ten-digit serial number when it is manufactured. The value is stored magnetically on the card itself, while the card's transaction history is held centrally in the Automated Fare Collection (AFC) Database... Whenever the card is swiped at a turnstile, the value of the card is read, the new value is written, the customer is let through, and then the central database is updated with the new transaction as soon as possible. Cards are not validated in real time against the database when swiped to pay the fare. There's also an old NYT article [2]: > But since the main computer communicates regularly with both the turnstiles and the booth computers, false cards have so far all been invalidated after they have been used once... Some hackers believe that because the turnstiles, the token booth computers and the mainframe computer, respectively, upload and download information at six-minute intervals, the length of time that a passenger can transfer without paying may actually be 18 minutes more than the advertised two hours, if the timing is right... ''I believe that, fundamentally, a system as complicated as the Metrocard cannot be absolutely secure,'' he said. ''What we have to make sure is that the hack won't scale. If a few people ride for free, that's no big deal.'' [1] https://en.wikipedia.org/wiki/MetroCard#Technology https://en.wikipedia.org/wiki/MetroCard#Technology [2] https://www.nytimes.com/1997/08/11/nyregion/what-galls-a-hacker-most-the-metrocard.html https://www.nytimes.com/1997/08/11/nyregion/what-galls-a-hac...
  - ardel95 4y ago
    That’s the thing with fraud. The goal is never to stop it all, but to keep it at some local minimum. The rest is just the cost of doing business.
    - roywiggins 4y ago
      In this case it just has to be harder to pull off than jumping the turnstile.
      - kingkawn 4y ago
        Which is unfathomably easy, so much so that it doesn’t even require physically jumping
        Tijdreiziger 4y ago
        It wouldn’t be easy with fare gates, but the cost is probably too high to justify for NYC (imagine having to rip out all the existing turnstiles and replace them).
        crazygringo 4y ago
        To be fair there are already gates everywhere there's an entrance that doesn't have a manned booth. There are only turnstiles that can be jumped at the entrances that are manned by an attendant 24/7.
        kingkawn 4y ago
        I think it’s less cost than needing to process high passenger volume in old stations
        Akronymus 4y ago
        Which is part of the reason why vienna doesn't have turnstiles. https://www.youtube.com/watch?v=kq-X25pH1XQ https://www.youtube.com/watch?v=kq-X25pH1XQ
- dehrmann 4y ago
  > Surely you can just run the card through a reader before and after taking some journeys and reverse engineer from there? I doubt they do this, but if the data's encrypted with a good cipher used correctly, all you'll be able to detect is that it's changed.
  - lxgr 4y ago
    That does not prevent replay attacks, though. To thwart these, a stored-value system needs to eventually go online and synchronize a compromised card list, use (actively tamper and replay resistant) smartcards, or both.
- walrus01 4y ago
  they're in the process of replacing the whole magnetic stripe system right now with a modern contactless system similar to london, vancouver, seattle, etc.
0x445442 4y ago
OS/2 failed because of IBM’s abysmal marketing, not because it was inferior tech.
- chasil 4y ago
  OS/2 was forced down to 80286 compatibility for version 1.1. That was major design damage, required by IBM because most of the PS/2 line was on this CPU. Both Linux and NT required an 80386 with a flat memory model. On capable hardware, this was far superior. Both Linux and NT quickly ran on multiple platforms besides x86. I think that OS/2 was ported to Power, but nothing else.
  - tengwar2 4y ago
    No, that was v1.0. v1.1 was incompatible with v1.1 in some respects for that reason. I had some stuff that was literally hard-wired with wire-wrap to do bit-shuffling from an image acquisition system (assembly wasn't fast enough) and I would have had to re-wire it to migrate to 1.1. There were also some details that wouldn't have affected most people, like moving from 8 byte entries to 16 byte entries in the descriptor tables. Sorry if I'm vague on the details - it was a while back.
    - chasil 4y ago
      Interesting, the wiki is not clear on this. "OS/2 1.x targets the Intel 80286 processor and DOS fundamentally doesn't. IBM insisted on supporting the 80286 processor, with its 16-bit segmented memory mode, because of commitments made to customers who had purchased many 80286-based PS/2s as a result of IBM's promises surrounding OS/2. Until release 2.0 in April 1992, OS/2 ran in 16-bit protected mode and therefore could not benefit from the Intel 80386's much simpler 32-bit flat memory model and virtual 8086 mode features." https://en.m.wikipedia.org/wiki/OS/2 https://en.m.wikipedia.org/wiki/OS/2 The developer of busybox/toybox also had 80286 discussion. https://www.landley.net/history/mirror/os2/OS2Warp.html https://www.landley.net/history/mirror/os2/OS2Warp.html
  - markhahn 4y ago
    I remember 386 support connected with 1.3, but probably still using descriptors. NSFT was still involved at 2.0 time, though DOS+Windows was pretty obviously going to win, so to speak, and NT OS/2 was still hush-hush. NT OS/2 ran on ia32, mips, alpha, and i860 (wow, freaky to remember that!) at the time - the "new technology" was as much the racy idea of portable OS code as RISC and microkernels...
not2b 4y ago
"hilariously robust", the article says.
haltingproblem 4y ago
The articles makes vague complains and leaves many questions unanswered. First, I don't get the primary complaint in the article. The MTA chose OS/2 in the early 1990s, when there no other choices and it has served the MTA for almost 30 years with minimal hassles of upgrades or outages. I can't recall a single day when the Metrocard was not working - sure there are problems but it is more a UI/UX issue - a app that shows you Metrocard usage and balance would be insanely useful but we lack small mercies but hey you can pay with your smartwatch /s. A piece of technology that is stable for 30 years is remarkable. Even if they had gone with Linux/NT the upgrade cycles themselves would have wreaked havoc. OS/2 seems like a brilliant choice in that regard. Questions left unaddressed: - What tech did they move to now with contactless payments? - The move to contactless payments is awesome but what happens to people who lack smartphones or do not have credit cards or <big-tech>-pay setup on their phones. Or just casual tourists visiting from overseas with no US cards. How do they pay? Do they keep supporting metrocards forever? Does that mean to support Metrocards they need OS/2 and assorted new systems. - What is the mainframe part of the stack and what function does it do.
- cnorthwood 4y ago
  I was recently in New York as a Brit and was really happy to see the OMNY system which just allowed me to use contactless to traverse the turnstiles (for everything but the JFK SkyTrain) with no issues, similar to how metro transport works in the UK for example. I don't think there's a requirement for it to be a US card, but instead any contactless card will work?
  - wikibob 4y ago
    As it turns out: “The OMNY system is designed by Cubic Transportation Systems, using technology licensed from Transport for London's Oyster card.” https://www.cubic.com/news-events/news/cubic-wins-contract-new-york-mta-replace-iconic-metrocard-system-world-class-new https://www.cubic.com/news-events/news/cubic-wins-contract-n...
  - deleted 4y ago
    [deleted]
dflock 4y ago
OS/2. Also used for Vancouver's SkyTrain system, afaik.
manv1 4y ago
OS/2 was known to be rock-solid back in the day. It was the OS of choice for a lot of financials. Everyone moved to Windows NT when 4 came out, though, because of the way Microsoft did licensing.
- bogomipz 4y ago
  >"Everyone moved to Windows NT when 4 came out, though, because of the way Microsoft did licensing." Can you elaborate on what the difference was between how the licensing worked on OS/2 vs NT 4? What was the big innovation?
  - giamma 4y ago
    Microsoft in 1999 was not the same company it is today and did some very unethical things. For example: https://www.itprotoday.com/windows-78/ibm-microsoft-warned-pc-makers-against-using-os2 https://www.itprotoday.com/windows-78/ibm-microsoft-warned-p... Other articles report that computer producers were given Windows "for free" as long as they did not offer nor support OS/2 on their hardware. Microsoft applied this practice for a few years and indeed lost income by giving Windows for free, but it ensured that the regular user never heard of OS/2 and then when IBM gave up the situation changed. Besides, Microsoft would still make money from Office and other products.
    - bogomipz 4y ago
      Your link is about Windows 95. My question about what was different about the licensing model of NT4 that prompted companies to switch. NT 4 also came out in the mid 90s.
supernova87a 4y ago
If you want a story into the other "operating system" that keeps the NYC subway system running (or from another point of view, never advancing), see this: https://www.theatlantic.com/technology/archive/2015/11/why-dont-we-know-where-all-the-trains-are/415152/ https://www.theatlantic.com/technology/archive/2015/11/why-d... "Why New York Subway Lines Are Missing Countdown Clocks" and how it's actually very about how managing a software/IT project can go wrong.
- ncmncm 4y ago
  The key to all big projects that work well is that they start with small projects that work well, and deliver something useful immediately, adding more continuously. They don't need to stay with equipment they delivered at first; they just need to keep what visibly works working. They can replace older equipment and software with what works better in a bigger system, as they grow. If that seems more expensive, look at the hundred-million dollar+ contracts. Electronics is cheap, under those numbers.
dtx1 4y ago
> The Forgotten Operating System That Keeps the NYC Subway System Alive It's OS/2. Hardly forgotten.
- doublerabbit 4y ago
  Give it a few more years. The iconic icon of a Floppy Disk has almost faded out of existance.
- lproven 4y ago
  I submitted the article about a week ago, because HN had just discovered Blue Lion and crashed Arca Noae's server. A commenter asked for an example of someone, anyone, still using OS/2 in production. I remembered this story and offered it as an answer of someone using it as recently as 3Y ago. So I posted it as a top-level story as well. I guess HN re-upped it and it caught on. But yes, I think that for millennial techies, this is something that came and went before they were in the industry, so for them, it is forgotten tech that they've probably never seen.
someweirdperson 4y ago
Chinas railway is running on flash player [0]. At least that choice doesn't limit it to one specific OS. [0] https://news.ycombinator.com/item?id=25786670 https://news.ycombinator.com/item?id=25786670
jackblemming 4y ago
How did this scale without kubernetes, docker, and microservices? How does this work without transpiling from TypeScript to another language? This wont scale without type safety!! Does it even have a JIRA backlog and dedicated product manager? No wonder this system is so mediocre. It probably wasnt even made following Agile guidelines!
- bpodgursky 4y ago
  The NY metro is infamously unable to scale, in large part because of extraordinary tech debt! The cost of each new line is stratospheric! They use century-old manual track routing cables! You might think this is a clever take but it's really not far from the truth. They NEED a revamp on a modern tech stack so they can use the trains, electronics, and technology trains in the rest of the world run on.
  - woodruffw 4y ago
    The subway’s issues are mostly physical, not in software: the city (really, the state) contracts virtually everything around fare collection out to Cubic, and they do an okay enough job. See for example the OMNY card, which rolled out surprisingly fast. In other words: the things that are currently falling apart (the physical relays, for example) are almost completely disconnected from fare collection (which is what the article’s about). The latter runs pretty well; replacing it with a modern tech stack rather than the current incremental approach wouldn’t have many benefits.
  - devwastaken 4y ago
    In 20 years all current software stacks will be legacy and equally if not more unusable. We rely significantly more now on tooling to just "do it for us". That tooling relies heavily on the OS it runs on and makes assumptions.
    - chinabot 4y ago
      If you diddnt do updates I seriously doubt the stack would survive 2 years. The joys of modern interconnected systems
tibbydudeza 4y ago
If it takes years and billions to build a simple high speed rail line from San Francisco to LA then NYC subway this will still be around in it's current state.
Meph504 4y ago
You would be shocked at the number of ATM machines that run on OS/2, and though I don't advocate it, it makes sense in a way. When is the last exploit you saw published for OS/2, many of the old OS/2 machines direct dial into an old modem bank (at a bank.) The thread window on these is tiny because of how far behind they are.
AnnoyedComment 4y ago
mmmpop 4y ago
Didn't realize people forgot about OS/2?
- lproven 4y ago
  See my reply above.

170 points by lproven 4y ago

How is it possible that the magnetic stripe encoding is a secret? Surely you can just run the card through a reader before and after taking some journeys and reverse engineer from there? Is the balance actually stored on the card, or is it just basically a primary key in a database somewhere that gets queried on each ride?

crazygringo 4y ago
It's both, and you can find the answer on Wikipedia. [1] > During a swipe, the MetroCard is read, re-written to, then check-read to verify correct encoding... Each MetroCard stored value card is assigned a unique, permanent ten-digit serial number when it is manufactured. The value is stored magnetically on the card itself, while the card's transaction history is held centrally in the Automated Fare Collection (AFC) Database... Whenever the card is swiped at a turnstile, the value of the card is read, the new value is written, the customer is let through, and then the central database is updated with the new transaction as soon as possible. Cards are not validated in real time against the database when swiped to pay the fare. There's also an old NYT article [2]: > But since the main computer communicates regularly with both the turnstiles and the booth computers, false cards have so far all been invalidated after they have been used once... Some hackers believe that because the turnstiles, the token booth computers and the mainframe computer, respectively, upload and download information at six-minute intervals, the length of time that a passenger can transfer without paying may actually be 18 minutes more than the advertised two hours, if the timing is right... ''I believe that, fundamentally, a system as complicated as the Metrocard cannot be absolutely secure,'' he said. ''What we have to make sure is that the hack won't scale. If a few people ride for free, that's no big deal.'' [1] https://en.wikipedia.org/wiki/MetroCard#Technology https://en.wikipedia.org/wiki/MetroCard#Technology [2] https://www.nytimes.com/1997/08/11/nyregion/what-galls-a-hacker-most-the-metrocard.html https://www.nytimes.com/1997/08/11/nyregion/what-galls-a-hac...
- ardel95 4y ago
  That’s the thing with fraud. The goal is never to stop it all, but to keep it at some local minimum. The rest is just the cost of doing business.
  - roywiggins 4y ago
    In this case it just has to be harder to pull off than jumping the turnstile.
    - kingkawn 4y ago
      Which is unfathomably easy, so much so that it doesn’t even require physically jumping
      - Tijdreiziger 4y ago
        It wouldn’t be easy with fare gates, but the cost is probably too high to justify for NYC (imagine having to rip out all the existing turnstiles and replace them).
        crazygringo 4y ago
        To be fair there are already gates everywhere there's an entrance that doesn't have a manned booth. There are only turnstiles that can be jumped at the entrances that are manned by an attendant 24/7.
        kingkawn 4y ago
        I think it’s less cost than needing to process high passenger volume in old stations
      - Akronymus 4y ago
        Which is part of the reason why vienna doesn't have turnstiles. https://www.youtube.com/watch?v=kq-X25pH1XQ https://www.youtube.com/watch?v=kq-X25pH1XQ
dehrmann 4y ago
> Surely you can just run the card through a reader before and after taking some journeys and reverse engineer from there? I doubt they do this, but if the data's encrypted with a good cipher used correctly, all you'll be able to detect is that it's changed.
- lxgr 4y ago
  That does not prevent replay attacks, though. To thwart these, a stored-value system needs to eventually go online and synchronize a compromised card list, use (actively tamper and replay resistant) smartcards, or both.
walrus01 4y ago
they're in the process of replacing the whole magnetic stripe system right now with a modern contactless system similar to london, vancouver, seattle, etc.

0x445442 4y ago

OS/2 failed because of IBM’s abysmal marketing, not because it was inferior tech.

chasil 4y ago
OS/2 was forced down to 80286 compatibility for version 1.1. That was major design damage, required by IBM because most of the PS/2 line was on this CPU. Both Linux and NT required an 80386 with a flat memory model. On capable hardware, this was far superior. Both Linux and NT quickly ran on multiple platforms besides x86. I think that OS/2 was ported to Power, but nothing else.
- tengwar2 4y ago
  No, that was v1.0. v1.1 was incompatible with v1.1 in some respects for that reason. I had some stuff that was literally hard-wired with wire-wrap to do bit-shuffling from an image acquisition system (assembly wasn't fast enough) and I would have had to re-wire it to migrate to 1.1. There were also some details that wouldn't have affected most people, like moving from 8 byte entries to 16 byte entries in the descriptor tables. Sorry if I'm vague on the details - it was a while back.
  - chasil 4y ago
    Interesting, the wiki is not clear on this. "OS/2 1.x targets the Intel 80286 processor and DOS fundamentally doesn't. IBM insisted on supporting the 80286 processor, with its 16-bit segmented memory mode, because of commitments made to customers who had purchased many 80286-based PS/2s as a result of IBM's promises surrounding OS/2. Until release 2.0 in April 1992, OS/2 ran in 16-bit protected mode and therefore could not benefit from the Intel 80386's much simpler 32-bit flat memory model and virtual 8086 mode features." https://en.m.wikipedia.org/wiki/OS/2 https://en.m.wikipedia.org/wiki/OS/2 The developer of busybox/toybox also had 80286 discussion. https://www.landley.net/history/mirror/os2/OS2Warp.html https://www.landley.net/history/mirror/os2/OS2Warp.html
- markhahn 4y ago
  I remember 386 support connected with 1.3, but probably still using descriptors. NSFT was still involved at 2.0 time, though DOS+Windows was pretty obviously going to win, so to speak, and NT OS/2 was still hush-hush. NT OS/2 ran on ia32, mips, alpha, and i860 (wow, freaky to remember that!) at the time - the "new technology" was as much the racy idea of portable OS code as RISC and microkernels...

not2b 4y ago

"hilariously robust", the article says.

haltingproblem 4y ago

The articles makes vague complains and leaves many questions unanswered. First, I don't get the primary complaint in the article. The MTA chose OS/2 in the early 1990s, when there no other choices and it has served the MTA for almost 30 years with minimal hassles of upgrades or outages. I can't recall a single day when the Metrocard was not working - sure there are problems but it is more a UI/UX issue - a app that shows you Metrocard usage and balance would be insanely useful but we lack small mercies but hey you can pay with your smartwatch /s. A piece of technology that is stable for 30 years is remarkable. Even if they had gone with Linux/NT the upgrade cycles themselves would have wreaked havoc. OS/2 seems like a brilliant choice in that regard. Questions left unaddressed: - What tech did they move to now with contactless payments? - The move to contactless payments is awesome but what happens to people who lack smartphones or do not have credit cards or <big-tech>-pay setup on their phones. Or just casual tourists visiting from overseas with no US cards. How do they pay? Do they keep supporting metrocards forever? Does that mean to support Metrocards they need OS/2 and assorted new systems. - What is the mainframe part of the stack and what function does it do.

cnorthwood 4y ago
I was recently in New York as a Brit and was really happy to see the OMNY system which just allowed me to use contactless to traverse the turnstiles (for everything but the JFK SkyTrain) with no issues, similar to how metro transport works in the UK for example. I don't think there's a requirement for it to be a US card, but instead any contactless card will work?
- wikibob 4y ago
  As it turns out: “The OMNY system is designed by Cubic Transportation Systems, using technology licensed from Transport for London's Oyster card.” https://www.cubic.com/news-events/news/cubic-wins-contract-new-york-mta-replace-iconic-metrocard-system-world-class-new https://www.cubic.com/news-events/news/cubic-wins-contract-n...
- deleted 4y ago
  [deleted]

dflock 4y ago

OS/2. Also used for Vancouver's SkyTrain system, afaik.

manv1 4y ago

OS/2 was known to be rock-solid back in the day. It was the OS of choice for a lot of financials. Everyone moved to Windows NT when 4 came out, though, because of the way Microsoft did licensing.

bogomipz 4y ago
>"Everyone moved to Windows NT when 4 came out, though, because of the way Microsoft did licensing." Can you elaborate on what the difference was between how the licensing worked on OS/2 vs NT 4? What was the big innovation?
- giamma 4y ago
  Microsoft in 1999 was not the same company it is today and did some very unethical things. For example: https://www.itprotoday.com/windows-78/ibm-microsoft-warned-pc-makers-against-using-os2 https://www.itprotoday.com/windows-78/ibm-microsoft-warned-p... Other articles report that computer producers were given Windows "for free" as long as they did not offer nor support OS/2 on their hardware. Microsoft applied this practice for a few years and indeed lost income by giving Windows for free, but it ensured that the regular user never heard of OS/2 and then when IBM gave up the situation changed. Besides, Microsoft would still make money from Office and other products.
  - bogomipz 4y ago
    Your link is about Windows 95. My question about what was different about the licensing model of NT4 that prompted companies to switch. NT 4 also came out in the mid 90s.

supernova87a 4y ago

If you want a story into the other "operating system" that keeps the NYC subway system running (or from another point of view, never advancing), see this: https://www.theatlantic.com/technology/archive/2015/11/why-dont-we-know-where-all-the-trains-are/415152/ https://www.theatlantic.com/technology/archive/2015/11/why-d... "Why New York Subway Lines Are Missing Countdown Clocks" and how it's actually very about how managing a software/IT project can go wrong.

ncmncm 4y ago
The key to all big projects that work well is that they start with small projects that work well, and deliver something useful immediately, adding more continuously. They don't need to stay with equipment they delivered at first; they just need to keep what visibly works working. They can replace older equipment and software with what works better in a bigger system, as they grow. If that seems more expensive, look at the hundred-million dollar+ contracts. Electronics is cheap, under those numbers.

dtx1 4y ago

> The Forgotten Operating System That Keeps the NYC Subway System Alive It's OS/2. Hardly forgotten.

doublerabbit 4y ago
Give it a few more years. The iconic icon of a Floppy Disk has almost faded out of existance.
lproven 4y ago
I submitted the article about a week ago, because HN had just discovered Blue Lion and crashed Arca Noae's server. A commenter asked for an example of someone, anyone, still using OS/2 in production. I remembered this story and offered it as an answer of someone using it as recently as 3Y ago. So I posted it as a top-level story as well. I guess HN re-upped it and it caught on. But yes, I think that for millennial techies, this is something that came and went before they were in the industry, so for them, it is forgotten tech that they've probably never seen.

someweirdperson 4y ago

Chinas railway is running on flash player [0]. At least that choice doesn't limit it to one specific OS. [0] https://news.ycombinator.com/item?id=25786670 https://news.ycombinator.com/item?id=25786670

jackblemming 4y ago

How did this scale without kubernetes, docker, and microservices? How does this work without transpiling from TypeScript to another language? This wont scale without type safety!! Does it even have a JIRA backlog and dedicated product manager? No wonder this system is so mediocre. It probably wasnt even made following Agile guidelines!

bpodgursky 4y ago
The NY metro is infamously unable to scale, in large part because of extraordinary tech debt! The cost of each new line is stratospheric! They use century-old manual track routing cables! You might think this is a clever take but it's really not far from the truth. They NEED a revamp on a modern tech stack so they can use the trains, electronics, and technology trains in the rest of the world run on.
- woodruffw 4y ago
  The subway’s issues are mostly physical, not in software: the city (really, the state) contracts virtually everything around fare collection out to Cubic, and they do an okay enough job. See for example the OMNY card, which rolled out surprisingly fast. In other words: the things that are currently falling apart (the physical relays, for example) are almost completely disconnected from fare collection (which is what the article’s about). The latter runs pretty well; replacing it with a modern tech stack rather than the current incremental approach wouldn’t have many benefits.
- devwastaken 4y ago
  In 20 years all current software stacks will be legacy and equally if not more unusable. We rely significantly more now on tooling to just "do it for us". That tooling relies heavily on the OS it runs on and makes assumptions.
  - chinabot 4y ago
    If you diddnt do updates I seriously doubt the stack would survive 2 years. The joys of modern interconnected systems

tibbydudeza 4y ago

If it takes years and billions to build a simple high speed rail line from San Francisco to LA then NYC subway this will still be around in it's current state.

Meph504 4y ago

You would be shocked at the number of ATM machines that run on OS/2, and though I don't advocate it, it makes sense in a way. When is the last exploit you saw published for OS/2, many of the old OS/2 machines direct dial into an old modem bank (at a bank.) The thread window on these is tiny because of how far behind they are.

AnnoyedComment 4y ago

mmmpop 4y ago

Didn't realize people forgot about OS/2?

lproven 4y ago
See my reply above.