7 ms·

How a Single Mechanical Failure Sparked 625 MTA Delays (2016)

62 points by CaptainZapp 9y ago

THE_PUN_STOPS 9y ago
“‘For 67 percent of the railroad’ — that is, every lettered train line except the L — ‘we don’t actually see train movement or control any signals and switches from the control center.’ Instead, they do it the same way they’ve been doing it for decades: train crews communicating by radio with a dispatcher. If there’s a delay, the dispatcher phones it in on the ‘6 wire,’ an open party line, and awaits instructions.” Wow. I knew the system was old and rickety but I didn’t realize it was this old and rickety.
- deleted 9y ago
  [deleted]
- lb1lf 9y ago
  The tram line in Trondheim, Norway (orders of magnitude smaller than anything operated by the MTA, but still...) had a signaling system operated by the drivers on a single-track line - on a pole next to the track there was a small bucket. When you rode uphill, you grabbed a token as the tram passed. Empty bucket meant thou shalt not pass. When you came back down again (leaving a vacant track behind), you dumped the token back in the bucket. This system was in use at least until I left town in 2005.
  - jacquesm 9y ago
    I love the simplicity of that solution, but the hacker in me wonders how hard it would be to duplicate the token...
    - TeMPOraL 9y ago
      A cracker in me wonders what would happen if some random person just came up and stole the token...
      - lb1lf 9y ago
        I believe the assessment at the time was that the speed was so low on this track anyway that a missing token would be an inconvenience more than an outright hazard - tram drivers getting into staring contests and the like. ("You back off!" "No, you back off!")
    - jboles 9y ago
      It is usually engraved with the two stations it is valid between and shaped differently to prevent confusion with adjacent segment's tokens. Besides, it is in the train crew's own interests to be holding the one true token, lest they be the ones unauthorized to be on the single track line and therefore the cause of an accident... https://en.m.wikipedia.org/wiki/Token_(railway_signalling) https://en.m.wikipedia.org/wiki/Token_(railway_signalling)
  - Animats 9y ago
    Token systems have been widely used on single track lines for over a century. The UK probably still has some.[1] It's for very low traffic lines only. [1] https://en.wikipedia.org/wiki/Token_(railway_signalling) https://en.wikipedia.org/wiki/Token_(railway_signalling)
katastic 9y ago
>the late ’70s, before a capital program rescued it from crumbling entirely — the system had about 320,000 recorded delays a year. From March 2013 through March 2014, according to the New York State Comptroller’s most recent audit, that number was 498,889. That's... really not much of an increase over >>45<< years of population increase. It's not "good", but it's certainly not as bad as plenty of public transport systems around the world.
bob_theslob646 9y ago
>MTA executives are naturally defensive about the criticism. They argue that, unlike in the ’70s, the current problems are a result of their own success — the subways are more popular than ever and therefore more crowded. Aren't these the same people that got caught destroying data so that people could not check the efficiency of management?
- LeifCarrotson 9y ago
  The current problems are a result of the city's success in spite of the MTA. Tourism and commuting population have both increased more than the ridership on the subway, which shows that the MTA is holding the city back.
nerdponx 9y ago
There are lots of small improvements that could be made to this system. Passengers are often left in the dark about what is happening, and so can't make informed decisions about which train to take, or whether to take the train at all. Announcements are rare except when trains are to be rerouted. The overhead signs that report arrival time estimates and delays are not always visible from the turnstile. And on the Lexington Ave line local track, most of those signs do not show the express train times, so transferring from the local to the express is always a gamble.
- pavel_lishin 9y ago
  > The overhead signs that report arrival time estimates and delays are not always visible from the turnstile. I would say that visibility-from-the-turnstile is the exception rather than the rule. Sometimes it's not even obvious where one of the two (you're lucky if there are two) overhead signs are, and you just have to wander down the platform until you see it. My favorite overhead sign position is either on the 81st or 86th stop on the BC line - it's the only one, and it's positioned right in front of the exit-only turnstile in the middle of the platform, and is completely invisible from the entrances.
- rbcgerard 9y ago
  It drives me crazy that train info isn’t out at street level
  - L_Rahman 9y ago
    They're starting to resolve this in some stations. Many Manhattan subway entrances now have boards above them showing the arrival times for the next two trains.
- luckyt 9y ago
  There's the Transit app, which tells you how long you have to wait for the next train. If more people used it, people would know about delays before entering the platform, and only take the train if they absolutely have to.
pavel_lishin 9y ago
I've had a lot of luck using this unofficial web-app: https://wheresthefuckingtrain.com/ https://wheresthefuckingtrain.com/
- iooi 9y ago
  Interesting, is it using different data than the official subway time app[1]? I use a script to check the times the trains arrive so I can leave the office at an optimal time. Something like: echo " #!/bin/bash curl -s http://traintimelb-367443097.us-east-1.elb.amazonaws.com/getTime/W/R18 http://traintimelb-367443097.us-east-1.elb.amazonaws.com/get... | jq '.direction1.times' " > /usr/local/bin/train && chmod +x /usr/local/bin/train It works really well. [1] http://subwaytime.mta.info/ http://subwaytime.mta.info/
  - pavel_lishin 9y ago
    Good question - I have no idea! I did find it funny that it's almost always returning data, even when the overhead signs are blank. (Also, howdy, neighbor - I leave the office and go to the 23rd street stop.)
dfraser992 9y ago
I have recently been doing a literature survey of research into train timetabling and re-scheduling for a potential contract (using machine learning et al). Dealing with train delays in real time is not an easy problem to solve; it's pretty much a NP complete problem to find the most optimal solution. So 'the best so far' is the goal given time constraints. So I'm not surprised at all about how one issue went and cascaded. Managing real world train networks is pretty complicated because of the interconnection of so many things - the train schedules, the crew schedules, managing the rolling stock, the passengers... and the profit/loss of the train company.
- jloughry 9y ago
  If you've not already found it, The Visual Display of Quantitative Information by Edward Tufte (Graphics Press, 1991) talks a lot about the representation of graphical train schedules by the slope of a line between stations; the intersection of two lines on the graph shows the time and place where two trains meet going in opposite directions. This was in the nineteenth century.
closeparen 9y ago
One thing I don’t see acknowledged in the enthusiasm for rail-based public transit is its propensity for cascading failure. A road network has the very nice property that a disabled vehicle can coast off the road, or only block a subset of lanes, leading to a “graceful” degradation and creating backpressure that encourages other drivers to take different routes, which exist. It isn’t fun, but it’s a pretty amazing resilience story compared to alternatives. Many commodity parts, failing in ways that the system as a whole recovers from. You would think software engineers would be more positive about that. We build train systems in such a way that if any component fails, the entire network is fucked. There’s only one way to get between A and B, and if it’s blocked by an emergency situation or the failure of some bespoke part, no trains will pass until the situation is resolved. From a distributed systems perspective, that’s horrifically bad engineering. Maybe there’s some way to make rail networks more road-like in this way, or to build better bus systems that people will enjoy and embrace, but as it stands, the differences in failure modes motivate drivers to stay in their cars.
- avianlyric 9y ago
  Rail based transport on cities makes a lot of sense. In cities usually the only way to significantly increase transport capacity it to dig tunnels. Building on the surface is just too difficult. When you build tunnels your cost proportional to the volume of earth you remove, which means your cost is the square of your tunnel diameter. Double the diameter, quadruple the cost. This is where rail shines. Rail let’s you run faster, longer, vehicles in much smaller spaces, because the rails guarantee you won’t collide with the walls or each other. Just look at the London tube, with trains only slightly smaller than the tunnels. This means smaller cheaper tunnels, so you can build more of them. In this system the tunnels already eliminate any real ability to reroute, so you may as well put the biggest vehicles in there and maintain them properly.
  - closeparen 9y ago
    A system experiencing cascading failure has no capacity, and at this point it should be considered engineering malpractice to imagine that American democratic institutions will do the maintenance on anything, in the face of overwhelming evidence to the contrary.
dkarl 9y ago
What strikes me is the massive difference in functionality between the system when it is running smoothly and when it hits a snag, and how people consider the very best case to be baseline against which performance should be judged. Perhaps operations can be improved, but pending those improvements, the system is essentially running overoptimistically, pretending that its performance when everything is working right reflects its true capability more than its usual performance does. Many people have made this point, but the quote I have to hand is from Rich Hickey: "In the real world, failures are all the time. A large system is in a state of partial failure almost continuously." In a system like this, there's no point in improving the best case or even the most common case. In fact, it can be counterproductive, because it makes it hard for riders to figure out how to plan their trips, and when they do figure out the best way to use the system (getting to work an hour early many days to ensure you're consistently on time) they resent it. Also, if you perform acceptably most of the time, people are going to assume that you're just a few tweaks or a modest increase in capacity away from performing acceptably all the time, which can be very far from the truth.
quietbritishjim 9y ago
> “Their nightmare scenario,” says Gene Russianoff of the Straphangers Campaign, the venerable subway gadfly group, “is there are these people who are like bouncers who are standing in front of the staircase at Grand Central and put up a felt line and say, ‘You can’t go down for 15 minutes, because it’s too crowded down there.’ And it’s not an insane, paranormal phenomenon. It happens in London. And it would be terrible.” I've had this happen while commuting in London, and it is definitely annoying. But it is not "terrible" and certainly not "insane" or a "nightmare scenario". Surely it is better than the vicious cycle of overcrowding forcing longer dwell times forcing more overcrowding? And that's not even taking into account the obvious danger of overcrowded platforms, with people at the back wanting to get to the front, and people at the front teetering over electrified rails as a train approaches.
- s0rce 9y ago
  This seems basically the same concept as metering lights at freeway on ramps. In the end waiting a few moments can prevent overcrowding make it faster for everyone.
- Jesus_Jones 9y ago
  I've always wondered, why don't they have fences a few feet from the edge of the platform, with many open gates? Then you could stand at the fence without being able to be pushed into the train's path. Since they aren't on the edge, you could get off the train okay. It would reduce free movement but would not matter unless there was heavy traffic - and in that case, it would be welcomed, by me at least. Let's see if I can do this with ascii art, just for fun: [trains, edge of tracks below] ========================================================== --- -------- -------- -------- ------- fence line (people, crowds .................................)
  - kuschku 9y ago
    What you’re suggesting would be a system similar to what is used today on rollercoasters.
    - 9y ago
Animats 9y ago
There are only gap fillers at two NYC subway stations, Union Square and old South Ferry. Since the new South Ferry station is open again (flooded in 2012), the old curved platform at South Ferry isn't being used any more. Those gap fillers are a tough piece of mechanical engineering. They have nearly a foot of travel, because they're on a sharply curved section of track. Some other subway systems use gap fillers, but they're usually much smaller, and are sometimes just passive rubber blocks. Gap fillers get banged by the train as it leaves the station, rather than being retracted before the train moves. That's to prevent dumping people onto the tracks. So gap fillers have to be very tough, and they have to be pushable. That means no electric ball screws or hydraulics; they have to be pneumatic. They have to work at temperatures from below freezing to above 100F. They have to resist ice. They're stepped on by people and people can drop things through the slats, so they have a potential jamming problem. They're long sliding devices, always a headache. They're long enough that being pushed near one end when the air cylinders don't respond fast enough may force them back misaligned and jam them, which is probably the failure mode that caused the problem here. Union Square's gap fillers are a one-off; there are no other copies of that design. If there were lots of those things, different designs would be tried and tested, as is normal for rugged mechanical design.

62 points by CaptainZapp 9y ago

“‘For 67 percent of the railroad’ — that is, every lettered train line except the L — ‘we don’t actually see train movement or control any signals and switches from the control center.’ Instead, they do it the same way they’ve been doing it for decades: train crews communicating by radio with a dispatcher. If there’s a delay, the dispatcher phones it in on the ‘6 wire,’ an open party line, and awaits instructions.” Wow. I knew the system was old and rickety but I didn’t realize it was this old and rickety.

deleted 9y ago
[deleted]
lb1lf 9y ago
The tram line in Trondheim, Norway (orders of magnitude smaller than anything operated by the MTA, but still...) had a signaling system operated by the drivers on a single-track line - on a pole next to the track there was a small bucket. When you rode uphill, you grabbed a token as the tram passed. Empty bucket meant thou shalt not pass. When you came back down again (leaving a vacant track behind), you dumped the token back in the bucket. This system was in use at least until I left town in 2005.
- jacquesm 9y ago
  I love the simplicity of that solution, but the hacker in me wonders how hard it would be to duplicate the token...
  - TeMPOraL 9y ago
    A cracker in me wonders what would happen if some random person just came up and stole the token...
    - lb1lf 9y ago
      I believe the assessment at the time was that the speed was so low on this track anyway that a missing token would be an inconvenience more than an outright hazard - tram drivers getting into staring contests and the like. ("You back off!" "No, you back off!")
  - jboles 9y ago
    It is usually engraved with the two stations it is valid between and shaped differently to prevent confusion with adjacent segment's tokens. Besides, it is in the train crew's own interests to be holding the one true token, lest they be the ones unauthorized to be on the single track line and therefore the cause of an accident... https://en.m.wikipedia.org/wiki/Token_(railway_signalling) https://en.m.wikipedia.org/wiki/Token_(railway_signalling)
- Animats 9y ago
  Token systems have been widely used on single track lines for over a century. The UK probably still has some.[1] It's for very low traffic lines only. [1] https://en.wikipedia.org/wiki/Token_(railway_signalling) https://en.wikipedia.org/wiki/Token_(railway_signalling)

katastic 9y ago

>the late ’70s, before a capital program rescued it from crumbling entirely — the system had about 320,000 recorded delays a year. From March 2013 through March 2014, according to the New York State Comptroller’s most recent audit, that number was 498,889. That's... really not much of an increase over >>45<< years of population increase. It's not "good", but it's certainly not as bad as plenty of public transport systems around the world.

bob_theslob646 9y ago

>MTA executives are naturally defensive about the criticism. They argue that, unlike in the ’70s, the current problems are a result of their own success — the subways are more popular than ever and therefore more crowded. Aren't these the same people that got caught destroying data so that people could not check the efficiency of management?

LeifCarrotson 9y ago
The current problems are a result of the city's success in spite of the MTA. Tourism and commuting population have both increased more than the ridership on the subway, which shows that the MTA is holding the city back.

nerdponx 9y ago

There are lots of small improvements that could be made to this system. Passengers are often left in the dark about what is happening, and so can't make informed decisions about which train to take, or whether to take the train at all. Announcements are rare except when trains are to be rerouted. The overhead signs that report arrival time estimates and delays are not always visible from the turnstile. And on the Lexington Ave line local track, most of those signs do not show the express train times, so transferring from the local to the express is always a gamble.

pavel_lishin 9y ago
> The overhead signs that report arrival time estimates and delays are not always visible from the turnstile. I would say that visibility-from-the-turnstile is the exception rather than the rule. Sometimes it's not even obvious where one of the two (you're lucky if there are two) overhead signs are, and you just have to wander down the platform until you see it. My favorite overhead sign position is either on the 81st or 86th stop on the BC line - it's the only one, and it's positioned right in front of the exit-only turnstile in the middle of the platform, and is completely invisible from the entrances.
rbcgerard 9y ago
It drives me crazy that train info isn’t out at street level
- L_Rahman 9y ago
  They're starting to resolve this in some stations. Many Manhattan subway entrances now have boards above them showing the arrival times for the next two trains.
luckyt 9y ago
There's the Transit app, which tells you how long you have to wait for the next train. If more people used it, people would know about delays before entering the platform, and only take the train if they absolutely have to.

pavel_lishin 9y ago

I've had a lot of luck using this unofficial web-app: https://wheresthefuckingtrain.com/ https://wheresthefuckingtrain.com/

iooi 9y ago
Interesting, is it using different data than the official subway time app[1]? I use a script to check the times the trains arrive so I can leave the office at an optimal time. Something like: echo " #!/bin/bash curl -s http://traintimelb-367443097.us-east-1.elb.amazonaws.com/getTime/W/R18 http://traintimelb-367443097.us-east-1.elb.amazonaws.com/get... | jq '.direction1.times' " > /usr/local/bin/train && chmod +x /usr/local/bin/train It works really well. [1] http://subwaytime.mta.info/ http://subwaytime.mta.info/
- pavel_lishin 9y ago
  Good question - I have no idea! I did find it funny that it's almost always returning data, even when the overhead signs are blank. (Also, howdy, neighbor - I leave the office and go to the 23rd street stop.)

dfraser992 9y ago

I have recently been doing a literature survey of research into train timetabling and re-scheduling for a potential contract (using machine learning et al). Dealing with train delays in real time is not an easy problem to solve; it's pretty much a NP complete problem to find the most optimal solution. So 'the best so far' is the goal given time constraints. So I'm not surprised at all about how one issue went and cascaded. Managing real world train networks is pretty complicated because of the interconnection of so many things - the train schedules, the crew schedules, managing the rolling stock, the passengers... and the profit/loss of the train company.

jloughry 9y ago
If you've not already found it, The Visual Display of Quantitative Information by Edward Tufte (Graphics Press, 1991) talks a lot about the representation of graphical train schedules by the slope of a line between stations; the intersection of two lines on the graph shows the time and place where two trains meet going in opposite directions. This was in the nineteenth century.

closeparen 9y ago

One thing I don’t see acknowledged in the enthusiasm for rail-based public transit is its propensity for cascading failure. A road network has the very nice property that a disabled vehicle can coast off the road, or only block a subset of lanes, leading to a “graceful” degradation and creating backpressure that encourages other drivers to take different routes, which exist. It isn’t fun, but it’s a pretty amazing resilience story compared to alternatives. Many commodity parts, failing in ways that the system as a whole recovers from. You would think software engineers would be more positive about that. We build train systems in such a way that if any component fails, the entire network is fucked. There’s only one way to get between A and B, and if it’s blocked by an emergency situation or the failure of some bespoke part, no trains will pass until the situation is resolved. From a distributed systems perspective, that’s horrifically bad engineering. Maybe there’s some way to make rail networks more road-like in this way, or to build better bus systems that people will enjoy and embrace, but as it stands, the differences in failure modes motivate drivers to stay in their cars.

avianlyric 9y ago
Rail based transport on cities makes a lot of sense. In cities usually the only way to significantly increase transport capacity it to dig tunnels. Building on the surface is just too difficult. When you build tunnels your cost proportional to the volume of earth you remove, which means your cost is the square of your tunnel diameter. Double the diameter, quadruple the cost. This is where rail shines. Rail let’s you run faster, longer, vehicles in much smaller spaces, because the rails guarantee you won’t collide with the walls or each other. Just look at the London tube, with trains only slightly smaller than the tunnels. This means smaller cheaper tunnels, so you can build more of them. In this system the tunnels already eliminate any real ability to reroute, so you may as well put the biggest vehicles in there and maintain them properly.
- closeparen 9y ago
  A system experiencing cascading failure has no capacity, and at this point it should be considered engineering malpractice to imagine that American democratic institutions will do the maintenance on anything, in the face of overwhelming evidence to the contrary.

dkarl 9y ago

What strikes me is the massive difference in functionality between the system when it is running smoothly and when it hits a snag, and how people consider the very best case to be baseline against which performance should be judged. Perhaps operations can be improved, but pending those improvements, the system is essentially running overoptimistically, pretending that its performance when everything is working right reflects its true capability more than its usual performance does. Many people have made this point, but the quote I have to hand is from Rich Hickey: "In the real world, failures are all the time. A large system is in a state of partial failure almost continuously." In a system like this, there's no point in improving the best case or even the most common case. In fact, it can be counterproductive, because it makes it hard for riders to figure out how to plan their trips, and when they do figure out the best way to use the system (getting to work an hour early many days to ensure you're consistently on time) they resent it. Also, if you perform acceptably most of the time, people are going to assume that you're just a few tweaks or a modest increase in capacity away from performing acceptably all the time, which can be very far from the truth.

quietbritishjim 9y ago

> “Their nightmare scenario,” says Gene Russianoff of the Straphangers Campaign, the venerable subway gadfly group, “is there are these people who are like bouncers who are standing in front of the staircase at Grand Central and put up a felt line and say, ‘You can’t go down for 15 minutes, because it’s too crowded down there.’ And it’s not an insane, paranormal phenomenon. It happens in London. And it would be terrible.” I've had this happen while commuting in London, and it is definitely annoying. But it is not "terrible" and certainly not "insane" or a "nightmare scenario". Surely it is better than the vicious cycle of overcrowding forcing longer dwell times forcing more overcrowding? And that's not even taking into account the obvious danger of overcrowded platforms, with people at the back wanting to get to the front, and people at the front teetering over electrified rails as a train approaches.

s0rce 9y ago
This seems basically the same concept as metering lights at freeway on ramps. In the end waiting a few moments can prevent overcrowding make it faster for everyone.
Jesus_Jones 9y ago
I've always wondered, why don't they have fences a few feet from the edge of the platform, with many open gates? Then you could stand at the fence without being able to be pushed into the train's path. Since they aren't on the edge, you could get off the train okay. It would reduce free movement but would not matter unless there was heavy traffic - and in that case, it would be welcomed, by me at least. Let's see if I can do this with ascii art, just for fun: [trains, edge of tracks below] ========================================================== --- -------- -------- -------- ------- fence line (people, crowds .................................)
- kuschku 9y ago
  What you’re suggesting would be a system similar to what is used today on rollercoasters.
  - 9y ago

Animats 9y ago

There are only gap fillers at two NYC subway stations, Union Square and old South Ferry. Since the new South Ferry station is open again (flooded in 2012), the old curved platform at South Ferry isn't being used any more. Those gap fillers are a tough piece of mechanical engineering. They have nearly a foot of travel, because they're on a sharply curved section of track. Some other subway systems use gap fillers, but they're usually much smaller, and are sometimes just passive rubber blocks. Gap fillers get banged by the train as it leaves the station, rather than being retracted before the train moves. That's to prevent dumping people onto the tracks. So gap fillers have to be very tough, and they have to be pushable. That means no electric ball screws or hydraulics; they have to be pneumatic. They have to work at temperatures from below freezing to above 100F. They have to resist ice. They're stepped on by people and people can drop things through the slats, so they have a potential jamming problem. They're long sliding devices, always a headache. They're long enough that being pushed near one end when the air cylinders don't respond fast enough may force them back misaligned and jam them, which is probably the failure mode that caused the problem here. Union Square's gap fillers are a one-off; there are no other copies of that design. If there were lots of those things, different designs would be tried and tested, as is normal for rugged mechanical design.