10 ms·
How MTA shut down my app for Penn Station commuters
- mjmasn 9y agoWould have been interesting to see if you could reliably crowd-source the data from commuters themselves.
- parm289 9y agoWas going to post this - have users submit the track data! That also might make it easier to expand to other stations where you aren't as familiar with the track layout.
- dmurray 9y agoGoogle almost certainly already could, without even asking the commuters, since they have real time location for so many Android phones. But there's a chicken and egg problem: it's hard to get the data before you're able to offer anything in return.
- mjmasn 9y agoChicken and egg yes, but he had the recent historical data and a bunch of paid-up users. I'm just amazed this app was even necessary in a first-world transport system. For all their faults, the UK rail network is pretty good at providing timely information (and as some other comments have mentioned have solid APIs for this stuff). Same with the LTA[1] in Singapore. [1] https://www.mytransport.sg/content/mytransport/home/dataMall.html https://www.mytransport.sg/content/mytransport/home/dataMall...
- ethryx 9y agoThis is something that Clever Commute does and it works really well!
- deleted 9y ago[deleted]
- zavulon 9y agoThank you! I definitely considered doing something like that, however — the amount of work required to crowd source the data would be significant. There would be a “chicken & egg” problem — the app wouldn’t be useful to first users, and without the first users, there wouldn’t be enough data to make it useful for everyone else — so this would require a fairly large marketing campaign to get off the ground. But — one day I may still do it :)
- natch 9y agoThere's hardly a chicken & egg problem if you already had a bunch of happy users. Just display "no data available" on the first day, and have a banner in the app explaining the situation, and ask them to contribute today's data. You'll most certainly get a handful of users who want to help. After 3 or so days of this, the problem will be solved.
- drinchev 9y agoWhy the OP did not continue with scraping the website then? I think there will be no legal issue if he does that.
- rayboy1995 9y ago"They also stopped publishing the track number on the website — so going back to scraping was not an option either."
- karrotwaltz 9y ago> They also stopped publishing the track number on the website — so going back to scraping was not an option either. So the only way to gather the data would be manually from the station.
- drinchev 9y agoAhhh.. Missed that. Thanks!
- Sir_Cmpwn 9y agoCarefully placed raspberry pi + OCR?
- vinnycoyne 9y agoI'd imagine that placing a "homemade" electronic device in a densely-populated, public space is going to attract a lot of unwanted attention.
- laylomo2 9y agoFrom all those military guards that parade up and down the station
- InitialLastName 9y agoYou joke, but Penn Station and Port Authority are effectively occupied territory. There are at least four different law enforcement agencies present and patrolling, stopping and frisking anybody who "looks out of place", hassling homeless people, and generally making you feel "safer".
- state_machine 9y agoI can see why the MTA wouldn't want you telling people to go to a track before it is actually assigned -- if the train ends up on a different track, now you need to get all those people back up, off that platform an on to the new one, clogging stairs that that people who actually wanted that platform might be trying to use too. Obviously it would be nice to assign tracks earlier, so that people can head straight to the right platform, but sending people to potentially the wrong platform seems even worse. EDIT: Penn is extremely platform/track constrained -- NJ Transit, LIRR and Amtrak are all sharing a fixed number of platforms, some of which are too short. To maximize platform utilization, they have to wait until the last minute to finalize track assignments -- if you reserve one too early and the train ends up late, you're wasting an empty platform. Once you send a horde of people to a platform, moving them to a different one is a challenge (stairs/bottlenecks, communication, etc).
- FussyZeus 9y agoThing is though, they could be providing that data MUCH earlier both on the displays and to a phone app. It's not like they don't decide where a train is going until it gets to the station, that would be ridiculous, they'd have to know a while in advance to make sure the track is available, clear, operating correctly, and do all the necessary switching (and more importantly, do so in such a way that doesn't impact other trains). MTA's appeared a number of times on this website and by all accounts it's where tech innovation goes to die, most of their service is still based on switch wire systems that were built in the 60's. Whether that's due to bureaucratic inertia, inadequate funding, or unions or whatever I have no idea, but that's the real problem.
- dreamcompiler 9y agoBased on the other comments here, I'm guessing they don't post train numbers until the arriving train has disembarked so its passengers can clear the platform before the horde arrives. It probably has nothing to do with technology.
- s73ver 9y agoGiven the other comments on this issue, it's not so much that MTA hates innovation, but that those who would be "disruptors" don't actually understand the entirety of the problem.
- lukegb 9y agoAs an interesting, although unrelated aside: platform data is available in the UK's National Rail's feeds, but the terms and conditions[1] explicitly prohibit displaying platform numbers early, as mentioned in their developer guidelines[2]. I guess the wording doesn't technically ban you from displaying historical platform information, but that would likely be a bad-faith use of the data anyway... [1] http://www.nationalrail.co.uk/static/documents/Terms_and_Conditions.pdf http://www.nationalrail.co.uk/static/documents/Terms_and_Con... [2] http://www.nationalrail.co.uk/static/documents/Developer_Guidelines.pdf http://www.nationalrail.co.uk/static/documents/Developer_Gui... "Occasionally Time-Bound Data will become available through NRE feeds before it is ready to be published to the public. [...] One such example of Time-Bound Data is platform numbers. Early display of platform numbers, particularly at origin and destination stations, can lead to platform overcrowding and/or staff not having sufficient time to prepare the train for oncoming passengers. In some instances, platform numbers will be available in Darwin before being displayed on screens in stations."
- manarth 9y agoOnce upon a time, the platform used to be displayed in the National Rail app, even when it wasn't shown on the departure boards…For a long time I used that to good effect to get a seat on rush-hour trains at Paddington. That changed a few years ago, interesting to see the formal policy behind it, and an idea for a side-project :-)
- davidjbain 9y agoUK rail data is quite open in comparison to this story, for instance check out these live track diagrams (of London Waterloo & many other areas) which show real-time movements of trains. This is how I often find out the platform numbers prior to travel and I also get information before staff during disruption. http://www.opentraintimes.com/maps/signalling/wat#T_WAT http://www.opentraintimes.com/maps/signalling/wat#T_WAT
- takno 9y agoThere are maps of many more routes at traksy https://traksy.uk/live/M+2+CARLILE https://traksy.uk/live/M+2+CARLILE
- falcolas 9y agoScraping the MTA site against the TOS, and intentionally obfuscating requests to limit the chance of the MTA identifying and shutting it down? Yeah, no surprises here. EDIT: Oops, missed that he didn't end up going live with the scraping. It would still be great for the MTA to publish this data intentionally; but I'm sure the potential for trains to switch tracks and the associated backlash when they do is what prevents them from doing it. Perhaps they could adopt the airline model, where there's a best effort to reach a particular pre-announced track, with notifications ringing out when they can't.
- teddythetwig 9y agoPretty sure that if you read a little further, you will find that the author decided to NOT go with the TOS violating process, and instead used an official MTA API.
- drinchev 9y agoThey could publish the log of passed trains, though. This will hurt nobody.
- boobsbr 9y ago> Scraping the MTA site against the TOS Where does it say it was against the TOS?
- Sir_Cmpwn 9y agoThe author didn't do that, though. As far as MTA is concerned the app was always using an API he had been given a legitimate key for. As an aside, fuck websites that prohibit scraping. If you send me some bytes I'm gonna do whatever I damn well please with them.
- EGreg 9y agoIn that case screw your API, if you allow DDOS I may as well do it for the lulz. Isn't it the same approach? If you can abuse a site you will, because they opened themselves up to the public?
- midnitewarrior 9y agoWhy didn't he switch to a crowd-sourced data model like GasBuddy?
- sologoub 9y agoWas thinking the exact same thing. One possible reason would be wrong info, but you could use some sort of weighted algorithm that would need more than one submission to match and/or trust level per submitter based on historical correctness.
- tedmiston 9y agoExactly. Like what Yelp / Foursquare / Google Places use.
- s73ver 9y agoHow do you get people to input the data in time for it to be useful?
- atestu 9y agoBuild a reward system… reputation might be enough?
- zavulon 9y agoSee my comment above - it would require a significant effort to get off the ground, as the app without data would not be useful to first users, and without the data provided by those first users, it wouldn't be useful to others.
- Pyxl101 9y agoThey already had an app and users. They could have extended it with this functionality.
- rocky1138 9y agoWhoever decided to close his access to the API should be fired.
- danesparza 9y agoIt sounds like The API changed. But same sentiment -- that was a shit design decision.
- zavulon 9y agoThe API didn't change, they just made it private "for MTA use only"
- strken 9y agoIt sounds like locking down the API was an entirely deliberate decision made at the same time as removing platform data from the website, which means Chesterton's fence[0] applies - we should work out why that decision was made before criticising it. Discussion upthread is about commuters being pushed onto the tracks, which sounds like a good reason not to display data that isn't 100% certain. [0] https://en.m.wikipedia.org/wiki/Wikipedia:Chesterton's https://en.m.wikipedia.org/wiki/Wikipedia:Chesterton's fence
- camiller 9y agohow about https://en.wikipedia.org/wiki/G._K._Chesterton#Chesterton.27s_fence https://en.wikipedia.org/wiki/G._K._Chesterton#Chesterton.27...
- rocky1138 9y ago> we should work out why that decision was made Good luck with that. I'd be amazed if you got a response.
- 9935c101ab17a66 9y agoI've never heard of "Chesterton's Fence" before and appreciate the introduction to the term. That said, I don't think your application is correct. The OP in this case is not the agent of change — he did not "reform" anything and does not have the capacity to act, he just analyzed/synthesized existing data. (Also, on reflection, "Chesterton's Fence" is an unnecessarily complicated and overly verbose way of saying: "The MTA probably has a reason for the change.")
- DontSueMeBro 9y agoI've been collecting Penn Station NJT and LIRR track assignment data for almost a year now. I've been intending to use ML to predict track assignment, but haven't gotten there yet. I should post it to Kaggle or somewhere similar.
- jrockway 9y agoDo you think the track assignment is something that's learnable? I feel like there is some schedule where, under normal operations, the train will end up. But then there are unpredictable events that cause this to change.
- DontSueMeBro 9y agoI think it is. Like in the article, you can see that given certain parameters like train number and time, you predictably end up with a certain track assignment. If you can also consider track assignments leading up to this time, you can discount those tracks. For example, the 8:02PM Long Beach train always leaves on track 18, except when any 7:56 train left on track 18. As a novice to ML it's currently beyond my ability to model, but my intuition is that it should be doable.
- bradleyjg 9y agoFor about a year a decade and a half ago, I commuted into and out of Penn Station with my father who had been doing so since the late 70s. Not only did he know which track any of a half dozen trains that he might take would come in on but he knew where the doors would open for each of the tracks. And he wasn't the only one either. If you went down before they announced the tracks you'd see little clusters of people waiting apart from each other on an otherwise empty track. We'd usually go to the same car in order to reduce the distance on the other side. Other commuters would do likewise and so trains would have a contingent of regulars. There's a fascinating kind of micro-expertise that develops when you do the same thing over and over again.
- Symbiote 9y agoThe London version is now an app. I think it used to be available as some kind of map. http://www.tubeexits.co.uk/ http://www.tubeexits.co.uk/
- genmon 9y agoIt's also a feature in Citymapper. Only front/middle/back but the integration means I use the information more often https://citymapper.com/news/726/where-to-get-on-the-train https://citymapper.com/news/726/where-to-get-on-the-train
- mark212 9y agosame thing on the subway, with an extra dimension of starting on the local, switching to the express, then transferring at another station. Pretty quickly you get to know where to stand to get on the most efficient end of the train to be closest to the stairway up to the transfer platform, etc. Makes an otherwise tedious trip somewhat interesting as a puzzle.
- cat199 9y agoAnd on some systems, one can usually tell if an inbound train is running faster/slower than normal and where to adjust position accordingly to end up at the right door/position/etc..
- rishabhsagar 9y agoOk, I understand that the Railway Authority don't want people to guess / predict platform numbers because of various reasons (problems with platform maintenance, safety and last min changes etc. But what I don't understand is that why can't they change the seat allocation process such that people can reserve remaining available seats on the trains for free if they have arrived in station lobby? This can result is reduced rush to grab seats and will help people to make a more informed decision around whether to wait for the next train.
- state_machine 9y agoit takes longer to load a train with assigned seating -- that's why say, Amtrak might do it, but commuter rail trains, that at peak run minutes apart and want to minimize dwell-times at highly contended platforms, optimize for loading speed.
- daburninatorrr 9y agoInterestingly enough, I have had this same exact idea for an app, also being one of those commuters who rely on LIRR trains, also being someone who noticed that my peak train home arrives almost always on the same track, and also being one of the (few) commuters who go down to the platform before the train is called. It is a shame that this had to happen to OP. I personally don't see the app as an issue, because the amount of commuters that would actually use such a thing is rather low (seriously, stand in the concourse and look around at people, a large amount of them are not using their phones). My original idea was to scrape the data from their webpage, or see if there was a way to get the data from the Train Time app's Arrival Countdown page, but according to the post, it has been removed from their website, so there goes that idea. The lesson to me is clear though: don't try to make an app that would make commuter's lives easier. The MTA does not seem to want that, especially during the track repairs. I find it interesting that it was shut down so close to the start of the Penn Station track repairs...
- calvinbhai 9y agoCrowdsourcing the track data based on where commuters' phones travel, something like waze for LIRR should help. Instead of asking users to report the track info, it can be automated to a great extent with the use of BTLE beacons on each platform (but that again needs permission from MTA / Penn station authorities unless a long lasting BT 4.0 beacon can be sneaked in somewhere). Sad that MTA is not cooperating. Can understand why the app had to be shut down!
- jfries 9y agoFor Penn Station in particular, one can beat the crowd by heading downstairs and looking at the small CRT under the stairs. You will see the platform assignment at the same time as the crowd, but you are closer to the platforms and don't have to use the same stairs as everyone else.
- x13 9y agoIt's a great explanation of the need for this app, and how zavulon fixed a problem. But organizations change how they work with customers/the public all the time, and I couldn't help but say Waaah. Things change; tell us about the alternatives you tried to move around this obstacle.
- harshaw 9y agoThere is a similar issue with the MBTA commuter rail in Boston. At south station, the signage in the main waiting area and at the platform only announce the train 10 minutes before departure. However, there are a couple of tools to get around this. For Starters, each revenue service train has a number (like 508). In a push-pull system like the MBTA the first coach in the consist is nearest to to the platform. This coach has a number like 1827. Luckily, the MBTA publishes which trainsets will be assigned to which departures. This makes it easy to know that 1827 is for train 508. All you need to do is walk out to the platform and see if 1827 is sitting there.. if so it's your train. You can get this mapping of trainset to train via a bunch of apps. This has been further expanded upon by micro-social apps like "MBTA Rail Tracker" which has a comment section for every train. The whole thread is basically "which platform" followed by a bunch of responses and then snarkiness on why the trains are horribly late all the time.
- dkrich 9y agoI wonder if there's any correlation or way to know based on where a train is arriving from to know where it's headed. If that were the case, you could potentially either look for that data, or if it isn't available, have a Waze-like app where commuters can report which train they are on and which track they are pulling into. As an aside, I wonder whether the OP considered charging a hefty price (>$20 or so) for the app. This would lessen the number of users and page requests. Having travelled from Penn many times and having had to deal with the massive stampede that ensues when they suddenly reveal the track number, I would pay dearly to have this info available to me. Even if I used it once every year, not having to fight crowds or look forever to find an available seat, that would be money well spent. Additionally, many of the peak travelers are business travelers who probably have more money to spend on an app. I realize this isn't the most democratic solution, but it could be a way to lessen the crowds nonetheless.
- troisx 9y agoWhy don't you sue them? It's a government agency, and I believe that the data should be available under New York's public records laws (I only have limited experience with NY's laws and IANAL). From personal experience, being nice to a government agency never got me anywhere, but our lawyer has a 95% win rate against them.
- phonon 9y agoThere is no law that NY government information has to be available through a realtime API.
- londons_explore 9y agoWhy would you want to know the train more than 10 minutes before departure? Surely you'd just get on the immediate next train? Are tickets only for a specific time of train? Or are trains less frequent than every 10 minutes?
- dsr_ 9y agoThe next train for your destination may be 30 minutes later, or over an hour away. LIRR has about 125 stations in 8 zones. It's not called Long Island because it's small and round.
- wfisher 9y agowww.njtwizard.com We do this for NJ Transit with about 90% accuracy. We spent at least two months figuring out how it all works, though. It's non-trivial to do right.
- bayyar 9y agoHow do you do it?