9 ms·

MTA Open Data Challenge

221 points by oftenwrong 2y ago

asjfkdlf 2y ago
The prize is very underwhelming. If they really want people to spend effort on it, they need to make the prize worth it.
- xtiansimon 2y ago
  > “The winner will receive a vintage New York City Transit item from our memorabilia collection.” Depends what it is. Long as it’s not something you could steal yourself. Ha!
  - jesterman 2y ago
    One of the options is literally a trash can! https://new.mta.info/document/85441 https://new.mta.info/document/85441 Or perhaps... a subway seat? https://new.mta.info/document/85661 https://new.mta.info/document/85661
    - erikaww 2y ago
      I’d give multiple weeks of time for a city trash can lol
  - mannyv 2y ago
    Their collection of vintage gum scrapings perhaps?
- zeroxfe 2y ago
  If you're doing it for the prize, then you're not the targeted audience :-)
- noitpmeder 2y ago
  Seems perfect actually! Attracts people that are interested in the subject matter, not just a proposed reward.
  - maxverse 2y ago
    "we're hiring people that really love programming and aren't just in it for the money"
    - 0cf8612b2e1e 2y ago
      It will look great in your portfolio.
- deleted 2y ago
  [deleted]
- nxobject 2y ago
  Never underestimate the value of surplus NYC subway memorabilia to a transit enthusiast. Especially signage from retired rolling stock.
- afavour 2y ago
  IMO it deliberately establishes a tone. This challenge is for rail fans, it’s not a generalised “use our API” hackathon type thing. Plus the MTA has a huge budget crunch. I really don’t think they could justify spending money on something with such an unclear outcome.
  - stevage 2y ago
    Even still it probably cost tens of thousands of dollars of staff time.
- IncreasePosts 2y ago
  The prize is being able to say you won the prize on your resume. I assume a lot of college kids in data science are going to be going at this.
- corytheboyd 2y ago
  I think it actually sounds kinda cool, if it’s something unique that couldn’t just be purchased!
mcfedr 2y ago
Why would you region block a webpage like this
- safeimp 2y ago
  Reading their terms, I'm guessing it's due to: > 3. Eligibility: The Challenge is open to legal residents of the United States. Entrants must be 18 years of age or older as of their date of entry. The Challenge is subject to federal, state, and local laws and regulations and is void where prohibited by law. Employees and contractors of the MTA, its subsidiaries, affiliates, and directors (collectively the “Employees”), as well as members of an Employee’s immediate family and/or those living in the same household, are ineligible to participate in the Challenge.
  - ratedgene 2y ago
    yeah but wouldn't you want to create enough buzz globally so word of mouth can spread to more US entrants?
    - safeimp 2y ago
      I don't disagree with you at all, I'm just speculating over why they'd block it.
  - n_plus_1_acc 2y ago
    You can be a resident of the US and be on vacation for a couple weeks
- nemo44x 2y ago
  Because the next thing you know the EU is suing you for billions of Euros.
  - cddotdotslash 2y ago
    Expect to see more of this, especially when the audience is local/US. IIRC, some newspapers are already doing region blocks. Why should website owners targeting US visitors spend _any_ amount of money making their content comply with asinine regulations (like cookie banners)?
sgtbr1 2y ago
can someone share the data?
- manvillej 2y ago
  what a tragedy, this person never learned how to read.
leanthonyrn 2y ago
Intersting challenge. Here is the NotebookLM Audio: MTA's Open Data program https://notebooklm.google.com/notebook/286a30b9-b17f-4dac-9ed4-ac888e5ed1d1?pli=1 https://notebooklm.google.com/notebook/286a30b9-b17f-4dac-9e...
rayrrr 2y ago
Hold my Metrocard.
- onemoresoop 2y ago
  Hold my bus transfer card.
whitej125 2y ago
Would be neat if instead an open-ended challenge ("here's some data, do something cool") the MTA instead shared a list of hypothetical or real problems to solve and provided data that could be potentially useful in the exploration/solution to the problem.
- maxverse 2y ago
  Also, considering they just got a 68 billion dollar budget approved [1] over the next 5 years, even a small monetary reward would be nice for this. It doesn't need to be a ton of money, but something other than "here's a piece of empty and memorabilia and we'll write a blog post" would be a good incentive [1] https://ny1.com/nyc/all-boroughs/news/2024/09/25/mta-board-approves--68-billion-capital-plan https://ny1.com/nyc/all-boroughs/news/2024/09/25/mta-board-a...
  - exegete 2y ago
    I think you are misinterpreting that article. The MTA board approved the plan to spend $68B but they depend on the state to give them funds. That’s the amount of money they are asking for based on the projects they want to complete. The state government has to pass a budget to fund that plan (or do something else). Additionally several current, already started projects are on hold due to the “pause” of congestion pricing which was going to be a funding source.
- doctorpangloss 2y ago
  Why would a cost center political institution enumerate all its problems? It is kind of miraculous they can engage with the public this way at all.
chaps 2y ago
I do work with "open data" on a near-obsessive basis and -- friends, please do not trust "open data" portals to reflect reality accurately. The datasets are often curated, categories changed during the ETL processes, rows missing, and things like that. For example, Chicago's "crimes" dataset intentionally doesn't include all homicides. Can't remember the exact dataset, but I once had a conversation with Chicago's head of open data who told me that they intentionally removed many rows because they were concerned that the public was going to misinterpret the results... but didn't make it clear that rows were missing. So I guess everybody gets the opportunity to misinterpret the results! FOIA is the better alternative because it gives you the original, pre-cleaned data. Open data is a lie.
- kalendos 2y ago
  I can only imagine. Many ETLs are already messy in companies with better tooling and processes. Would love to read more about your experience with Open Data. Any place where I can reach out?
  - chaps 2y ago
    Here's something about shotspotter data in Chicago: https://x.com/foiachap/status/1775296597850480663 https://x.com/foiachap/status/1775296597850480663 And this one makes some rounds: https://mchap.io/that-time-the-city-of-seattle-accidentally-gave-me-32m-emails-for-40-dollars4997.html https://mchap.io/that-time-the-city-of-seattle-accidentally-... Feel free to reach out!
- pjot 2y ago
  This is super true. For my city’s portal as well. I’ve found one way around this by versioning the dataset - that is, committing the diffs in git. Credit to Simon Willison’s git-scraping technique. I do this with my power company’s outage map: https://github.com/patricktrainer/entergy-outages https://github.com/patricktrainer/entergy-outages 67k commits! https://simonwillison.net/2020/Oct/9/git-scraping/ https://simonwillison.net/2020/Oct/9/git-scraping/
  - 2y ago
thecosas 2y ago
Time for someone to crack their knuckles and do a Power Broker-style MTA Open Data mashup :-) https://en.wikipedia.org/wiki/The_Power_Broker https://en.wikipedia.org/wiki/The_Power_Broker
krebby 2y ago
Some really nice example visualizations from Matt Yarri and Julia Lynn at the MTA: https://www.linkedin.com/posts/matt-yarri_some-of-the-data-we-get-to-work-with-at-the-activity-7222644015995777025-f7er https://www.linkedin.com/posts/matt-yarri_some-of-the-data-w... https://new.mta.info/article/introducing-subway-origin-destination-ridership-dataset https://new.mta.info/article/introducing-subway-origin-desti...
nocman 2y ago
I keep clicking on these 'MTA' articles expecting them to be about a "message transfer agent". Then I think, oh, right, wrong MTA. Guess I've spent too much time dealing with email servers.
slt2021 2y ago
I could not find dataset with payroll hours reported and overtime reimbursed for each MTA employee. I wanted to investigate how well MTA is managing its workforce and compensation (as to require additional tax in form of Congestion Pricing to fix its budget hole), but there seems to be no dataset for that. Does anyone have links to MTA payroll/hours/overtime related dataset? or alternatively, I need dataset to study each and every subway improvement project, and components of each project in materials, labor and etc
- WUMBOWUMBO 2y ago
  perhaps this could be covered in a FOIA request
- deleted 2y ago
  [deleted]
aaron695 2y ago
[dead]
stevage 2y ago
Interesting, these open data challenges were all the rage 10 years ago. Wonder why the sudden trip down memory lane.
deleted 2y ago
[deleted]
shrikar 2y ago
Tried building something with Cursor + Chatgpt in 30mins not bad for the initial exploration https://www.youtube.com/watch?v=w3mkXPdTVlI https://www.youtube.com/watch?v=w3mkXPdTVlI and the demo link: https://mtachallenge.streamlit.app/ https://mtachallenge.streamlit.app/

221 points by oftenwrong 2y ago

The prize is very underwhelming. If they really want people to spend effort on it, they need to make the prize worth it.

xtiansimon 2y ago
> “The winner will receive a vintage New York City Transit item from our memorabilia collection.” Depends what it is. Long as it’s not something you could steal yourself. Ha!
- jesterman 2y ago
  One of the options is literally a trash can! https://new.mta.info/document/85441 https://new.mta.info/document/85441 Or perhaps... a subway seat? https://new.mta.info/document/85661 https://new.mta.info/document/85661
  - erikaww 2y ago
    I’d give multiple weeks of time for a city trash can lol
- mannyv 2y ago
  Their collection of vintage gum scrapings perhaps?
zeroxfe 2y ago
If you're doing it for the prize, then you're not the targeted audience :-)
noitpmeder 2y ago
Seems perfect actually! Attracts people that are interested in the subject matter, not just a proposed reward.
- maxverse 2y ago
  "we're hiring people that really love programming and aren't just in it for the money"
  - 0cf8612b2e1e 2y ago
    It will look great in your portfolio.
deleted 2y ago
[deleted]
nxobject 2y ago
Never underestimate the value of surplus NYC subway memorabilia to a transit enthusiast. Especially signage from retired rolling stock.
afavour 2y ago
IMO it deliberately establishes a tone. This challenge is for rail fans, it’s not a generalised “use our API” hackathon type thing. Plus the MTA has a huge budget crunch. I really don’t think they could justify spending money on something with such an unclear outcome.
- stevage 2y ago
  Even still it probably cost tens of thousands of dollars of staff time.
IncreasePosts 2y ago
The prize is being able to say you won the prize on your resume. I assume a lot of college kids in data science are going to be going at this.
corytheboyd 2y ago
I think it actually sounds kinda cool, if it’s something unique that couldn’t just be purchased!

mcfedr 2y ago

Why would you region block a webpage like this

safeimp 2y ago
Reading their terms, I'm guessing it's due to: > 3. Eligibility: The Challenge is open to legal residents of the United States. Entrants must be 18 years of age or older as of their date of entry. The Challenge is subject to federal, state, and local laws and regulations and is void where prohibited by law. Employees and contractors of the MTA, its subsidiaries, affiliates, and directors (collectively the “Employees”), as well as members of an Employee’s immediate family and/or those living in the same household, are ineligible to participate in the Challenge.
- ratedgene 2y ago
  yeah but wouldn't you want to create enough buzz globally so word of mouth can spread to more US entrants?
  - safeimp 2y ago
    I don't disagree with you at all, I'm just speculating over why they'd block it.
- n_plus_1_acc 2y ago
  You can be a resident of the US and be on vacation for a couple weeks
nemo44x 2y ago
Because the next thing you know the EU is suing you for billions of Euros.
- cddotdotslash 2y ago
  Expect to see more of this, especially when the audience is local/US. IIRC, some newspapers are already doing region blocks. Why should website owners targeting US visitors spend _any_ amount of money making their content comply with asinine regulations (like cookie banners)?

sgtbr1 2y ago

can someone share the data?

manvillej 2y ago
what a tragedy, this person never learned how to read.

leanthonyrn 2y ago

Intersting challenge. Here is the NotebookLM Audio: MTA's Open Data program https://notebooklm.google.com/notebook/286a30b9-b17f-4dac-9ed4-ac888e5ed1d1?pli=1 https://notebooklm.google.com/notebook/286a30b9-b17f-4dac-9e...

rayrrr 2y ago

Hold my Metrocard.

onemoresoop 2y ago
Hold my bus transfer card.

whitej125 2y ago

Would be neat if instead an open-ended challenge ("here's some data, do something cool") the MTA instead shared a list of hypothetical or real problems to solve and provided data that could be potentially useful in the exploration/solution to the problem.

maxverse 2y ago
Also, considering they just got a 68 billion dollar budget approved [1] over the next 5 years, even a small monetary reward would be nice for this. It doesn't need to be a ton of money, but something other than "here's a piece of empty and memorabilia and we'll write a blog post" would be a good incentive [1] https://ny1.com/nyc/all-boroughs/news/2024/09/25/mta-board-approves--68-billion-capital-plan https://ny1.com/nyc/all-boroughs/news/2024/09/25/mta-board-a...
- exegete 2y ago
  I think you are misinterpreting that article. The MTA board approved the plan to spend $68B but they depend on the state to give them funds. That’s the amount of money they are asking for based on the projects they want to complete. The state government has to pass a budget to fund that plan (or do something else). Additionally several current, already started projects are on hold due to the “pause” of congestion pricing which was going to be a funding source.
doctorpangloss 2y ago
Why would a cost center political institution enumerate all its problems? It is kind of miraculous they can engage with the public this way at all.

chaps 2y ago

I do work with "open data" on a near-obsessive basis and -- friends, please do not trust "open data" portals to reflect reality accurately. The datasets are often curated, categories changed during the ETL processes, rows missing, and things like that. For example, Chicago's "crimes" dataset intentionally doesn't include all homicides. Can't remember the exact dataset, but I once had a conversation with Chicago's head of open data who told me that they intentionally removed many rows because they were concerned that the public was going to misinterpret the results... but didn't make it clear that rows were missing. So I guess everybody gets the opportunity to misinterpret the results! FOIA is the better alternative because it gives you the original, pre-cleaned data. Open data is a lie.

kalendos 2y ago
I can only imagine. Many ETLs are already messy in companies with better tooling and processes. Would love to read more about your experience with Open Data. Any place where I can reach out?
- chaps 2y ago
  Here's something about shotspotter data in Chicago: https://x.com/foiachap/status/1775296597850480663 https://x.com/foiachap/status/1775296597850480663 And this one makes some rounds: https://mchap.io/that-time-the-city-of-seattle-accidentally-gave-me-32m-emails-for-40-dollars4997.html https://mchap.io/that-time-the-city-of-seattle-accidentally-... Feel free to reach out!
pjot 2y ago
This is super true. For my city’s portal as well. I’ve found one way around this by versioning the dataset - that is, committing the diffs in git. Credit to Simon Willison’s git-scraping technique. I do this with my power company’s outage map: https://github.com/patricktrainer/entergy-outages https://github.com/patricktrainer/entergy-outages 67k commits! https://simonwillison.net/2020/Oct/9/git-scraping/ https://simonwillison.net/2020/Oct/9/git-scraping/
- 2y ago

thecosas 2y ago

Time for someone to crack their knuckles and do a Power Broker-style MTA Open Data mashup :-) https://en.wikipedia.org/wiki/The_Power_Broker https://en.wikipedia.org/wiki/The_Power_Broker

krebby 2y ago

Some really nice example visualizations from Matt Yarri and Julia Lynn at the MTA: https://www.linkedin.com/posts/matt-yarri_some-of-the-data-we-get-to-work-with-at-the-activity-7222644015995777025-f7er https://www.linkedin.com/posts/matt-yarri_some-of-the-data-w... https://new.mta.info/article/introducing-subway-origin-destination-ridership-dataset https://new.mta.info/article/introducing-subway-origin-desti...

nocman 2y ago

I keep clicking on these 'MTA' articles expecting them to be about a "message transfer agent". Then I think, oh, right, wrong MTA. Guess I've spent too much time dealing with email servers.

slt2021 2y ago

I could not find dataset with payroll hours reported and overtime reimbursed for each MTA employee. I wanted to investigate how well MTA is managing its workforce and compensation (as to require additional tax in form of Congestion Pricing to fix its budget hole), but there seems to be no dataset for that. Does anyone have links to MTA payroll/hours/overtime related dataset? or alternatively, I need dataset to study each and every subway improvement project, and components of each project in materials, labor and etc

WUMBOWUMBO 2y ago
perhaps this could be covered in a FOIA request
deleted 2y ago
[deleted]

aaron695 2y ago

[dead]

stevage 2y ago

Interesting, these open data challenges were all the rage 10 years ago. Wonder why the sudden trip down memory lane.

deleted 2y ago

[deleted]

shrikar 2y ago

Tried building something with Cursor + Chatgpt in 30mins not bad for the initial exploration https://www.youtube.com/watch?v=w3mkXPdTVlI https://www.youtube.com/watch?v=w3mkXPdTVlI and the demo link: https://mtachallenge.streamlit.app/ https://mtachallenge.streamlit.app/