7 ms·

NYC's Taxi and Limousine Commission Trip Record Data for 2014-2015

68 points by bko 11y ago

bko 11y ago
I'm not able to review the data yet but I wonder if TLC took in any of the feedback to better anonymize the data. [0] Also, I would love to see an analysis of whether traffic is actually getting worse compared to that of last year. This claim was made by mayor de Blasio as a reason to cap Uber rides. [0] http://research.neustar.biz/2014/09/15/riding-with-the-stars-passenger-privacy-in-the-nyc-taxicab-dataset/ http://research.neustar.biz/2014/09/15/riding-with-the-stars...
- minimaxir 11y ago
  Also: https://medium.com/@vijayp/of-taxis-and-rainbows-f6bc289679a1 https://medium.com/@vijayp/of-taxis-and-rainbows-f6bc289679a... Downloading one of the CSVs to check it out. Each one is about 2GB. EDIT: Per the BigQuery table schema, medallion is no longer a field.
- drd1008 11y ago
  http://iquantny.tumblr.com/post/125760518214/funny-math-fact-check-de-blasio-vs-uber http://iquantny.tumblr.com/post/125760518214/funny-math-fact...
danso 11y ago
FWIW: - 2013 data as FOILed by Chris Whong http://chriswhong.com/open-data/foil_nyc_taxi/ http://chriswhong.com/open-data/foil_nyc_taxi/ - 2008 to 2013 data as FOILed by me, on BigQuery https://bigquery.cloud.google.com/table/alien-climber-851:nyc_taxi_redacted.trip_data?pli=1 https://bigquery.cloud.google.com/table/alien-climber-851:ny... ...note that after Whong's request, the TLC redacted the medallion numbers, making it virtually impossible to analyze trips by cabbie.
- misiti3780 11y ago
  is google charging you for hosting that data in Big Query ?
  - danso 11y ago
    I think so...? There was a free period and now I'm getting charged about $1.50 a month (though on my credit card, it's billed as Google AdWords...). However, I just checked the actual invoice and I don't see a line item for the taxi data, just for the 40GB of other data that I have online. The taxi data is about 90GB.
    - minimaxir 11y ago
      Hosting data on BigQuery isn't free. (only 1TB of processing is free per month)
      - misiti3780 11y ago
        ok so just so i understand (i am new to big query) - even though I am not hosting this dataset (and thus not being charged) - I can, for free, query 1TB worth of queries for free using big query ?
        minimaxir 11y ago
        Correct. https://cloud.google.com/bigquery/pricing https://cloud.google.com/bigquery/pricing
        misiti3780 11y ago
        grazie!
        vgt 11y ago
        I work on BigQuery. Whoever owns Storage, NYC TLC in this case, pays a minimal fee of 2 cents per GB per month for storage. This includes multiple factors of replication/durability. Whoever is doing the querying - this can be you - pays 5 dollars per TB queried. First 1TB per month is free.
- aw3c2 11y ago
  > 2008 to 2013 data as FOILed by me Is that data set available for people who do not use Google Accounts as well? Maybe you could upload it to https://archive.org https://archive.org.
  - deleted 11y ago
    [deleted]
paulsutter 11y ago
Local governments should be demanding detailed data from companies like Uber in exchange for legalization, even raw data on locations of available cars. They have the leverage to get it now but they're wasting the chance. This data could eventually be used to avoid a true monopoly. http://www.washingtonpost.com/news/wonkblog/wp/2014/10/30/ubers-data-could-be-a-treasure-trove-for-cites-but-theyre-wasting-the-chance-to-get-it/ http://www.washingtonpost.com/news/wonkblog/wp/2014/10/30/ub...
- Shivetya 11y ago
  we already have enough hoops for companies to jump through to prevent competition so we certainly do not need any more. New York is a perfect example were regulation and so contorted the market you can make money selling your permission to run the business to the point it might be more profitable than running the business. From medallions to food cart and restaurant permits, regulation is keeping competition out while rewarding those who merely sit on permits and rent their use. It is nearly an identical situation to how badly patents are managed and rewarded
  - jdeibele 11y ago
    One of the reasons that taxis are licensed is that they're supposed to pick up everyone and anyone. Uber and Lyft don't seem to have that problem with race (AFAIK) but there are two other situations where people have trouble: (1) small children, where you need a car seat (or two) (2) people with disabilities - service animal, wheelchair, etc. Analyzing the data would hopefully show whether people had to wait for 2 hours. Also, as a substitute for race, it might be possible to see if certain areas are under-served or not served at all. Perhaps drivers are avoiding picking up in Harlem.
    - mlrtime 11y ago
      "One of the reasons that taxis are licensed is that they're supposed to pick up everyone and anyone." Which is clearly violated thousands of times per day. Sure you can call 311 in NYC but very few people do.
minimaxir 11y ago
Press Release: http://www.nyc.gov/html/tlc/downloads/pdf/press_release_08_03_15.pdf http://www.nyc.gov/html/tlc/downloads/pdf/press_release_08_0... BigQuery tables for the data: https://www.reddit.com/r/bigquery/comments/3fo9ao/nyc_taxi_trips_now_officially_shared_by_the_nyc/ https://www.reddit.com/r/bigquery/comments/3fo9ao/nyc_taxi_t...
eck 11y ago
I downloaded the first year dataset and didn't get very far in my analysis... but this was the first result, which was kind of pretty: http://i.imgur.com/ov6K6mt.jpg http://i.imgur.com/ov6K6mt.jpg
- dankohn1 11y ago
  Beautiful! Love that LaGuardia and JFK glow.
- amdlla 11y ago
  Where would one start learning methods of data vis like this?
  - aw3c2 11y ago
    That is just each point plotted on a map. So, check out software that can make maps? QGIS is nice.
    - minimaxir 11y ago
      You don't even need dedicated mapping programs. It's just latitudes and longitudes, so any 2D plotting program would be sufficient (as long as the data can fit into memory!). I made this map awhile ago for Instagram photos using R/ggplot2: http://i.imgur.com/IvGox1f.png http://i.imgur.com/IvGox1f.png Although the country boundaries aren't strictly necessary. Since there appears to be a demand for generating these maps, I'll work on a tutorial for this NYC data set.
      - misiti3780 11y ago
        you didnt have to install anything but ggplot2 to build that. wanna share the source :)
      - aw3c2 11y ago
        You murdered Greenland .D
    - 11y ago
JoshTriplett 11y ago
This is a disturbing dataset, in that it seems straightforward to extract personal data from it. Consider the information revealed by a taxi ride between a personal address and a workplace, a sensitive location, or another personal address... This kind of data was previously an issue with Uber, too.

68 points by bko 11y ago

bko 11y ago

I'm not able to review the data yet but I wonder if TLC took in any of the feedback to better anonymize the data. [0] Also, I would love to see an analysis of whether traffic is actually getting worse compared to that of last year. This claim was made by mayor de Blasio as a reason to cap Uber rides. [0] http://research.neustar.biz/2014/09/15/riding-with-the-stars-passenger-privacy-in-the-nyc-taxicab-dataset/ http://research.neustar.biz/2014/09/15/riding-with-the-stars...

minimaxir 11y ago
Also: https://medium.com/@vijayp/of-taxis-and-rainbows-f6bc289679a1 https://medium.com/@vijayp/of-taxis-and-rainbows-f6bc289679a... Downloading one of the CSVs to check it out. Each one is about 2GB. EDIT: Per the BigQuery table schema, medallion is no longer a field.
drd1008 11y ago
http://iquantny.tumblr.com/post/125760518214/funny-math-fact-check-de-blasio-vs-uber http://iquantny.tumblr.com/post/125760518214/funny-math-fact...

danso 11y ago

FWIW: - 2013 data as FOILed by Chris Whong http://chriswhong.com/open-data/foil_nyc_taxi/ http://chriswhong.com/open-data/foil_nyc_taxi/ - 2008 to 2013 data as FOILed by me, on BigQuery https://bigquery.cloud.google.com/table/alien-climber-851:nyc_taxi_redacted.trip_data?pli=1 https://bigquery.cloud.google.com/table/alien-climber-851:ny... ...note that after Whong's request, the TLC redacted the medallion numbers, making it virtually impossible to analyze trips by cabbie.

misiti3780 11y ago
is google charging you for hosting that data in Big Query ?
- danso 11y ago
  I think so...? There was a free period and now I'm getting charged about $1.50 a month (though on my credit card, it's billed as Google AdWords...). However, I just checked the actual invoice and I don't see a line item for the taxi data, just for the 40GB of other data that I have online. The taxi data is about 90GB.
  - minimaxir 11y ago
    Hosting data on BigQuery isn't free. (only 1TB of processing is free per month)
    - misiti3780 11y ago
      ok so just so i understand (i am new to big query) - even though I am not hosting this dataset (and thus not being charged) - I can, for free, query 1TB worth of queries for free using big query ?
      - minimaxir 11y ago
        Correct. https://cloud.google.com/bigquery/pricing https://cloud.google.com/bigquery/pricing
        misiti3780 11y ago
        grazie!
      - vgt 11y ago
        I work on BigQuery. Whoever owns Storage, NYC TLC in this case, pays a minimal fee of 2 cents per GB per month for storage. This includes multiple factors of replication/durability. Whoever is doing the querying - this can be you - pays 5 dollars per TB queried. First 1TB per month is free.
aw3c2 11y ago
> 2008 to 2013 data as FOILed by me Is that data set available for people who do not use Google Accounts as well? Maybe you could upload it to https://archive.org https://archive.org.
- deleted 11y ago
  [deleted]

paulsutter 11y ago

Local governments should be demanding detailed data from companies like Uber in exchange for legalization, even raw data on locations of available cars. They have the leverage to get it now but they're wasting the chance. This data could eventually be used to avoid a true monopoly. http://www.washingtonpost.com/news/wonkblog/wp/2014/10/30/ubers-data-could-be-a-treasure-trove-for-cites-but-theyre-wasting-the-chance-to-get-it/ http://www.washingtonpost.com/news/wonkblog/wp/2014/10/30/ub...

Shivetya 11y ago
we already have enough hoops for companies to jump through to prevent competition so we certainly do not need any more. New York is a perfect example were regulation and so contorted the market you can make money selling your permission to run the business to the point it might be more profitable than running the business. From medallions to food cart and restaurant permits, regulation is keeping competition out while rewarding those who merely sit on permits and rent their use. It is nearly an identical situation to how badly patents are managed and rewarded
- jdeibele 11y ago
  One of the reasons that taxis are licensed is that they're supposed to pick up everyone and anyone. Uber and Lyft don't seem to have that problem with race (AFAIK) but there are two other situations where people have trouble: (1) small children, where you need a car seat (or two) (2) people with disabilities - service animal, wheelchair, etc. Analyzing the data would hopefully show whether people had to wait for 2 hours. Also, as a substitute for race, it might be possible to see if certain areas are under-served or not served at all. Perhaps drivers are avoiding picking up in Harlem.
  - mlrtime 11y ago
    "One of the reasons that taxis are licensed is that they're supposed to pick up everyone and anyone." Which is clearly violated thousands of times per day. Sure you can call 311 in NYC but very few people do.

minimaxir 11y ago

Press Release: http://www.nyc.gov/html/tlc/downloads/pdf/press_release_08_03_15.pdf http://www.nyc.gov/html/tlc/downloads/pdf/press_release_08_0... BigQuery tables for the data: https://www.reddit.com/r/bigquery/comments/3fo9ao/nyc_taxi_trips_now_officially_shared_by_the_nyc/ https://www.reddit.com/r/bigquery/comments/3fo9ao/nyc_taxi_t...

eck 11y ago

I downloaded the first year dataset and didn't get very far in my analysis... but this was the first result, which was kind of pretty: http://i.imgur.com/ov6K6mt.jpg http://i.imgur.com/ov6K6mt.jpg

dankohn1 11y ago
Beautiful! Love that LaGuardia and JFK glow.
amdlla 11y ago
Where would one start learning methods of data vis like this?
- aw3c2 11y ago
  That is just each point plotted on a map. So, check out software that can make maps? QGIS is nice.
  - minimaxir 11y ago
    You don't even need dedicated mapping programs. It's just latitudes and longitudes, so any 2D plotting program would be sufficient (as long as the data can fit into memory!). I made this map awhile ago for Instagram photos using R/ggplot2: http://i.imgur.com/IvGox1f.png http://i.imgur.com/IvGox1f.png Although the country boundaries aren't strictly necessary. Since there appears to be a demand for generating these maps, I'll work on a tutorial for this NYC data set.
    - misiti3780 11y ago
      you didnt have to install anything but ggplot2 to build that. wanna share the source :)
    - aw3c2 11y ago
      You murdered Greenland .D
  - 11y ago

JoshTriplett 11y ago

This is a disturbing dataset, in that it seems straightforward to extract personal data from it. Consider the information revealed by a taxi ride between a personal address and a workplace, a sensitive location, or another personal address... This kind of data was previously an issue with Uber, too.