11 ms·

Google launches Public Datasets program

423 points by vgt 11y ago

ortusdux 11y ago
"HACKER NEWS - A dataset that contains all stories and comments from Hacker News since its launch in 2006." I know what I'm doing this weekend.
- panarky 11y ago
  I'm dying to get comment scores, not just submission scores. There's a lot you could do with that to find the best comments, which is really why HN is so awesome.
  - Alex3917 11y ago
    > There's a lot you could do with that to find the best comments. In any given situation, how often does best comment you could write coincide with the one that would get you the most upvotes? I'm guessing not frequently.
    - Estragon 11y ago
      Your best is different from his/her best.
  - minimaxir 11y ago
    Blame HN for that. It was removed from public access right after I made a blog post about it, although it was coincidental I think.
    - jrockway 11y ago
      Though the data could be assembled; I can see my comment scores, you can see your comment scores...
- pella 11y ago
  similar: https://hn.algolia.com/ https://hn.algolia.com/ ( Search Hacker News )
- jrockway 11y ago
  It's mildly amusing: > SELECT sum(length(text)) FROM [bigquery-public-data:hacker_news.comments] where author="jrockway" 4830955 That's almost 5 MB of comments I've written.
  - toomuchtodo 11y ago
    Get outside more :)
    - bcook 11y ago
      Being outside does not preclude internet access... :)
- jonesb6 11y ago
  Would be interesting to find out the value (upvotes) of sponsored content vs. unsponsered content, as well as comments from green users vs normal users.
  - minimaxir 11y ago
    There isn't sponsored content on HN. (if you are referring to the job ads, those do not receive upvotes and follow a steady rank decay.)
    - teddyh 11y ago
      What? The lack of upvotes (or steady decay) does not make it “not content”.
  - deleted 11y ago
    [deleted]
- fhoffa 11y ago
  Some resources to get you started: - How to use the Hacker News dataset https://medium.com/google-cloud/big-data-stories-in-seconds-hacker-news-abe52bc5caad#.pjfjzr76j https://medium.com/google-cloud/big-data-stories-in-seconds-... - Discussion of the HN dataset announcement here https://news.ycombinator.com/item?id=10440502 https://news.ycombinator.com/item?id=10440502 - An iPython notebook: https://github.com/fhoffa/notebooks/blob/master/analyzing%20hacker%20news.ipynb https://github.com/fhoffa/notebooks/blob/master/analyzing%20... - More: http://debarghyadas.com/writes/looking-back-at-9-years-of-hacker-news/ http://debarghyadas.com/writes/looking-back-at-9-years-of-ha... Disclaimer: I'm Felipe Hoffa, and I work at Google. (https://twitter.com/felipehoffa https://twitter.com/felipehoffa)
  - bambax 11y ago
    How fresh is the data, and how often is it refreshed? It doesn't seem to be described anywhere...
    - dwmintz 11y ago
      Looks like the data goes up through 2015-10-13. I created a Look (disclosure: work for Looker) that shows story counts by day for the last 365 days here: https://looker.com/publicdata/looks/169?show=viz https://looker.com/publicdata/looks/169?show=viz
baabaa 11y ago
Site doesn't render well on my Moto X.
benhamner 11y ago
In a similar vein, Kaggle datasets enables you to run Python, R, Julia, and SQL on many public datasets https://www.kaggle.com/datasets https://www.kaggle.com/datasets
dwmintz 11y ago
Great to see folks digging into this. Looker (where I work) is what you're seeing visualize and make the underlying datasets explorable. Our founder did a great blog post explaining how it works and how quickly these datasets yield interesting insights with just a few lines of code. Full blog post explaining and walking through the process (and giving access to explore the data fully yourself, no SQL required) is here: http://looker.com/blog/hacking-hacker-news http://looker.com/blog/hacking-hacker-news Felipe and all the other folks at Google have done a great job getting this project off the ground and we're psyched to partner with them. We're working on some new public datasets now, but if you have particular ones you'd like to explore, let us know. What should we look at next? Census? Medicare?
- vgt 11y ago
  Kudos to Looker, great partners here! We're just getting started :)
voltagex_ 11y ago
Any chance of getting GNAF and some other Australian datasets up there? http://data.gov.au/dataset/geocoded-national-address-file-g-naf http://data.gov.au/dataset/geocoded-national-address-file-g-... A CKAN->BigQuery connector would be interesting (think of an "Open in BigQuery" button)
- fhoffa 11y ago
  Twitter noticed: https://twitter.com/_pwalsh/status/715054406073524224 https://twitter.com/_pwalsh/status/715054406073524224
polartx 11y ago
I work and play in data. By far the best resource I've encountered is https://app.enigma.io/ https://app.enigma.io/ Signups are free. The aggregated public data is plentiful and easily discovered, indexed, filtered, and exported. Free account have API limitations, but as far as govt data is concerned, I don't find that its updated often enough to peg my API rate limiter anyway.
- frikk 11y ago
  Awesome. Do you have any other resources to share? I run occasional local hackathons and am always looking for inspiring data sets.
- vijayr 11y ago
  can you talk more about your work? what kind of projects do you work on? what tools do you use other than enigma?
vgt 11y ago
One large difference between this program and alternative programs is that data already resides in Google BigQuery: - You do not need to spin up a database to work with BigQuery - You can simply start writing SQL on top of BigQuery - You may leverage Dataflow and MapReduce connectors to work with this data directly in Hadoop, Spark, or Dataflow - BigQuery has a free tier - one Terabyte of data processed per month Finally, for folks who would like to share their datasets, BigQuery offers free hosting and credits to help get a pipeline going. (disclosure: work on BigQuery)
- dwmintz 11y ago
  And if you've got public data in BigQuery that you want to make explorable/shareable/visualizable to anybody (no SQL required), let Looker know and we'll see what we can do. (disclosure: work at Looker)
  - vgt 11y ago
    Looker has been a great launch partner for this program. Really helped us develop and deliver great interactive dashboards.
- Veratyr 11y ago
  > Finally, for folks who would like to share their datasets, BigQuery offers free hosting and credits to help get a pipeline going. Is this available in an automated fashion or is it something one has to contact Google for?
  - vgt 11y ago
    You can always create a dataset and make it public by setting open ACLs to it. The only downside is paying $0.02 per GB per month. To participate in the Public Datasets program, you must contact us - there's a lot of legal mumbo jumbo associated with us hosting your datasets that I wish weren't the case :)
kuschku 11y ago
Can we extract the datasets to work with them on our own (cheaper to query) servers?
- vgt 11y ago
  Of course! BigQuery export operations are free. Although compared with other cloud technologies, BigQuery is incredibly cost-effective :)
  - kuschku 11y ago
    Well, I’ve checked the prices, and dedicated servers from old-style hosters are still far cheaper if you’re going to query data 24/7.
- Bytes 11y ago
  Most of the datasets seem to be available to download here https://www.reddit.com/r/bigquery/wiki/datasets https://www.reddit.com/r/bigquery/wiki/datasets.
  - kuschku 11y ago
    After looking at those: What’s "big" about any of them? That’s literally a standard database size, and can be easily done with any postgres install. All of that data combined can be stored and quickly queried on just 10 dedicated servers, for an overall price of 200€/month. And with "all" I mean "all 20TB". And with "quickly" I mean "faster than a network request to Google". ________________ Due to rate limiting, I can't answer you at the moment, vgt. So my answer will happen here, inline: > Well, like this: > If someone wants to decide if they want to build a business on bigquery, they want to evaluate it's performance on an average data set before for free. > Assuming the test data is thought for similar cases it has to be of similar size to the average dataset used with BigQuery.
    - vgt 11y ago
      Apologies, but where do you find the implication that these datasets are "big"? There is "big" in BigQuery, so apologize if you misunderstood. We try to keep datasets reasonable, so that folks can get the most out of the BigQuery free pricing tier :) If you want bigger, here's an example of a 10TB dataset: https://cloud.google.com/genomics/data/1000-genomes https://cloud.google.com/genomics/data/1000-genomes
danvoell 11y ago
https://cloud.google.com/bigquery/public-data/hacker-news https://cloud.google.com/bigquery/public-data/hacker-news
th0br0 11y ago
I get 404s for all Looker iframes...
- tosh 11y ago
  Same here in Safari. Chrome seems to work.
- apahwa 11y ago
  where are you seeing Looker being used?
  - dwmintz 11y ago
    Looker is powering all the iframes (any 404s should be fixed now). Bigquery is hosting the data, but Looker is generating the queries and visualizations of the data.
    - apahwa 11y ago
      Interesting, I'm surprised Google is using Looker rather than their own tool edit: ah, I see, it is just one of the partners they have with BigQuery and they are apparently using it for their website (I'm guessing because Looker lets you iframe graphs directly into the page for free).
      - dwmintz 11y ago
        Yup, Looker is a Premier Partner for the GCP launch (https://cloud.google.com/partners/?q=Looker#search https://cloud.google.com/partners/?q=Looker#search). And then we (I work at Looker) are specifically partnering with GCP for the Public Datasets project. Because LookML (Looker's modeling language) makes it easy to explore and visualize big datasets, we're building out models and dashboards so visitors can get a sense of BigQuery/Looker's power and find insights from the datasets quickly (whether or not they write SQL). Frankly, there's a ton of public data that's "available" in the sense that you can technically download and clean a CSV, but isn't actually easy to extract meaning from. So we figured we'd select some interesting datasets, do the cleaning, uploading and modeling, and then let folks have at it for free. If there are specific datasets you'd be interested in seeing, let us know and we'll see what we can do.
nl 11y ago
This is a much more useful list: https://www.reddit.com/r/bigquery/wiki/datasets https://www.reddit.com/r/bigquery/wiki/datasets I'm not sure why Google hosts it on Reddit. There's some interesting (and more up-to-date) stuff on there.
- jpatokal 11y ago
  That's a list of random data from random people, curated by Googler felipehoffa in an unofficial capacity. The ones in the submitted page are maintained by Google itself in the public-data project.
  - nl 11y ago
    I believe that the GDelt, Freebase and Genomics tables (at least) are officially supported by Google.
- kyrra 11y ago
  This page may interest you: https://support.google.com/cloud/answer/3466163?hl=en https://support.google.com/cloud/answer/3466163?hl=en I think the Google Cloud team watches the community and helps promote the forums that have the most active for a given product.
talles 11y ago
Who is the King of Hacker News? https://cloud.google.com/bigquery/public-data/hacker-news#who_is_the_king_of_hacker_news https://cloud.google.com/bigquery/public-data/hacker-news#wh...
- lloydt 11y ago
  Click on the Stories Count and you can see the individual stories :)
pbh101 11y ago
I remember the earlier Google attempt at the same thing. I was in undergrad and an engineer came to my Uni to give a presentation on the topic. A big area of concern was "Why won't Google just shut this down when the going gets rough?" The question was mostly batted away along the lines of "this is lunch money to Google anyway." But just a couple years later, it did indeed bite the dust. Covered in Wired: http://www.wired.com/2008/01/google-to-provi/ http://www.wired.com/2008/01/google-to-provi/ Update: And the shutdown article, 11 months later: http://www.wired.com/2008/12/googlescienceda/ http://www.wired.com/2008/12/googlescienceda/
- vgt 11y ago
  Google BigQuery is a paid service, Dremel under the hood, used internally at Google by ~80% of Googlers every week. It's simply not going away because Google relies on it heavily. Externally, companies like Spotify and Kabam Games rely heavily on BigQuery as well: https://www.youtube.com/watch?v=LTVFg6YOjWo&list=PLIivdWyY5sqLpj5W-tdqT3O2NQ-_LgiPL&index=7 https://www.youtube.com/watch?v=LTVFg6YOjWo&list=PLIivdWyY5s... https://www.youtube.com/watch?v=6Nv18xmJirs&list=PLIivdWyY5sqLpj5W-tdqT3O2NQ-_LgiPL&index=6 https://www.youtube.com/watch?v=6Nv18xmJirs&list=PLIivdWyY5s... So the story of Google Cloud products, which are paid, supported, and Enterprise-grade, and which are covered by a clear 1 year deprecation policy, are very different from the story of free services. Public Datasets is just a function of BigQuery's unique ACL functionality.
  - awqrre 11y ago
    Google employees using something internally doesn't mean that it won't go away for everyone else... Also, they could decide to honor existing contracts and stop issuing new ones...
    - vgt 11y ago
      BigQuery is a truly fully-managed service. This means that SREs are on call 24/7. It's not software that one downloads. Hence it really doesn't make a lot of sense to have it available to just a few folks, incremental overhead is low. I'd love to hear industry standards around deprecation of paid services. This is really a way of doing business in the entire world, and I'm not sure why Google is being held to an unreasonable and impossible standard of "never deprecate anything ever!". For example, here is Microsoft's: https://en.wikipedia.org/wiki/Category:Discontinued_Microsoft_software https://en.wikipedia.org/wiki/Category:Discontinued_Microsof...
bakztfuture 11y ago
Any chance we'll be seeing the Common Crawl data on there anytime soon?
- vgt 11y ago
  Sounds like a good one. We'll be sure to reach out to them. Or, if you know them, have them shoot us a note at: bq-public-data@google.com
tmannen 11y ago
I just noticed the Freebase data available as one of the public datasets, so been wondering what happened to the Freebase team? I liked what Freebase, DBpedia etc were doing few years ago with "semantic db's". I remember Freebase Gridworks being extremely useful for data cleanup. Haven't had a reason to follow the space...is it dead? Haven't seen much semantic talk off late. Where would one go to catch up on the latest news?
- fhoffa 11y ago
  Freebase data is being moved into Wikidata. I loaded Wikidata into BigQuery too! Fun with movies and cats: https://medium.com/google-cloud/oscars-2016-movies-that-got-the-most-attention-on-wikipedia-151cd56f4fc0 https://medium.com/google-cloud/oscars-2016-movies-that-got-... Implementation notes: https://lists.wikimedia.org/pipermail/wikidata/2016-March/008414.html https://lists.wikimedia.org/pipermail/wikidata/2016-March/00... More notes: https://lists.wikimedia.org/pipermail/wikidata/2016-March/008427.html https://lists.wikimedia.org/pipermail/wikidata/2016-March/00...
  - krishna2 11y ago
    Super interesting. Thanks for all the curation and relevant set of links that you are promptly posting/replying. StackOverflow data is available for free too - that would be one awesome dataset to let folks get their hands on. Any plans for that?
flashman 11y ago
I just ran a Markov text generator over the USA Name Database and created a few thousand girls' names, including: Aracella, Ashla, Blakelyne, Carylou, Damariah, Enchantrelle, Francenza, Iridia, Jalexius, Lilliotte and Scotlanta. Hey, maybe I can start a bespoke baby name business...
- alexwebb2 11y ago
  I'm building a baby name app and this is a fantastic idea that I am probably going to steal.
temuze 11y ago
I hope someone submits a voice recognition data! It's fantastically hard to find audio->phoneme datasets and only slightly less difficult to find audio->word and word->possible_phonemes...
marchenko 11y ago
Looking a the suggested textual analyses of the Reddit and HN datasets, and thinking about how they can be combined, makes me wonder if anonymity-through.multiple-avatars will even be remotely possible 5 years from now.
cosmeen 11y ago
Seems like it has a dataset of hacker news since it launched in 2006 -- SELECT * FROM [bigquery-public-data:hacker_news.stories] ORDER BY score DESC LIMIT 100 -- made a screenshot with Top 20 stories on HN http://i.imgur.com/czXYEyQ.png http://i.imgur.com/czXYEyQ.png
dcdevito 11y ago
Sorry Google but my trust in cloud solutions reside in AWS and Azure. Why? Because when Amazon and Microsoft announce something I know there's a good chance it will still exist 24 months later
- vgt 11y ago
  This narrative gets repeated time and time again, and it really doesn't hold up to even surface debate: - Google TI and Cloud merged. Same teams. Per Urs Holzle in Jan2014 - Just last week Sundar (CEO), Eric Schmidt (chairman), Jeff Dean, Urs Holzle, and Diane were on stage talking about how they spent $10 billion last year investing in infrastructure and how serious Google is about cloud (https://www.youtube.com/watch?v=HgWHeT_OwHc&list=PLIivdWyY5sqIFd0E6JG1hVr8sXQaLmmBP https://www.youtube.com/watch?v=HgWHeT_OwHc&list=PLIivdWyY5s...) - In many cases, publically available technologies are actually replacing internal technologies (BigQuery and Dremel, Dataflow and FlumeJava/Millwheel for example). - Disney, Coca Cola, Snapchat, Spotify, Home Depot are just some of the customers running on Google Cloud. It'd be very foolish to abandon these. - If you watch the day two GCP NEXT keynote, you'll see how Google is the cloud leader in environmental responsibility and open source. We're the good guys :) (https://www.youtube.com/watch?v=axhdIa_co2o&list=PLIivdWyY5sqIFd0E6JG1hVr8sXQaLmmBP&index=2 https://www.youtube.com/watch?v=axhdIa_co2o&list=PLIivdWyY5s...) Happy to debate further!
javiramos 11y ago
There's a Hacker News dataset. In case anyone wants to take a crack at it: https://cloud.google.com/bigquery/public-data/hacker-news https://cloud.google.com/bigquery/public-data/hacker-news
deleted 11y ago
[deleted]
mark_l_watson 11y ago
Public data is good! I have used their book text data on two projects to generate common ngrams. These newly released data sets look useful and using BigQuery is reasonable if you just need parts of data sets. Amazon does something similar: keeping useful data in S3 for cheap access when running on AWS.
kwoff 11y ago
How can they copy our comments from here to there and sell queries on them...?

423 points by vgt 11y ago

"HACKER NEWS - A dataset that contains all stories and comments from Hacker News since its launch in 2006." I know what I'm doing this weekend.

panarky 11y ago
I'm dying to get comment scores, not just submission scores. There's a lot you could do with that to find the best comments, which is really why HN is so awesome.
- Alex3917 11y ago
  > There's a lot you could do with that to find the best comments. In any given situation, how often does best comment you could write coincide with the one that would get you the most upvotes? I'm guessing not frequently.
  - Estragon 11y ago
    Your best is different from his/her best.
- minimaxir 11y ago
  Blame HN for that. It was removed from public access right after I made a blog post about it, although it was coincidental I think.
  - jrockway 11y ago
    Though the data could be assembled; I can see my comment scores, you can see your comment scores...
pella 11y ago
similar: https://hn.algolia.com/ https://hn.algolia.com/ ( Search Hacker News )
jrockway 11y ago
It's mildly amusing: > SELECT sum(length(text)) FROM [bigquery-public-data:hacker_news.comments] where author="jrockway" 4830955 That's almost 5 MB of comments I've written.
- toomuchtodo 11y ago
  Get outside more :)
  - bcook 11y ago
    Being outside does not preclude internet access... :)
jonesb6 11y ago
Would be interesting to find out the value (upvotes) of sponsored content vs. unsponsered content, as well as comments from green users vs normal users.
- minimaxir 11y ago
  There isn't sponsored content on HN. (if you are referring to the job ads, those do not receive upvotes and follow a steady rank decay.)
  - teddyh 11y ago
    What? The lack of upvotes (or steady decay) does not make it “not content”.
- deleted 11y ago
  [deleted]
fhoffa 11y ago
Some resources to get you started: - How to use the Hacker News dataset https://medium.com/google-cloud/big-data-stories-in-seconds-hacker-news-abe52bc5caad#.pjfjzr76j https://medium.com/google-cloud/big-data-stories-in-seconds-... - Discussion of the HN dataset announcement here https://news.ycombinator.com/item?id=10440502 https://news.ycombinator.com/item?id=10440502 - An iPython notebook: https://github.com/fhoffa/notebooks/blob/master/analyzing%20hacker%20news.ipynb https://github.com/fhoffa/notebooks/blob/master/analyzing%20... - More: http://debarghyadas.com/writes/looking-back-at-9-years-of-hacker-news/ http://debarghyadas.com/writes/looking-back-at-9-years-of-ha... Disclaimer: I'm Felipe Hoffa, and I work at Google. (https://twitter.com/felipehoffa https://twitter.com/felipehoffa)
- bambax 11y ago
  How fresh is the data, and how often is it refreshed? It doesn't seem to be described anywhere...
  - dwmintz 11y ago
    Looks like the data goes up through 2015-10-13. I created a Look (disclosure: work for Looker) that shows story counts by day for the last 365 days here: https://looker.com/publicdata/looks/169?show=viz https://looker.com/publicdata/looks/169?show=viz

baabaa 11y ago

Site doesn't render well on my Moto X.

benhamner 11y ago

In a similar vein, Kaggle datasets enables you to run Python, R, Julia, and SQL on many public datasets https://www.kaggle.com/datasets https://www.kaggle.com/datasets

dwmintz 11y ago

Great to see folks digging into this. Looker (where I work) is what you're seeing visualize and make the underlying datasets explorable. Our founder did a great blog post explaining how it works and how quickly these datasets yield interesting insights with just a few lines of code. Full blog post explaining and walking through the process (and giving access to explore the data fully yourself, no SQL required) is here: http://looker.com/blog/hacking-hacker-news http://looker.com/blog/hacking-hacker-news Felipe and all the other folks at Google have done a great job getting this project off the ground and we're psyched to partner with them. We're working on some new public datasets now, but if you have particular ones you'd like to explore, let us know. What should we look at next? Census? Medicare?

vgt 11y ago
Kudos to Looker, great partners here! We're just getting started :)

voltagex_ 11y ago

Any chance of getting GNAF and some other Australian datasets up there? http://data.gov.au/dataset/geocoded-national-address-file-g-naf http://data.gov.au/dataset/geocoded-national-address-file-g-... A CKAN->BigQuery connector would be interesting (think of an "Open in BigQuery" button)

fhoffa 11y ago
Twitter noticed: https://twitter.com/_pwalsh/status/715054406073524224 https://twitter.com/_pwalsh/status/715054406073524224

polartx 11y ago

I work and play in data. By far the best resource I've encountered is https://app.enigma.io/ https://app.enigma.io/ Signups are free. The aggregated public data is plentiful and easily discovered, indexed, filtered, and exported. Free account have API limitations, but as far as govt data is concerned, I don't find that its updated often enough to peg my API rate limiter anyway.

frikk 11y ago
Awesome. Do you have any other resources to share? I run occasional local hackathons and am always looking for inspiring data sets.
vijayr 11y ago
can you talk more about your work? what kind of projects do you work on? what tools do you use other than enigma?

vgt 11y ago

One large difference between this program and alternative programs is that data already resides in Google BigQuery: - You do not need to spin up a database to work with BigQuery - You can simply start writing SQL on top of BigQuery - You may leverage Dataflow and MapReduce connectors to work with this data directly in Hadoop, Spark, or Dataflow - BigQuery has a free tier - one Terabyte of data processed per month Finally, for folks who would like to share their datasets, BigQuery offers free hosting and credits to help get a pipeline going. (disclosure: work on BigQuery)

dwmintz 11y ago
And if you've got public data in BigQuery that you want to make explorable/shareable/visualizable to anybody (no SQL required), let Looker know and we'll see what we can do. (disclosure: work at Looker)
- vgt 11y ago
  Looker has been a great launch partner for this program. Really helped us develop and deliver great interactive dashboards.
Veratyr 11y ago
> Finally, for folks who would like to share their datasets, BigQuery offers free hosting and credits to help get a pipeline going. Is this available in an automated fashion or is it something one has to contact Google for?
- vgt 11y ago
  You can always create a dataset and make it public by setting open ACLs to it. The only downside is paying $0.02 per GB per month. To participate in the Public Datasets program, you must contact us - there's a lot of legal mumbo jumbo associated with us hosting your datasets that I wish weren't the case :)

kuschku 11y ago

Can we extract the datasets to work with them on our own (cheaper to query) servers?

vgt 11y ago
Of course! BigQuery export operations are free. Although compared with other cloud technologies, BigQuery is incredibly cost-effective :)
- kuschku 11y ago
  Well, I’ve checked the prices, and dedicated servers from old-style hosters are still far cheaper if you’re going to query data 24/7.
Bytes 11y ago
Most of the datasets seem to be available to download here https://www.reddit.com/r/bigquery/wiki/datasets https://www.reddit.com/r/bigquery/wiki/datasets.
- kuschku 11y ago
  After looking at those: What’s "big" about any of them? That’s literally a standard database size, and can be easily done with any postgres install. All of that data combined can be stored and quickly queried on just 10 dedicated servers, for an overall price of 200€/month. And with "all" I mean "all 20TB". And with "quickly" I mean "faster than a network request to Google". ________________ Due to rate limiting, I can't answer you at the moment, vgt. So my answer will happen here, inline: > Well, like this: > If someone wants to decide if they want to build a business on bigquery, they want to evaluate it's performance on an average data set before for free. > Assuming the test data is thought for similar cases it has to be of similar size to the average dataset used with BigQuery.
  - vgt 11y ago
    Apologies, but where do you find the implication that these datasets are "big"? There is "big" in BigQuery, so apologize if you misunderstood. We try to keep datasets reasonable, so that folks can get the most out of the BigQuery free pricing tier :) If you want bigger, here's an example of a 10TB dataset: https://cloud.google.com/genomics/data/1000-genomes https://cloud.google.com/genomics/data/1000-genomes

danvoell 11y ago

https://cloud.google.com/bigquery/public-data/hacker-news https://cloud.google.com/bigquery/public-data/hacker-news

th0br0 11y ago

I get 404s for all Looker iframes...

tosh 11y ago
Same here in Safari. Chrome seems to work.
apahwa 11y ago
where are you seeing Looker being used?
- dwmintz 11y ago
  Looker is powering all the iframes (any 404s should be fixed now). Bigquery is hosting the data, but Looker is generating the queries and visualizations of the data.
  - apahwa 11y ago
    Interesting, I'm surprised Google is using Looker rather than their own tool edit: ah, I see, it is just one of the partners they have with BigQuery and they are apparently using it for their website (I'm guessing because Looker lets you iframe graphs directly into the page for free).
    - dwmintz 11y ago
      Yup, Looker is a Premier Partner for the GCP launch (https://cloud.google.com/partners/?q=Looker#search https://cloud.google.com/partners/?q=Looker#search). And then we (I work at Looker) are specifically partnering with GCP for the Public Datasets project. Because LookML (Looker's modeling language) makes it easy to explore and visualize big datasets, we're building out models and dashboards so visitors can get a sense of BigQuery/Looker's power and find insights from the datasets quickly (whether or not they write SQL). Frankly, there's a ton of public data that's "available" in the sense that you can technically download and clean a CSV, but isn't actually easy to extract meaning from. So we figured we'd select some interesting datasets, do the cleaning, uploading and modeling, and then let folks have at it for free. If there are specific datasets you'd be interested in seeing, let us know and we'll see what we can do.

nl 11y ago

This is a much more useful list: https://www.reddit.com/r/bigquery/wiki/datasets https://www.reddit.com/r/bigquery/wiki/datasets I'm not sure why Google hosts it on Reddit. There's some interesting (and more up-to-date) stuff on there.

jpatokal 11y ago
That's a list of random data from random people, curated by Googler felipehoffa in an unofficial capacity. The ones in the submitted page are maintained by Google itself in the public-data project.
- nl 11y ago
  I believe that the GDelt, Freebase and Genomics tables (at least) are officially supported by Google.
kyrra 11y ago
This page may interest you: https://support.google.com/cloud/answer/3466163?hl=en https://support.google.com/cloud/answer/3466163?hl=en I think the Google Cloud team watches the community and helps promote the forums that have the most active for a given product.

talles 11y ago

Who is the King of Hacker News? https://cloud.google.com/bigquery/public-data/hacker-news#who_is_the_king_of_hacker_news https://cloud.google.com/bigquery/public-data/hacker-news#wh...

lloydt 11y ago
Click on the Stories Count and you can see the individual stories :)

pbh101 11y ago

I remember the earlier Google attempt at the same thing. I was in undergrad and an engineer came to my Uni to give a presentation on the topic. A big area of concern was "Why won't Google just shut this down when the going gets rough?" The question was mostly batted away along the lines of "this is lunch money to Google anyway." But just a couple years later, it did indeed bite the dust. Covered in Wired: http://www.wired.com/2008/01/google-to-provi/ http://www.wired.com/2008/01/google-to-provi/ Update: And the shutdown article, 11 months later: http://www.wired.com/2008/12/googlescienceda/ http://www.wired.com/2008/12/googlescienceda/

vgt 11y ago
Google BigQuery is a paid service, Dremel under the hood, used internally at Google by ~80% of Googlers every week. It's simply not going away because Google relies on it heavily. Externally, companies like Spotify and Kabam Games rely heavily on BigQuery as well: https://www.youtube.com/watch?v=LTVFg6YOjWo&list=PLIivdWyY5sqLpj5W-tdqT3O2NQ-_LgiPL&index=7 https://www.youtube.com/watch?v=LTVFg6YOjWo&list=PLIivdWyY5s... https://www.youtube.com/watch?v=6Nv18xmJirs&list=PLIivdWyY5sqLpj5W-tdqT3O2NQ-_LgiPL&index=6 https://www.youtube.com/watch?v=6Nv18xmJirs&list=PLIivdWyY5s... So the story of Google Cloud products, which are paid, supported, and Enterprise-grade, and which are covered by a clear 1 year deprecation policy, are very different from the story of free services. Public Datasets is just a function of BigQuery's unique ACL functionality.
- awqrre 11y ago
  Google employees using something internally doesn't mean that it won't go away for everyone else... Also, they could decide to honor existing contracts and stop issuing new ones...
  - vgt 11y ago
    BigQuery is a truly fully-managed service. This means that SREs are on call 24/7. It's not software that one downloads. Hence it really doesn't make a lot of sense to have it available to just a few folks, incremental overhead is low. I'd love to hear industry standards around deprecation of paid services. This is really a way of doing business in the entire world, and I'm not sure why Google is being held to an unreasonable and impossible standard of "never deprecate anything ever!". For example, here is Microsoft's: https://en.wikipedia.org/wiki/Category:Discontinued_Microsoft_software https://en.wikipedia.org/wiki/Category:Discontinued_Microsof...

bakztfuture 11y ago

Any chance we'll be seeing the Common Crawl data on there anytime soon?

vgt 11y ago
Sounds like a good one. We'll be sure to reach out to them. Or, if you know them, have them shoot us a note at: bq-public-data@google.com

tmannen 11y ago

I just noticed the Freebase data available as one of the public datasets, so been wondering what happened to the Freebase team? I liked what Freebase, DBpedia etc were doing few years ago with "semantic db's". I remember Freebase Gridworks being extremely useful for data cleanup. Haven't had a reason to follow the space...is it dead? Haven't seen much semantic talk off late. Where would one go to catch up on the latest news?

fhoffa 11y ago
Freebase data is being moved into Wikidata. I loaded Wikidata into BigQuery too! Fun with movies and cats: https://medium.com/google-cloud/oscars-2016-movies-that-got-the-most-attention-on-wikipedia-151cd56f4fc0 https://medium.com/google-cloud/oscars-2016-movies-that-got-... Implementation notes: https://lists.wikimedia.org/pipermail/wikidata/2016-March/008414.html https://lists.wikimedia.org/pipermail/wikidata/2016-March/00... More notes: https://lists.wikimedia.org/pipermail/wikidata/2016-March/008427.html https://lists.wikimedia.org/pipermail/wikidata/2016-March/00...
- krishna2 11y ago
  Super interesting. Thanks for all the curation and relevant set of links that you are promptly posting/replying. StackOverflow data is available for free too - that would be one awesome dataset to let folks get their hands on. Any plans for that?

flashman 11y ago

I just ran a Markov text generator over the USA Name Database and created a few thousand girls' names, including: Aracella, Ashla, Blakelyne, Carylou, Damariah, Enchantrelle, Francenza, Iridia, Jalexius, Lilliotte and Scotlanta. Hey, maybe I can start a bespoke baby name business...

alexwebb2 11y ago
I'm building a baby name app and this is a fantastic idea that I am probably going to steal.

temuze 11y ago

I hope someone submits a voice recognition data! It's fantastically hard to find audio->phoneme datasets and only slightly less difficult to find audio->word and word->possible_phonemes...

marchenko 11y ago

Looking a the suggested textual analyses of the Reddit and HN datasets, and thinking about how they can be combined, makes me wonder if anonymity-through.multiple-avatars will even be remotely possible 5 years from now.

cosmeen 11y ago

Seems like it has a dataset of hacker news since it launched in 2006 -- SELECT * FROM [bigquery-public-data:hacker_news.stories] ORDER BY score DESC LIMIT 100 -- made a screenshot with Top 20 stories on HN http://i.imgur.com/czXYEyQ.png http://i.imgur.com/czXYEyQ.png

dcdevito 11y ago

Sorry Google but my trust in cloud solutions reside in AWS and Azure. Why? Because when Amazon and Microsoft announce something I know there's a good chance it will still exist 24 months later

vgt 11y ago
This narrative gets repeated time and time again, and it really doesn't hold up to even surface debate: - Google TI and Cloud merged. Same teams. Per Urs Holzle in Jan2014 - Just last week Sundar (CEO), Eric Schmidt (chairman), Jeff Dean, Urs Holzle, and Diane were on stage talking about how they spent $10 billion last year investing in infrastructure and how serious Google is about cloud (https://www.youtube.com/watch?v=HgWHeT_OwHc&list=PLIivdWyY5sqIFd0E6JG1hVr8sXQaLmmBP https://www.youtube.com/watch?v=HgWHeT_OwHc&list=PLIivdWyY5s...) - In many cases, publically available technologies are actually replacing internal technologies (BigQuery and Dremel, Dataflow and FlumeJava/Millwheel for example). - Disney, Coca Cola, Snapchat, Spotify, Home Depot are just some of the customers running on Google Cloud. It'd be very foolish to abandon these. - If you watch the day two GCP NEXT keynote, you'll see how Google is the cloud leader in environmental responsibility and open source. We're the good guys :) (https://www.youtube.com/watch?v=axhdIa_co2o&list=PLIivdWyY5sqIFd0E6JG1hVr8sXQaLmmBP&index=2 https://www.youtube.com/watch?v=axhdIa_co2o&list=PLIivdWyY5s...) Happy to debate further!

javiramos 11y ago

There's a Hacker News dataset. In case anyone wants to take a crack at it: https://cloud.google.com/bigquery/public-data/hacker-news https://cloud.google.com/bigquery/public-data/hacker-news

deleted 11y ago

[deleted]

mark_l_watson 11y ago

Public data is good! I have used their book text data on two projects to generate common ngrams. These newly released data sets look useful and using BigQuery is reasonable if you just need parts of data sets. Amazon does something similar: keeping useful data in S3 for cheap access when running on AWS.

kwoff 11y ago

How can they copy our comments from here to there and sell queries on them...?