3 ms·
What is the best way to download this dataset? Last time I messed with it I had to pay for a Google Cloud bucket and run through some awkward sequence of steps
by bigdict 6y ago
What is the best way to download this dataset? Last time I messed with it I had to pay for a Google Cloud bucket and run through some awkward sequence of steps to eventually get a local copy.
I think I ended up following the advice here: https://stackoverflow.com/questions/18493533/how-to-download-all-data-in-a-google-bigquery-dataset https://stackoverflow.com/questions/18493533/how-to-download....
- minimaxir 6y agoThat's essentially it (export the BQ table as a CSV to a Google Cloud Storage bucket, then download it from there), but you can do that entirely in the web UI, no CLI needed. If you just want a subset of the data, run a query, then save the query as a table in your project and export from there.
- wslh 6y agoDoes it cost money to do that or it is only time consuming?
- minimaxir 6y agoIt’s not free: Only storage and egress, which are trivial. Running the query in BQ is free up to 1 TB and the 16000 row download is free so I recommend that if necessary.
- wslh 6y agoI think an alternative option could be to have a torrent (or other file sharing mechanism) with the public HN information. Am I missing something? 16k rows seems very tiny for doing an analysis of HN.
- minimaxir 6y agoThe vast majority of potential HN analysis can be done in SQL alone within BigQuery, which is the proper way to handle a data warehouse regardless. You mostly need the raw data for ML model training which is a very niche use case.
- ZeroCool2u 6y agoYou can read it directly into a pandas DataFrame. [1] https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_gbq.html https://pandas.pydata.org/pandas-docs/stable/reference/api/p...