2 ms·
> It would be cool if we could use the public Big Query data set in Google cloud, but it is missing some data that we'd like to report on. Namely users as well
by minimaxir 8y ago
> It would be cool if we could use the public Big Query data set in Google cloud, but it is missing some data that we'd like to report on. Namely users as well as enhancing the posts with fields like (domain name of a story).
The BigQuery dataset has both of the fields necessary to do the analysis, so I'm confused. (you do need to use the NET.HOST function to extract a HN-esque domain name from the url; more info in this notebook: https://www.kaggle.com/minimaxir/hacker-news-submission-score-predictor https://www.kaggle.com/minimaxir/hacker-news-submission-scor...)
- harrisreynolds 8y agoThanks for the link! Will take a look. Would love to dump our Redshift instance!
- sbr464 8y agoI'm curious, what's the current size of your instance (disk space/# records) for this dataset? I'd be interested in a dump of the raw data, to save having to access via the API. I'd be willing to pay for the bandwidth/temp charges if any.
- scapecast 8y agoWhat is it that makes you want to dump Redshift? Sounds like there's a specific frustration?
- harrisreynolds 8y agoI actually like Redshift fine. But we are paying about $250/month for it just to host this data and if it was already in a publicly available database it would be nice to drop that. But using Redshift for this has been great for us in terms of testing our tool.