3 ms·
How does the data get to BigQuery? Anything special/fun or just repeatedly polling the API endpoint?
by koolba 10y ago
How does the data get to BigQuery? Anything special/fun or just repeatedly polling the API endpoint?
- fhoffa 10y agoI'm not sure if we are ready to document the process, but I can tell you that having an API based on Firebase helps a lot: - https://github.com/HackerNews/API https://github.com/HackerNews/API
- koolba 10y agoI'd be interested in reading up on it if you ever do write it up. I put something similar together (for HN data). One hitch is tracking historical changes as I don't think you can get a raw feed of it so stuck with polling a historical range to ensure you get the latest version of a comment.
- fhoffa 10y agoWe need to test this - but I'm almost sure we have the latest version of each comment in BigQuery.