4 ms·

Tell HN: Hacker News BigQuery dataset is updated daily again

51 points by wietsevenema 2y ago

As you might know, there's a Hacker News BigQuery dataset [1]. I recently noticed that the export was broken (since Nov 2022) and I worked with the team to help fix it. It'll now again run daily. SELECT timestamp FROM `bigquery-public-data.hacker_news.full` ORDER BY id DESC LIMIT 1 # 2024-06-09 09:12:55 UTC [1]: https://console.cloud.google.com/marketplace/product/y-combinator/hacker-news

dang 2y ago
Thanks! I tried emailing my contacts on that team several times and never heard back, so I figured we no longer had contacts on the that team.
- izyda 2y ago
  Any interest in getting this dataset on Snowflake too? (I run a Snowflake funded data provider on Snowflake marketplace)
  - dang 2y ago
    The BigQuery dataset is entirely 3rd party, based on the public HN data (presumably via the HN Firebase API - https://github.com/HackerNews/API https://github.com/HackerNews/API), so anyone is free to do this.
    - Raed667 2y ago
      a bit off topic: is there a way to display a HN page using the firebase API/sdk without doing N+1 requests ? (sorry for noob question)
      - dang 2y ago
        No, unfortunately. We're going to make a new API that will be much easier, but there's at least one other major project that needs to get done first.
        Raed667 2y ago
        Thanks for the response!
    - wietsevenema 2y ago
      Yes, that’s how it works. Same data as in the API.
throwaway211 2y ago
How big is it?
- wietsevenema 2y ago
  Closing in on 41 million items
zX41ZdbW 2y ago
Daily is too infrequently. I have this dataset being updated in ClickHouse in real time: https://play.clickhouse.com/play?user=play#U0VMRUNUIG1heCh0aW1lKSBGUk9NIGhhY2tlcm5ld3NfaGlzdG9yeQ== https://play.clickhouse.com/play?user=play#U0VMRUNUIG1heCh0a... I also provide a way to export it or attach it to clickhouse-local and analyze it locally: https://github.com/ClickHouse/ClickHouse/issues/29693#issuecomment-2185388881 https://github.com/ClickHouse/ClickHouse/issues/29693#issuec...
- tarasglek 2y ago
  I love that this dataset is accessible via cors from web frontends. Got me into experiments with clickhouse Same with github, much better than than throttled apis

51 points by wietsevenema 2y ago

dang 2y ago

Thanks! I tried emailing my contacts on that team several times and never heard back, so I figured we no longer had contacts on the that team.

izyda 2y ago
Any interest in getting this dataset on Snowflake too? (I run a Snowflake funded data provider on Snowflake marketplace)
- dang 2y ago
  The BigQuery dataset is entirely 3rd party, based on the public HN data (presumably via the HN Firebase API - https://github.com/HackerNews/API https://github.com/HackerNews/API), so anyone is free to do this.
  - Raed667 2y ago
    a bit off topic: is there a way to display a HN page using the firebase API/sdk without doing N+1 requests ? (sorry for noob question)
    - dang 2y ago
      No, unfortunately. We're going to make a new API that will be much easier, but there's at least one other major project that needs to get done first.
      - Raed667 2y ago
        Thanks for the response!
  - wietsevenema 2y ago
    Yes, that’s how it works. Same data as in the API.

throwaway211 2y ago

How big is it?

wietsevenema 2y ago
Closing in on 41 million items

zX41ZdbW 2y ago

Daily is too infrequently. I have this dataset being updated in ClickHouse in real time: https://play.clickhouse.com/play?user=play#U0VMRUNUIG1heCh0aW1lKSBGUk9NIGhhY2tlcm5ld3NfaGlzdG9yeQ== https://play.clickhouse.com/play?user=play#U0VMRUNUIG1heCh0a... I also provide a way to export it or attach it to clickhouse-local and analyze it locally: https://github.com/ClickHouse/ClickHouse/issues/29693#issuecomment-2185388881 https://github.com/ClickHouse/ClickHouse/issues/29693#issuec...

tarasglek 2y ago
I love that this dataset is accessible via cors from web frontends. Got me into experiments with clickhouse Same with github, much better than than throttled apis