4 ms·

Nice, so it periodically retrieves a list of HN posts and queries that list locally, so you're not telling algolia any specific site details. There are million

by gitgud 6y ago

Nice, so it periodically retrieves a list of HN posts and queries that list locally, so you're not telling algolia any specific site details. There are millions of posts on HN, how many submissions does it retrieve? Surely if you find an obscure site, it might not be in the local list. Maybe you could just always query many urls together, with only 1 of them being the real url you want. That would make it hard to track too. Also I made a similar extension, but it queries sources on every page load. (extension) https://newsit.benwinding.com/ https://newsit.benwinding.com/ (source) https://github.com/benwinding/newsit https://github.com/benwinding/newsit

jstrieb 6y ago
The extension retrieves a (few) Bloom filter(s). In case you're unfamiliar, Bloom filters are data structures approximately equivalent to condensed representations of all of the post URLs. The code uses GitHub Actions to generate Bloom filters once every 24 hours, corresponding to the update frequency of the Hacker News BigQuery database that it pulls from. The 16MB Bloom filter for all stories ever has about 4M entries at the moment. As far as I know, this is every single story that has been previously submitted to HN. Since Bloom filters are just long bit strings, you can OR them together to add stuff. This means you can download a smaller Bloom filter to update your current one, rather than downloading 16MB files all the time. The extension also auto-adds any stories linked on news.ycombinator.com pages you visit, so that stories too new to be in the Bloom filter are still added.
- gitgud 6y ago
  Thanks for clearing that up, yes I'm not that familiar with Bloom filters, seems like an interesting and useful concept. It could probably (pun intended) be applied to many applications to increase privacy. I like the [1] Workflow file you've made, the comments really help with reading shell code. I'm also amazed you can query 4M entries everyday with BigQuery, I thought that might be fairly expensive to do right? Or is this below a free tier? [1] https://github.com/jstrieb/hackernews-button/actions/runs/611181332/workflow https://github.com/jstrieb/hackernews-button/actions/runs/61...
  - jstrieb 6y ago
    Thank you for reading the code! The comments will be helpful for me when I have to fix something six months from now, but I'm glad to hear they're useful to others, too :) Querying that many entries processes (according to the BigQuery web interface) less than 300MB each time. Not sure how they arrive at that figure, but whatever. BigQuery has a free tier that includes up to 1TB/month of data processing. Since the Bloom filter is compiled once each day, I'm well under that limit. https://cloud.google.com/bigquery/pricing#free-tier https://cloud.google.com/bigquery/pricing#free-tier

by gitgud 6y ago

jstrieb 6y ago

The extension retrieves a (few) Bloom filter(s). In case you're unfamiliar, Bloom filters are data structures approximately equivalent to condensed representations of all of the post URLs. The code uses GitHub Actions to generate Bloom filters once every 24 hours, corresponding to the update frequency of the Hacker News BigQuery database that it pulls from. The 16MB Bloom filter for all stories ever has about 4M entries at the moment. As far as I know, this is every single story that has been previously submitted to HN. Since Bloom filters are just long bit strings, you can OR them together to add stuff. This means you can download a smaller Bloom filter to update your current one, rather than downloading 16MB files all the time. The extension also auto-adds any stories linked on news.ycombinator.com pages you visit, so that stories too new to be in the Bloom filter are still added.

gitgud 6y ago
Thanks for clearing that up, yes I'm not that familiar with Bloom filters, seems like an interesting and useful concept. It could probably (pun intended) be applied to many applications to increase privacy. I like the [1] Workflow file you've made, the comments really help with reading shell code. I'm also amazed you can query 4M entries everyday with BigQuery, I thought that might be fairly expensive to do right? Or is this below a free tier? [1] https://github.com/jstrieb/hackernews-button/actions/runs/611181332/workflow https://github.com/jstrieb/hackernews-button/actions/runs/61...
- jstrieb 6y ago
  Thank you for reading the code! The comments will be helpful for me when I have to fix something six months from now, but I'm glad to hear they're useful to others, too :) Querying that many entries processes (according to the BigQuery web interface) less than 300MB each time. Not sure how they arrive at that figure, but whatever. BigQuery has a free tier that includes up to 1TB/month of data processing. Since the Bloom filter is compiled once each day, I'm well under that limit. https://cloud.google.com/bigquery/pricing#free-tier https://cloud.google.com/bigquery/pricing#free-tier