4 ms·
Beautiful! Thanks for open sourcing this! I've been working on a project [1] that probably wants to become a live crawler like this, but it's currently batch b
by 8organicbits 2y ago
Beautiful! Thanks for open sourcing this!
I've been working on a project [1] that probably wants to become a live crawler like this, but it's currently batch based. I'm focused on RSS feeds and microformats [2]. Can you share any details on what kind of performance / operational costs you're seeing while you're on the HN front page? The fly.toml looks like $5/month could suffice?
[1] https://alexsci.com/rss-blogroll-network/ https://alexsci.com/rss-blogroll-network/
[2] https://microformats.org/wiki/Main_Page https://microformats.org/wiki/Main_Page
- spaceship__sun 2y agoI'm not OP; I received ~100 thousand requests being on the front page once. It was an AI app and I quickly got rate limited to GCP Vertex AI lol.
- SahAssar 2y ago> 100 thousand requests Over how long time? Even if it's just over an hour that's just under 30 rps, over a day it's a little over 1 rps.