4 ms·

How do you have costs that are directly attributable to scraping? Unless you are using a serverless platform that bills per request or your pages are large enou

How do you have costs that are directly attributable to scraping? Unless you are using a serverless platform that bills per request or your pages are large enough that egress bandwidth is expensive enough, I’m not convinced most sites would save much doing this.

buro9 3y ago
It's a multi-tenant platform (about 300 forums, with the biggest being around 250K visitors per month), the database is on a vertically scaled box that is too excessive now the traffic has reduced, but I was able to delete a few of the Linodes that were horizontally scaling the API and Web UI (the Web UI is just a client of the API hence those could be saved too). I've also noticed that my cache hit rate is extraordinary now, which I assume is because humans read recent stuff and bots read the long-tail of old stuff.
- mike_d 3y ago
  As someone who does targeted scraping of forums, I can say having a good open API and caching is probably the best way to decrease load. If you use Cloudflare, turn off their anti-bot stuff. It is far more efficient to let them just serve bots from the cache than having scrapers use tricks to bypass them and go directly to your origin server.
  - Spivak 3y ago
    Yeah there's basically no way I would ever read what you just wrote and choose to do anything but block harder even at higher cost. Bots end running around CF would guarantee turning on authenticated origin pull.
qeternity 3y ago
I'm not really sure how you arrive at the conclusion that only serverless platforms result in costs? It's not just 40% of cpu or egress, it's 40% of database load, 40% of logging, 40% of APM/instrumentation. 40% is 40%. Maybe 40% of their cost isn't enough to warrant whatever time these efforts cost them, but for many people out there it will be.
- iancarroll 3y ago
  Sounds like the original commenter had a reasonable case, but I just don’t think it’s likely to save anything for small sites on traditional stacks. If you are running on i.e. EC2 and RDS instances, you’re not saving anything by using 40% less of the CPU, unless you can actually downsize the instance as a result. Read-only traffic is also not that hard to scale out, but with forums etc, you can be stuck with some legacy systems for sure.

buro9 3y ago

It's a multi-tenant platform (about 300 forums, with the biggest being around 250K visitors per month), the database is on a vertically scaled box that is too excessive now the traffic has reduced, but I was able to delete a few of the Linodes that were horizontally scaling the API and Web UI (the Web UI is just a client of the API hence those could be saved too). I've also noticed that my cache hit rate is extraordinary now, which I assume is because humans read recent stuff and bots read the long-tail of old stuff.

mike_d 3y ago
As someone who does targeted scraping of forums, I can say having a good open API and caching is probably the best way to decrease load. If you use Cloudflare, turn off their anti-bot stuff. It is far more efficient to let them just serve bots from the cache than having scrapers use tricks to bypass them and go directly to your origin server.
- Spivak 3y ago
  Yeah there's basically no way I would ever read what you just wrote and choose to do anything but block harder even at higher cost. Bots end running around CF would guarantee turning on authenticated origin pull.

qeternity 3y ago

I'm not really sure how you arrive at the conclusion that only serverless platforms result in costs? It's not just 40% of cpu or egress, it's 40% of database load, 40% of logging, 40% of APM/instrumentation. 40% is 40%. Maybe 40% of their cost isn't enough to warrant whatever time these efforts cost them, but for many people out there it will be.

iancarroll 3y ago
Sounds like the original commenter had a reasonable case, but I just don’t think it’s likely to save anything for small sites on traditional stacks. If you are running on i.e. EC2 and RDS instances, you’re not saving anything by using 40% less of the CPU, unless you can actually downsize the instance as a result. Read-only traffic is also not that hard to scale out, but with forums etc, you can be stuck with some legacy systems for sure.