15 ms·

I run some forums, some of them are quite large. Recently the big increase in scraping by the search engines (Bing has had the greatest increase) caused me to q

by buro9 3y ago

I run some forums, some of them are quite large. Recently the big increase in scraping by the search engines (Bing has had the greatest increase) caused me to question why. It used to be that the cost of scraping came with the benefit of being search engine listed which drove traffic, but that feels less true than it used to (for a lot of reasons). But now the cost of scraping doesn't feel in the favour of a website. Scraping and bots are for search engines listing, technology tests / experiments, advert / audience measurements, brand protection, IP tracking, copyright enforcement, screenshots for links on other websites (i.e. Facebook), Pinterest linkbacks, training of LLMs (my hypothesis on Bing's massive increase), spam, etc, etc. With the search engine value lowered by less traffic, yet a solid community still growing via word of mouth... the rest of those things offer no value to me or the community. So I asked the community, what do you want to do here? Leave them all? Ban some? Ban all? Some midway thing? Almost unanimously the community (who fund the costs by donations, and at least 30% of all traffic and costs were known to be associated to bots) chose to block every bot. So that's what we've done. We've blocked every major hosting and Cloud ASN, or put a challenge up to the few known to be proxies (i.e. Google Data Saver), and we've blocked hundreds of bot user agents, we've blocked requests where no Accept header was present where it should be, we've blocked TLS ciphers that aren't modern web browsers — I looked at requests by Python, Go, Curl, Wget, etc... and blocked everything that obviously differed from a valid browser. In the end we blocked about 40% of our traffic, and so far not a single real human has said (and it's a tight-knit but large community with lots of ways of contacting me) that they've had any issue at all. We appear to have reduced our traffic and associated costs, with no loss to us at all.

gleenn 3y ago
I am definitely happy you asked your users what they thought and made your decision. But saying "no human complained" might not be a good metric if people use Google or whatever to discover your site or it's info. People don't complain about things they don't know exist.
- jacquesm 3y ago
  That's fine though. People will discover the site because other people recommend it. Probably higher quality too.
ajsnigrutin 3y ago
And you'll abandon those forums one day, and a month after, i'll search for some niche problem, someone will link to that forum, and there will be no archive(.org) page, no google cached page, no nothing to get the answer.
- buro9 3y ago
  I've only ever shuttered one forum, and that was at the behest of the community itself. A decade in and it had become a toxic place, everyone agreed that they wanted to give multiple new things a try and that the place in question should be deleted. Not archived, not available forever, but deleted and nothing kept of it. I obliged. When the day comes that I shutter another I'll ask the active members at the time what they want to happen to their data. They may desire to leave it as a resource, they may want to delete it, if there's a clear majority in the decision I'll go with whatever they desire. I value the choice of those whose data it is, who contributed to creating it, over anyone else's hypothetical needs.
  - cesarb 3y ago
    > When the day comes that I shutter another I'll ask the active members at the time what they want to happen to their data. You might not have that chance; unless you have a co-admin with full access to everything, the reason for the forum to shut down might be because you're no longer there.
    - KyeRussell 3y ago
      And? In this hypothetical situation where they “aren’t around”, I don’t think that people searching for answers to tech support issues are high on their list of concerns, and - probably - almost certainly those of the forum members.
      - hammyhavoc 3y ago
        "And" the availability of information is important. Respectfully, on most forums, I don't care about the community, I care about the content, that's why I'm there, to have discourse and generate meaningful value in the form of knowledge. If someone passes, yes, that sucks, but that's life, we're all snuffing it at some point. However, the world carries on spinning, and that information should continue to be available, especially if the forum is for a niche and frequently generates useful information. If a forum is becoming "toxic" then that sounds like a moderation problem.
iancarroll 3y ago
How do you have costs that are directly attributable to scraping? Unless you are using a serverless platform that bills per request or your pages are large enough that egress bandwidth is expensive enough, I’m not convinced most sites would save much doing this.
- buro9 3y ago
  It's a multi-tenant platform (about 300 forums, with the biggest being around 250K visitors per month), the database is on a vertically scaled box that is too excessive now the traffic has reduced, but I was able to delete a few of the Linodes that were horizontally scaling the API and Web UI (the Web UI is just a client of the API hence those could be saved too). I've also noticed that my cache hit rate is extraordinary now, which I assume is because humans read recent stuff and bots read the long-tail of old stuff.
  - mike_d 3y ago
    As someone who does targeted scraping of forums, I can say having a good open API and caching is probably the best way to decrease load. If you use Cloudflare, turn off their anti-bot stuff. It is far more efficient to let them just serve bots from the cache than having scrapers use tricks to bypass them and go directly to your origin server.
    - Spivak 3y ago
      Yeah there's basically no way I would ever read what you just wrote and choose to do anything but block harder even at higher cost. Bots end running around CF would guarantee turning on authenticated origin pull.
- qeternity 3y ago
  I'm not really sure how you arrive at the conclusion that only serverless platforms result in costs? It's not just 40% of cpu or egress, it's 40% of database load, 40% of logging, 40% of APM/instrumentation. 40% is 40%. Maybe 40% of their cost isn't enough to warrant whatever time these efforts cost them, but for many people out there it will be.
herbst 3y ago
About a year ago I noticed one of my websites going down about weekly for an hour or so. My website had one endpoint that was available in a few hundred thousand versions. Ment for the user, for the bots it was just a few thousand variants. Setup in sitemap, setup in robots and including the right meta tags. Ment to update every few months. But we'll, not with the bing bot. It ignored my timeouts and queried hundreds of thousands, for him identical, pages every single week. Not one connection, not two or three but about 10 IPs hammering my servers at once. No second between request, not even pausing when the server is going down. Something even 'bad bots' usually do. I assumed it was just any bot calling itself Bing. But no, it was their IP ranges. I blocked nearly all of their IPs. Which appears to be the only way to make sure it doesn't ddos me again. Bing is like 1% of my traffic, not even worth the hustle.
- 2000UltraDeluxe 3y ago
  Yeah, Bing has gone completely nuts the last six months or so. They'll happily send the equivalent of a small DDoS at sites hosted on shared hosting, knocking them completely offline for a while. Nuts.
  - herbst 3y ago
    Could have been in the last 6 months for me too. And yes its crazy, most my sites have between 8 and 16 concurrent database connections available. In real world this works for thousands of daily users. But for a bing bot it's simply not enough.
    - hammyhavoc 3y ago
      Why aren't you caching your forum as static pages for users who aren't logged in at the very least? E.g., rebuild the cache every x time as a cron task, but even then, every page load shouldn't be incurring database overhead if someone isn't even logged in. Equally, you can force a rebuild of cache for x relevant pages when someone posts a new thread or comment. Otherwise, if someone alt+clicks a bunch of a category's threads as they look interesting then you're going to have a bad time.
boomboomsubban 3y ago
Bizarrely, I can't remember the last time duckduckgo (basically bing) gave me a forum as a search result, though it used to regularly give me results from them. Maybe it's the admins blocking crawlers but it feels more like a conscious decision.
- acdha 3y ago
  I’ve been wondering how much of it comes down to optimizing for ad impressions. If you search, get a result, and it answers your question they sell one page of keyword ads. If you go back and forth a dozen times, they sell a dozen times as many impressions.
  - boomboomsubban 3y ago
    Given my usual behavior is to check two pages then add !g, where I check two pages and decide I don't need more info, I don't think that's a strong move.
    - acdha 3y ago
      I’m not saying it’s smart, just that I could easily imagine someone chasing the wrong metric or trying to balance revenue against the likelihood that you’ll stop using them. For example, in your scenario that’s still twice as many impressions so unless you make Google tour primary maybe that’s a win.
guggle 3y ago
> We've blocked every major hosting and Cloud ASN How did you do that ? Years ago there was a script to block AWS and I made substantial savings by running it.
- buro9 3y ago
  I designed most of, and built a chunk of, the WAF and firewall stuff at Cloudflare. That includes wirefilter (a wireshark display filter inspired firewall), and coupled with Cloudflare using maxmind you get to block ASNs in addition to other characteristics of the request. With that context, I used bgp.he.net to look up the big ones I know and then wrote the rules.
- alam2000 3y ago
  [dead]
- reincoder 3y ago
  You can try out our free IP to Country ASN database[0]. You can just grep the IP addresses by looking up the ASN or AS domains. Then just extract the IP address range and you should be good to go. [1] The paid databases comes with AS type (hosting, ISP, business etc.) and we have a VPN detection database as well. [0] https://ipinfo.io/developers/ip-to-country-asn-database https://ipinfo.io/developers/ip-to-country-asn-database [1] https://community.ipinfo.io/t/filtering-asn-database/395 https://community.ipinfo.io/t/filtering-asn-database/395
pers0n 3y ago
Can you share the blocklist?
ploum 3y ago
While I perfectly understand, I’m a bit worried about my own web browser ( Offpunk ) which uses python-requests and is thus very often associated with being a bot. The browser has the goal of being light and downloading only the text and pictures (no css, no js). So we have the same goal here.
- tenebrisalietum 3y ago
  change the user agent
AndrewCopeland 3y ago
Shameless plug, if you do not want to spend the time aggregating all datacenter IP addresses you can use the IPDetective.io API to easily detect of an IP address comes from a datacenter, VPN, proxy or botnet.
NetOpWibby 3y ago
Would you happen to have a list of your blocks?
tourmalinetaco 3y ago
If you aren’t doing so already, I highly suggest working alongside the Internet Archive to preserve the information on your forums. One day they will close down, and your users will want to see their posts, refer to now broken bookmarks, and generally access the information.

by buro9 3y ago

gleenn 3y ago

I am definitely happy you asked your users what they thought and made your decision. But saying "no human complained" might not be a good metric if people use Google or whatever to discover your site or it's info. People don't complain about things they don't know exist.

jacquesm 3y ago
That's fine though. People will discover the site because other people recommend it. Probably higher quality too.

ajsnigrutin 3y ago

And you'll abandon those forums one day, and a month after, i'll search for some niche problem, someone will link to that forum, and there will be no archive(.org) page, no google cached page, no nothing to get the answer.

buro9 3y ago
I've only ever shuttered one forum, and that was at the behest of the community itself. A decade in and it had become a toxic place, everyone agreed that they wanted to give multiple new things a try and that the place in question should be deleted. Not archived, not available forever, but deleted and nothing kept of it. I obliged. When the day comes that I shutter another I'll ask the active members at the time what they want to happen to their data. They may desire to leave it as a resource, they may want to delete it, if there's a clear majority in the decision I'll go with whatever they desire. I value the choice of those whose data it is, who contributed to creating it, over anyone else's hypothetical needs.
- cesarb 3y ago
  > When the day comes that I shutter another I'll ask the active members at the time what they want to happen to their data. You might not have that chance; unless you have a co-admin with full access to everything, the reason for the forum to shut down might be because you're no longer there.
  - KyeRussell 3y ago
    And? In this hypothetical situation where they “aren’t around”, I don’t think that people searching for answers to tech support issues are high on their list of concerns, and - probably - almost certainly those of the forum members.
    - hammyhavoc 3y ago
      "And" the availability of information is important. Respectfully, on most forums, I don't care about the community, I care about the content, that's why I'm there, to have discourse and generate meaningful value in the form of knowledge. If someone passes, yes, that sucks, but that's life, we're all snuffing it at some point. However, the world carries on spinning, and that information should continue to be available, especially if the forum is for a niche and frequently generates useful information. If a forum is becoming "toxic" then that sounds like a moderation problem.

iancarroll 3y ago

How do you have costs that are directly attributable to scraping? Unless you are using a serverless platform that bills per request or your pages are large enough that egress bandwidth is expensive enough, I’m not convinced most sites would save much doing this.

buro9 3y ago
It's a multi-tenant platform (about 300 forums, with the biggest being around 250K visitors per month), the database is on a vertically scaled box that is too excessive now the traffic has reduced, but I was able to delete a few of the Linodes that were horizontally scaling the API and Web UI (the Web UI is just a client of the API hence those could be saved too). I've also noticed that my cache hit rate is extraordinary now, which I assume is because humans read recent stuff and bots read the long-tail of old stuff.
- mike_d 3y ago
  As someone who does targeted scraping of forums, I can say having a good open API and caching is probably the best way to decrease load. If you use Cloudflare, turn off their anti-bot stuff. It is far more efficient to let them just serve bots from the cache than having scrapers use tricks to bypass them and go directly to your origin server.
  - Spivak 3y ago
    Yeah there's basically no way I would ever read what you just wrote and choose to do anything but block harder even at higher cost. Bots end running around CF would guarantee turning on authenticated origin pull.
qeternity 3y ago
I'm not really sure how you arrive at the conclusion that only serverless platforms result in costs? It's not just 40% of cpu or egress, it's 40% of database load, 40% of logging, 40% of APM/instrumentation. 40% is 40%. Maybe 40% of their cost isn't enough to warrant whatever time these efforts cost them, but for many people out there it will be.

herbst 3y ago

About a year ago I noticed one of my websites going down about weekly for an hour or so. My website had one endpoint that was available in a few hundred thousand versions. Ment for the user, for the bots it was just a few thousand variants. Setup in sitemap, setup in robots and including the right meta tags. Ment to update every few months. But we'll, not with the bing bot. It ignored my timeouts and queried hundreds of thousands, for him identical, pages every single week. Not one connection, not two or three but about 10 IPs hammering my servers at once. No second between request, not even pausing when the server is going down. Something even 'bad bots' usually do. I assumed it was just any bot calling itself Bing. But no, it was their IP ranges. I blocked nearly all of their IPs. Which appears to be the only way to make sure it doesn't ddos me again. Bing is like 1% of my traffic, not even worth the hustle.

2000UltraDeluxe 3y ago
Yeah, Bing has gone completely nuts the last six months or so. They'll happily send the equivalent of a small DDoS at sites hosted on shared hosting, knocking them completely offline for a while. Nuts.
- herbst 3y ago
  Could have been in the last 6 months for me too. And yes its crazy, most my sites have between 8 and 16 concurrent database connections available. In real world this works for thousands of daily users. But for a bing bot it's simply not enough.
  - hammyhavoc 3y ago
    Why aren't you caching your forum as static pages for users who aren't logged in at the very least? E.g., rebuild the cache every x time as a cron task, but even then, every page load shouldn't be incurring database overhead if someone isn't even logged in. Equally, you can force a rebuild of cache for x relevant pages when someone posts a new thread or comment. Otherwise, if someone alt+clicks a bunch of a category's threads as they look interesting then you're going to have a bad time.

boomboomsubban 3y ago

Bizarrely, I can't remember the last time duckduckgo (basically bing) gave me a forum as a search result, though it used to regularly give me results from them. Maybe it's the admins blocking crawlers but it feels more like a conscious decision.

acdha 3y ago
I’ve been wondering how much of it comes down to optimizing for ad impressions. If you search, get a result, and it answers your question they sell one page of keyword ads. If you go back and forth a dozen times, they sell a dozen times as many impressions.
- boomboomsubban 3y ago
  Given my usual behavior is to check two pages then add !g, where I check two pages and decide I don't need more info, I don't think that's a strong move.
  - acdha 3y ago
    I’m not saying it’s smart, just that I could easily imagine someone chasing the wrong metric or trying to balance revenue against the likelihood that you’ll stop using them. For example, in your scenario that’s still twice as many impressions so unless you make Google tour primary maybe that’s a win.

guggle 3y ago

> We've blocked every major hosting and Cloud ASN How did you do that ? Years ago there was a script to block AWS and I made substantial savings by running it.

buro9 3y ago
I designed most of, and built a chunk of, the WAF and firewall stuff at Cloudflare. That includes wirefilter (a wireshark display filter inspired firewall), and coupled with Cloudflare using maxmind you get to block ASNs in addition to other characteristics of the request. With that context, I used bgp.he.net to look up the big ones I know and then wrote the rules.
alam2000 3y ago
[dead]
reincoder 3y ago
You can try out our free IP to Country ASN database[0]. You can just grep the IP addresses by looking up the ASN or AS domains. Then just extract the IP address range and you should be good to go. [1] The paid databases comes with AS type (hosting, ISP, business etc.) and we have a VPN detection database as well. [0] https://ipinfo.io/developers/ip-to-country-asn-database https://ipinfo.io/developers/ip-to-country-asn-database [1] https://community.ipinfo.io/t/filtering-asn-database/395 https://community.ipinfo.io/t/filtering-asn-database/395

pers0n 3y ago

Can you share the blocklist?

ploum 3y ago

While I perfectly understand, I’m a bit worried about my own web browser ( Offpunk ) which uses python-requests and is thus very often associated with being a bot. The browser has the goal of being light and downloading only the text and pictures (no css, no js). So we have the same goal here.

tenebrisalietum 3y ago
change the user agent

AndrewCopeland 3y ago

Shameless plug, if you do not want to spend the time aggregating all datacenter IP addresses you can use the IPDetective.io API to easily detect of an IP address comes from a datacenter, VPN, proxy or botnet.

NetOpWibby 3y ago

Would you happen to have a list of your blocks?

tourmalinetaco 3y ago

If you aren’t doing so already, I highly suggest working alongside the Internet Archive to preserve the information on your forums. One day they will close down, and your users will want to see their posts, refer to now broken bookmarks, and generally access the information.