3 ms·

I have the same problem. I have 6M URL’s per domain. 8 different domains. 80% of search traffic is long tail. If I don’t block, 95% of my resources will be spe

I have the same problem. I have 6M URL’s per domain. 8 different domains. 80% of search traffic is long tail. If I don’t block, 95% of my resources will be spent on feeding bots. I had to block all “official” AI useragents and entire countries like Singapore and China. But there are so many unofficial bots which spread their work over dozes of IP addresses that it seems impossible to block on the reverse proxy level. How do you block those?

decremental 9mo ago
[dead]
kjok 9mo ago
Block based on cookies (i.e., set a cookie on the browser and check on the server whether it exists).
- direwolf20 9mo ago
  This project implements a variety of similar JSless checks, such as image loading https://github.com/WeebDataHoarder/go-away https://github.com/WeebDataHoarder/go-away
- bigbadfeline 9mo ago
  That helps, but the big bot farms use clients that support cookies, we need to add more defenses on top of them.
JasonADrury 9mo ago
>If I don’t block, 95% of my resources will be spent on feeding bots. Okay, but why should you care? Resource usage for a regular website that isn't doing some heavy dynamic stuff or video streaming tends to be rather negligible. You can easily serve 100k+ requests per second on servers that costs less than $100/mo. It really shouldn't be worth your time to do anything about the bots unless you're doing something unusual.
- spiderfarmer 9mo ago
  Believe it or not, but the website is not a static txt file.
  - JasonADrury 9mo ago
    Anything significantly more complicated than CRUD apps like HN is pretty rare on the web. If the resource usage of a website is a concern, either your code is straight up broken or you're doing something rather unusual. While doing unusual things, it's normal to encounter unusual problems. However, when encountering an unusual problem it's good to stop for a moment and consider if your approach is wrong. At some point the only good way to stop scraping becomes paywalls. You can't defeat sophisticated scrapers through any other means.
    - spiderfarmer 9mo ago
      So you’re blaming the destruction of the open internet on the technical prowess of indie developers like me and not on the greedy big tech leeches with thousands of mindless developers who do everything in their power to make life worse for the little guys.
      - JasonADrury 9mo ago
        I don't think the open internet is being destroyed at all. This is just the usual complaining about internet background noise that's been happening for decades. Is there more background noise than before? Yes, probably. Is it a big deal yet? Still not.
        bigbadfeline 9mo ago
        > Yes, probably. Is it a big deal yet? Still not. "Trust me bro, not a big deal... YET, pay up and move along, nothing to see here" That might be true for you, but it definitely isn't true for everybody, if you don't want to stop bots, nobody is stopping you from not stopping them, but you keep arguing as if your life depends on it... Are you a bot too? Who in his right mind would wait for some problem to become a really big deal without seeking a way to prevent it?
bigbadfeline 9mo ago
> If I don’t block, 95% of my resources will be spent on feeding bots. How do you block those? A very good and important question. I was thinking about some combination of proof of work and a dynamic list of offending IPs which servers would update periodically, similar to how such lists are used by ad-blockers. It would be ideal to have some legislative protection from ddos together with technical means of disclosure and prevention, bot scraping is a from of soft ddos.

decremental 9mo ago

[dead]

kjok 9mo ago

Block based on cookies (i.e., set a cookie on the browser and check on the server whether it exists).

direwolf20 9mo ago
This project implements a variety of similar JSless checks, such as image loading https://github.com/WeebDataHoarder/go-away https://github.com/WeebDataHoarder/go-away
bigbadfeline 9mo ago
That helps, but the big bot farms use clients that support cookies, we need to add more defenses on top of them.

JasonADrury 9mo ago

>If I don’t block, 95% of my resources will be spent on feeding bots. Okay, but why should you care? Resource usage for a regular website that isn't doing some heavy dynamic stuff or video streaming tends to be rather negligible. You can easily serve 100k+ requests per second on servers that costs less than $100/mo. It really shouldn't be worth your time to do anything about the bots unless you're doing something unusual.

spiderfarmer 9mo ago
Believe it or not, but the website is not a static txt file.
- JasonADrury 9mo ago
  Anything significantly more complicated than CRUD apps like HN is pretty rare on the web. If the resource usage of a website is a concern, either your code is straight up broken or you're doing something rather unusual. While doing unusual things, it's normal to encounter unusual problems. However, when encountering an unusual problem it's good to stop for a moment and consider if your approach is wrong. At some point the only good way to stop scraping becomes paywalls. You can't defeat sophisticated scrapers through any other means.
  - spiderfarmer 9mo ago
    So you’re blaming the destruction of the open internet on the technical prowess of indie developers like me and not on the greedy big tech leeches with thousands of mindless developers who do everything in their power to make life worse for the little guys.
    - JasonADrury 9mo ago
      I don't think the open internet is being destroyed at all. This is just the usual complaining about internet background noise that's been happening for decades. Is there more background noise than before? Yes, probably. Is it a big deal yet? Still not.
      - bigbadfeline 9mo ago
        > Yes, probably. Is it a big deal yet? Still not. "Trust me bro, not a big deal... YET, pay up and move along, nothing to see here" That might be true for you, but it definitely isn't true for everybody, if you don't want to stop bots, nobody is stopping you from not stopping them, but you keep arguing as if your life depends on it... Are you a bot too? Who in his right mind would wait for some problem to become a really big deal without seeking a way to prevent it?

bigbadfeline 9mo ago

> If I don’t block, 95% of my resources will be spent on feeding bots. How do you block those? A very good and important question. I was thinking about some combination of proof of work and a dynamic list of offending IPs which servers would update periodically, similar to how such lists are used by ad-blockers. It would be ideal to have some legislative protection from ddos together with technical means of disclosure and prevention, bot scraping is a from of soft ddos.