3 ms·

>If I don’t block, 95% of my resources will be spent on feeding bots. Okay, but why should you care? Resource usage for a regular website that isn't doing some

>If I don’t block, 95% of my resources will be spent on feeding bots. Okay, but why should you care? Resource usage for a regular website that isn't doing some heavy dynamic stuff or video streaming tends to be rather negligible. You can easily serve 100k+ requests per second on servers that costs less than $100/mo. It really shouldn't be worth your time to do anything about the bots unless you're doing something unusual.

spiderfarmer 9mo ago
Believe it or not, but the website is not a static txt file.
- JasonADrury 9mo ago
  Anything significantly more complicated than CRUD apps like HN is pretty rare on the web. If the resource usage of a website is a concern, either your code is straight up broken or you're doing something rather unusual. While doing unusual things, it's normal to encounter unusual problems. However, when encountering an unusual problem it's good to stop for a moment and consider if your approach is wrong. At some point the only good way to stop scraping becomes paywalls. You can't defeat sophisticated scrapers through any other means.
  - spiderfarmer 9mo ago
    So you’re blaming the destruction of the open internet on the technical prowess of indie developers like me and not on the greedy big tech leeches with thousands of mindless developers who do everything in their power to make life worse for the little guys.
    - JasonADrury 9mo ago
      I don't think the open internet is being destroyed at all. This is just the usual complaining about internet background noise that's been happening for decades. Is there more background noise than before? Yes, probably. Is it a big deal yet? Still not.
      - bigbadfeline 9mo ago
        > Yes, probably. Is it a big deal yet? Still not. "Trust me bro, not a big deal... YET, pay up and move along, nothing to see here" That might be true for you, but it definitely isn't true for everybody, if you don't want to stop bots, nobody is stopping you from not stopping them, but you keep arguing as if your life depends on it... Are you a bot too? Who in his right mind would wait for some problem to become a really big deal without seeking a way to prevent it?
        JasonADrury 9mo ago
        > That might be true for you, but it definitely isn't true for everybody, if you don't want to stop bots, nobody is stopping you from not stopping them, but you keep arguing as if your life depends on it... Are you a bot too? Feel free to "stop the bots", it just probably isn't a sensible use of your time. >Who in his right mind would wait for some problem to become a really big deal without seeking a way to prevent it? You're starting from the huge assumption that this will ever become a big deal.
bigbadfeline 9mo ago
>> If I don’t block, 95% of my resources will be spent on feeding bots... How do you block those? A very important question which deserves a good answer. > Okay, but why should you care? Not that kind of answer - this is bad manners or worse. The counter-question "Why do you ask that" isn't an honest or meaningful answer - it's indefensible defense of allowing meaningless traffic without any defenses. > Resource usage for a regular website that isn't doing some heavy dynamic stuff or video streaming tends to be rather negligible. Maybe it is doing video streaming, or audio, or lots of images or very dynamic, or written in Python - the trend is to use more and more bandwidth to make the sites more attractive. > You can easily serve 100k+ requests per second on servers that costs less than $100/mo. Maybe $100/mo isn't a trivial amount for a site that has no video, isn't dynamic, etc - your assumptions contradict themselves and reality. Without any maybe, the prices of RAM, bandwidth and hosting are going up while usage limits are going down - inflation. There's no reason to sacrifice $100/mo to hostile bot daemons which are sure to ask for more and more in the future. It's absolutely clear that giving a free reign to bots will encourage more bots and more sinister behavior because the boundary between bot scraping and ddos is blurry, you're essentially arguing for allowing soft ddos which can be turned up at times just to make a site ineffective when it's needed.
- JasonADrury 9mo ago
  >Not that kind of answer - this is bad manners or worse. The counter-question "Why do you ask that" isn't an honest or meaningful answer - it's indefensible defense of allowing meaningless traffic without any defenses. Nah. Based on years of experience, the typical person asking this question is asking because they're bothered by log entries. They're not asking it because the requests are actually being somehow disruptive. The correct answer is "don't stress out about normal background noise". >Maybe $100/mo isn't a trivial amount for a site that has no video, isn't dynamic, etc - your assumptions contradict themselves and reality. $100/mo is a trivial amount for anything that is visited hundred thousand times in a second. >It's absolutely clear that giving a free reign to bots will encourage more bots and more sinister behavior because the boundary between bot scraping and ddos is blurry, you're essentially arguing for allowing soft ddos which can be turned up at times just to make a site ineffective when it's needed. Not really. The bots have a strong incentive to not be disruptive. >Without any maybe, the prices of RAM, bandwidth and hosting are going up while usage limits are going down - inflation. There's no reason to sacrifice $100/mo to hostile bot daemons which are sure to ask for more and more in the future. Prices of bandwidth and hosting are not going up, usage limits are not going down. This is not a real thing that's happening. Servers (and bandwidth) are in fact cheaper than ever, except perhaps for the RAM.

spiderfarmer 9mo ago

Believe it or not, but the website is not a static txt file.

JasonADrury 9mo ago
Anything significantly more complicated than CRUD apps like HN is pretty rare on the web. If the resource usage of a website is a concern, either your code is straight up broken or you're doing something rather unusual. While doing unusual things, it's normal to encounter unusual problems. However, when encountering an unusual problem it's good to stop for a moment and consider if your approach is wrong. At some point the only good way to stop scraping becomes paywalls. You can't defeat sophisticated scrapers through any other means.
- spiderfarmer 9mo ago
  So you’re blaming the destruction of the open internet on the technical prowess of indie developers like me and not on the greedy big tech leeches with thousands of mindless developers who do everything in their power to make life worse for the little guys.
  - JasonADrury 9mo ago
    I don't think the open internet is being destroyed at all. This is just the usual complaining about internet background noise that's been happening for decades. Is there more background noise than before? Yes, probably. Is it a big deal yet? Still not.
    - bigbadfeline 9mo ago
      > Yes, probably. Is it a big deal yet? Still not. "Trust me bro, not a big deal... YET, pay up and move along, nothing to see here" That might be true for you, but it definitely isn't true for everybody, if you don't want to stop bots, nobody is stopping you from not stopping them, but you keep arguing as if your life depends on it... Are you a bot too? Who in his right mind would wait for some problem to become a really big deal without seeking a way to prevent it?
      - JasonADrury 9mo ago
        > That might be true for you, but it definitely isn't true for everybody, if you don't want to stop bots, nobody is stopping you from not stopping them, but you keep arguing as if your life depends on it... Are you a bot too? Feel free to "stop the bots", it just probably isn't a sensible use of your time. >Who in his right mind would wait for some problem to become a really big deal without seeking a way to prevent it? You're starting from the huge assumption that this will ever become a big deal.

bigbadfeline 9mo ago

>> If I don’t block, 95% of my resources will be spent on feeding bots... How do you block those? A very important question which deserves a good answer. > Okay, but why should you care? Not that kind of answer - this is bad manners or worse. The counter-question "Why do you ask that" isn't an honest or meaningful answer - it's indefensible defense of allowing meaningless traffic without any defenses. > Resource usage for a regular website that isn't doing some heavy dynamic stuff or video streaming tends to be rather negligible. Maybe it is doing video streaming, or audio, or lots of images or very dynamic, or written in Python - the trend is to use more and more bandwidth to make the sites more attractive. > You can easily serve 100k+ requests per second on servers that costs less than $100/mo. Maybe $100/mo isn't a trivial amount for a site that has no video, isn't dynamic, etc - your assumptions contradict themselves and reality. Without any maybe, the prices of RAM, bandwidth and hosting are going up while usage limits are going down - inflation. There's no reason to sacrifice $100/mo to hostile bot daemons which are sure to ask for more and more in the future. It's absolutely clear that giving a free reign to bots will encourage more bots and more sinister behavior because the boundary between bot scraping and ddos is blurry, you're essentially arguing for allowing soft ddos which can be turned up at times just to make a site ineffective when it's needed.

JasonADrury 9mo ago
>Not that kind of answer - this is bad manners or worse. The counter-question "Why do you ask that" isn't an honest or meaningful answer - it's indefensible defense of allowing meaningless traffic without any defenses. Nah. Based on years of experience, the typical person asking this question is asking because they're bothered by log entries. They're not asking it because the requests are actually being somehow disruptive. The correct answer is "don't stress out about normal background noise". >Maybe $100/mo isn't a trivial amount for a site that has no video, isn't dynamic, etc - your assumptions contradict themselves and reality. $100/mo is a trivial amount for anything that is visited hundred thousand times in a second. >It's absolutely clear that giving a free reign to bots will encourage more bots and more sinister behavior because the boundary between bot scraping and ddos is blurry, you're essentially arguing for allowing soft ddos which can be turned up at times just to make a site ineffective when it's needed. Not really. The bots have a strong incentive to not be disruptive. >Without any maybe, the prices of RAM, bandwidth and hosting are going up while usage limits are going down - inflation. There's no reason to sacrifice $100/mo to hostile bot daemons which are sure to ask for more and more in the future. Prices of bandwidth and hosting are not going up, usage limits are not going down. This is not a real thing that's happening. Servers (and bandwidth) are in fact cheaper than ever, except perhaps for the RAM.