11 ms·

>We’ve already done the work to render the page, and we’re trying to shed load, so why would I want to increase load by generating challenges and verifying resp

>We’ve already done the work to render the page, and we’re trying to shed load, so why would I want to increase load by generating challenges and verifying responses? It annoys me when I click a seemingly popular blog post and immediately get challenged, when I’m 99.9% certain that somebody else clicked it two seconds before me. Why isn’t it in cache? We must have different objectives in what we’re trying to accomplish. Or who we’re trying to irritate. +1000 I feel like so much bot detection (and fraud prevention against human actors, too) is so emotionally-driven. Some people hate these things so much, they're willing to cut off their nose to spite their face.

jitl 1y ago
Really? If I’m an unsophisticated blog not using a CDN, and I get a $1000 bill for bandwidth overage or something, I’m gonna google a solve and slap it on there because I don’t want to pay another $1000 for Big Basilisk. I don’t think that’s emotional response, it’s common sense.
- phantompeace 1y ago
  Wouldn't it be easier to put the unsophisticated blog behind cloudflare
  - mhuffman 1y ago
    As much as I like to shit on cloudflare at every opportunity, it would obviously be easier to put it behind CF than install bot detection plugins.
- marginalia_nu 1y ago
  Seems like you've made profoundly questionable hosting or design choices for that to happen. Flat rate web hosting exists, and blogs (especially unsophisticated ones) do not require much bandwidth or processing power. Misbehaving crawlers are a huge problem but bloggers are among the least affected by them. Something like a wiki or a forum is a better example, as they're in a category of websites where each page visit is almost unavoidably rendered on the fly using multiple expensive SQL queries due to the rapidly mutating nature of their datasets. Git forges, like the one TFA is discussing, are also fairly expensive, especially as crawlers traverse historical states. When the crawler is poorly implemented they'll get stuck doing this basically forever. Detecting and dealing with git hosts is an absolute must for any web crawler due to this.
  - mtlynch 1y ago
    >Flat rate web hosting exists, and blogs (especially unsophisticated ones) do not require much bandwidth or processing power. I actually find this surprisingly difficult to find. I just want static hosting (like Netlify or Firebase Hosting), but there aren't many hosts that offer that. There are lots of providers where I can buy a VPS somewhere and be in charge of configuring and patching it, but if I just want to hand someone a set of HTML files and some money in exchange for hosting, not many hosts fit the bill.
    - marginalia_nu 1y ago
      If you just want to host HTML for personal use github pages is free (and works with a custom domain). There are bandwidth limitations, but they definitely won't pull an AWS on you and send a bill that would cover a new car because a crawler acted up.
      - mtlynch 1y ago
        Github Pages' "soft" bandwidth limit is 100 MB. I typically use 300-500 GB per month at Netlify, so I'm over Github's limit. I actually don't even want a free option. I want to pay a vendor that cares about keeping my website online. I'm fine paying $20-50/mo as long as it's bounded and they don't just take my site offline if I see a spike from HN.
    - diggan 1y ago
      > There are lots of providers where I can buy a VPS somewhere and be in charge of configuring and patching it, but if I just want to hand someone a set of HTML files and some money in exchange for hosting, not many hosts fit the bill. Yeah, that's true, there isn't a lot of "I give you money and HTML, you host it" services out there, surprisingly. Probably the most mature, cheapest and most reliable one today would be good ol' neocities.org (run by HN user kyledrake) which basically gives you 3TB/month for $5, pretty good deal :) Sometimes when I miss StumbleUpon I go to https://neocities.org/browse?sort_by=random https://neocities.org/browse?sort_by=random which gives a fun little glimpse of the hobby/curiosity/creative web.
    - 1y ago
bayindirh 1y ago
My view on this is simple: If you're a bot which will ignore all the licenses I put on that content, then I don't want to you to be able to reach that content. No, any amount of monetary compensation is not welcome either. I use these licenses as a matter of principle, and my principles are not for sale. That's all, thanks.
- warkdarrior 1y ago
  How can you tell a bot will ignore all your content licenses?
  - bayindirh 1y ago
    Currently all AI companies argue that the content they use falls under fair use, and disregard all licenses. This means any future ones respecting these licenses needs to be whitelisted.
    - diggan 1y ago
      How do you know that that bot is part of those AI companies? Maybe it's my personal bot you're blocking, should I also not have (indirectly) access to the content?
      - bayindirh 1y ago
        You can use a honest user string denoting that it's your bot. Some AI companies label their bots transparently, they show up on the logs I keep. While I understand that you may need a personal bot to crawl or mirror a site, I can't guarantee that I'll grant you access. I don't like to be that heavy-handed in the first place, but capitalism is making it harder to trust entities which you can't see and talk face to face.
      - simianparrot 1y ago
        No. Access to my content is a privilege I grant you. I decide how you get to access it, and via a bot that my setup confuses for an AI crawler belonging to an anti-human AI corporation is not a valid way to access it. Get off my virtual lawn.
Vegenoid 1y ago
I think it’s better viewed through a lens of effort. Implementing systems that try harder to not challenge humans takes more work than just throwing up a catch-all challenge wall. The author’s goal is admirable: “My primary principle is that I’d rather not annoy real humans more than strictly intended”. However, the primary goal for many people hosting content will be “block bots and allow humans with minimal effort and tuning”.
- deleted 1y ago
  [deleted]

jitl 1y ago

Really? If I’m an unsophisticated blog not using a CDN, and I get a $1000 bill for bandwidth overage or something, I’m gonna google a solve and slap it on there because I don’t want to pay another $1000 for Big Basilisk. I don’t think that’s emotional response, it’s common sense.

phantompeace 1y ago
Wouldn't it be easier to put the unsophisticated blog behind cloudflare
- mhuffman 1y ago
  As much as I like to shit on cloudflare at every opportunity, it would obviously be easier to put it behind CF than install bot detection plugins.
marginalia_nu 1y ago
Seems like you've made profoundly questionable hosting or design choices for that to happen. Flat rate web hosting exists, and blogs (especially unsophisticated ones) do not require much bandwidth or processing power. Misbehaving crawlers are a huge problem but bloggers are among the least affected by them. Something like a wiki or a forum is a better example, as they're in a category of websites where each page visit is almost unavoidably rendered on the fly using multiple expensive SQL queries due to the rapidly mutating nature of their datasets. Git forges, like the one TFA is discussing, are also fairly expensive, especially as crawlers traverse historical states. When the crawler is poorly implemented they'll get stuck doing this basically forever. Detecting and dealing with git hosts is an absolute must for any web crawler due to this.
- mtlynch 1y ago
  >Flat rate web hosting exists, and blogs (especially unsophisticated ones) do not require much bandwidth or processing power. I actually find this surprisingly difficult to find. I just want static hosting (like Netlify or Firebase Hosting), but there aren't many hosts that offer that. There are lots of providers where I can buy a VPS somewhere and be in charge of configuring and patching it, but if I just want to hand someone a set of HTML files and some money in exchange for hosting, not many hosts fit the bill.
  - marginalia_nu 1y ago
    If you just want to host HTML for personal use github pages is free (and works with a custom domain). There are bandwidth limitations, but they definitely won't pull an AWS on you and send a bill that would cover a new car because a crawler acted up.
    - mtlynch 1y ago
      Github Pages' "soft" bandwidth limit is 100 MB. I typically use 300-500 GB per month at Netlify, so I'm over Github's limit. I actually don't even want a free option. I want to pay a vendor that cares about keeping my website online. I'm fine paying $20-50/mo as long as it's bounded and they don't just take my site offline if I see a spike from HN.
  - diggan 1y ago
    > There are lots of providers where I can buy a VPS somewhere and be in charge of configuring and patching it, but if I just want to hand someone a set of HTML files and some money in exchange for hosting, not many hosts fit the bill. Yeah, that's true, there isn't a lot of "I give you money and HTML, you host it" services out there, surprisingly. Probably the most mature, cheapest and most reliable one today would be good ol' neocities.org (run by HN user kyledrake) which basically gives you 3TB/month for $5, pretty good deal :) Sometimes when I miss StumbleUpon I go to https://neocities.org/browse?sort_by=random https://neocities.org/browse?sort_by=random which gives a fun little glimpse of the hobby/curiosity/creative web.
  - 1y ago

bayindirh 1y ago

My view on this is simple: If you're a bot which will ignore all the licenses I put on that content, then I don't want to you to be able to reach that content. No, any amount of monetary compensation is not welcome either. I use these licenses as a matter of principle, and my principles are not for sale. That's all, thanks.

warkdarrior 1y ago
How can you tell a bot will ignore all your content licenses?
- bayindirh 1y ago
  Currently all AI companies argue that the content they use falls under fair use, and disregard all licenses. This means any future ones respecting these licenses needs to be whitelisted.
  - diggan 1y ago
    How do you know that that bot is part of those AI companies? Maybe it's my personal bot you're blocking, should I also not have (indirectly) access to the content?
    - bayindirh 1y ago
      You can use a honest user string denoting that it's your bot. Some AI companies label their bots transparently, they show up on the logs I keep. While I understand that you may need a personal bot to crawl or mirror a site, I can't guarantee that I'll grant you access. I don't like to be that heavy-handed in the first place, but capitalism is making it harder to trust entities which you can't see and talk face to face.
    - simianparrot 1y ago
      No. Access to my content is a privilege I grant you. I decide how you get to access it, and via a bot that my setup confuses for an AI crawler belonging to an anti-human AI corporation is not a valid way to access it. Get off my virtual lawn.

Vegenoid 1y ago

I think it’s better viewed through a lens of effort. Implementing systems that try harder to not challenge humans takes more work than just throwing up a catch-all challenge wall. The author’s goal is admirable: “My primary principle is that I’d rather not annoy real humans more than strictly intended”. However, the primary goal for many people hosting content will be “block bots and allow humans with minimal effort and tuning”.

deleted 1y ago
[deleted]