7 ms·

To stretch the analogy to the breaking point: If you send 10,000 personal shoppers all at once to the same store just to check prices, the store's going to be r

To stretch the analogy to the breaking point: If you send 10,000 personal shoppers all at once to the same store just to check prices, the store's going to be rightfully annoyed that they aren't making sales because legit buyers can't get in.

sublinear 1y ago
Too bad. Build a bigger store or publish this information so we don't need 10,000 personal shoppers. Was this not the whole point of having a website? Who distorted that simple idea into the garbage websites we have now?
- recursive 1y ago
  Weird take. The store doesn't owe your personal shippers anything.
  - the_real_cher 1y ago
    In the same token the personal shoppers don't owe the store anything either.
    - eddythompson80 1y ago
      Surely they owe them money for the goods and service, no? I thought that's how stores worked.
      - the_real_cher 1y ago
        Context friend. This article and entire comments sections is about questionable web page access. Context.
        eddythompson80 1y ago
        You're replying in a store metaphor thread though. Context matters.
    - recursive 1y ago
      Then they can't complain if they're barred entry.
      - the_real_cher 1y ago
        http is neutral. it's up to the client to ignore robots.txt You can block IP's at the host level but there's pretty easy ways around that with proxy networks.
        eddythompson80 1y ago
        > http is neutral. Who misled you with that statement?
        the_real_cher 1y ago
        Http doesnt have emotions or thought last time I checked.
        eddythompson80 1y ago
        It seems that a 403 makes you sad though.
        the_real_cher 1y ago
        iproyal.com makes me smile again
        eddythompson80 1y ago
        And Cloudflare makes you cry. See, it's not neutral. Glad you learned something today. The more one learns everyday, the less stupid you become.
        drdaeman 1y ago
        IETF?
  - drdaeman 1y ago
    That's fair, but if there's enough of supply and demand for this to get traction (and online shopping is bug, and autonomous agents are sort of trending), this conflict of interest paired with a no-compromise "we don't own you anything" attitude is bound to escalate in an arms race. And YMMV but I don't like where that race may possibly end. If store businesses at least partially relies on obscurity of information that can be solved through automated means (e.g. storefronts tend to push visitors towards products they don't want, and buyer agents are fighting that and looking for something buyers instructed them) just playing this cat and mouse game of blocking agents, finding workarounds, and repeating the cycle is only creating perverse technological contraptions that neither party is really interested in - but both are circumstantially forced to invest into.
- dabockster 1y ago
  > Who distorted that simple idea into the garbage websites we have now? Corporate America. Where clean code goes to die.
hombre_fatal 1y ago
Your comment and the above comment of course show different cases. An agent making a request on the explicit behalf of someone else is probably something most of us agree is reasonable. "What are the current stories on Hacker News?" -- the agent is just doing the same request to the same website that I would have done anyways. But the sort of non-explicit just-in-case crawling that Perplexity might do for a general question where it crawls 4-6 sources isn't as easy to defend. "Are polar bears always white?" -- Now it's making requests I wouldn't have necessarily made, and it could even been seen as a sort of amplification attack. That said, TFA's example is where they register secretexample.com and then ask Perplexity "what is secretexample.com about?" and Perplexity sends a request to answer the question, so that's an example of the first case, not the second.
- bayindirh 1y ago
  As a person who has a couple of sites out there, and witnesses AI crawlers coming and fetching pages from these sites, I have a question: What prevents these companies from keeping a copy of that particular page, which I specifically disallowed for bot scraping, and feed it to their next training cycle? Pinky promises? Ethics? Laws? Technical limitations? Leeroy Jenkins?
  - tempfile 1y ago
    The way to prevent people from downloading your pages and using them is to take them off the public internet. There are laws to prevent people from violating your copyright or from preventing access to your service (by excessive traffic). But there is (thankfully) no magical right that stops people from reading your content and describing it.
    - bayindirh 1y ago
      Many site operators want people to access their content, but prevent AI companies from scraping their sites for training data. People who think like that made tools like Anubis, and it works. I also want to keep this distinction on the sites I own. I also use licenses to signal that this site is not good to use for AI training, because it's CC BY-NC-SA-2.0. So, I license my content appropriately (No derivative, Non-commercial, shareable with the same license with attribution), add technical countermeasures on top, because companies doesn't respect these licenses (because monies), and circumvent these mechanisms (because monies), and I'm the one to suck this up and shut-up (because their monies)? Makes no sense whatsoever.
cellis 1y ago
Easy "By Appointment only" or "rate limited to authenticated users" done.
p3rls 1y ago
That is not the breaking point at all of the analogy-- that literally happens to my custom CMS/wiki/image host I built for my niche, kpopping.com. We are constantly attacked by crawlers. Meanwhile google rewards wordpress slop that buys backlinks with #1 pageranks for years. Welcome to the internet.

sublinear 1y ago

Too bad. Build a bigger store or publish this information so we don't need 10,000 personal shoppers. Was this not the whole point of having a website? Who distorted that simple idea into the garbage websites we have now?

recursive 1y ago
Weird take. The store doesn't owe your personal shippers anything.
- the_real_cher 1y ago
  In the same token the personal shoppers don't owe the store anything either.
  - eddythompson80 1y ago
    Surely they owe them money for the goods and service, no? I thought that's how stores worked.
    - the_real_cher 1y ago
      Context friend. This article and entire comments sections is about questionable web page access. Context.
      - eddythompson80 1y ago
        You're replying in a store metaphor thread though. Context matters.
  - recursive 1y ago
    Then they can't complain if they're barred entry.
    - the_real_cher 1y ago
      http is neutral. it's up to the client to ignore robots.txt You can block IP's at the host level but there's pretty easy ways around that with proxy networks.
      - eddythompson80 1y ago
        > http is neutral. Who misled you with that statement?
        the_real_cher 1y ago
        Http doesnt have emotions or thought last time I checked.
        eddythompson80 1y ago
        It seems that a 403 makes you sad though.
        the_real_cher 1y ago
        iproyal.com makes me smile again
        eddythompson80 1y ago
        And Cloudflare makes you cry. See, it's not neutral. Glad you learned something today. The more one learns everyday, the less stupid you become.
        drdaeman 1y ago
        IETF?
- drdaeman 1y ago
  That's fair, but if there's enough of supply and demand for this to get traction (and online shopping is bug, and autonomous agents are sort of trending), this conflict of interest paired with a no-compromise "we don't own you anything" attitude is bound to escalate in an arms race. And YMMV but I don't like where that race may possibly end. If store businesses at least partially relies on obscurity of information that can be solved through automated means (e.g. storefronts tend to push visitors towards products they don't want, and buyer agents are fighting that and looking for something buyers instructed them) just playing this cat and mouse game of blocking agents, finding workarounds, and repeating the cycle is only creating perverse technological contraptions that neither party is really interested in - but both are circumstantially forced to invest into.
dabockster 1y ago
> Who distorted that simple idea into the garbage websites we have now? Corporate America. Where clean code goes to die.

hombre_fatal 1y ago

Your comment and the above comment of course show different cases. An agent making a request on the explicit behalf of someone else is probably something most of us agree is reasonable. "What are the current stories on Hacker News?" -- the agent is just doing the same request to the same website that I would have done anyways. But the sort of non-explicit just-in-case crawling that Perplexity might do for a general question where it crawls 4-6 sources isn't as easy to defend. "Are polar bears always white?" -- Now it's making requests I wouldn't have necessarily made, and it could even been seen as a sort of amplification attack. That said, TFA's example is where they register secretexample.com and then ask Perplexity "what is secretexample.com about?" and Perplexity sends a request to answer the question, so that's an example of the first case, not the second.

bayindirh 1y ago
As a person who has a couple of sites out there, and witnesses AI crawlers coming and fetching pages from these sites, I have a question: What prevents these companies from keeping a copy of that particular page, which I specifically disallowed for bot scraping, and feed it to their next training cycle? Pinky promises? Ethics? Laws? Technical limitations? Leeroy Jenkins?
- tempfile 1y ago
  The way to prevent people from downloading your pages and using them is to take them off the public internet. There are laws to prevent people from violating your copyright or from preventing access to your service (by excessive traffic). But there is (thankfully) no magical right that stops people from reading your content and describing it.
  - bayindirh 1y ago
    Many site operators want people to access their content, but prevent AI companies from scraping their sites for training data. People who think like that made tools like Anubis, and it works. I also want to keep this distinction on the sites I own. I also use licenses to signal that this site is not good to use for AI training, because it's CC BY-NC-SA-2.0. So, I license my content appropriately (No derivative, Non-commercial, shareable with the same license with attribution), add technical countermeasures on top, because companies doesn't respect these licenses (because monies), and circumvent these mechanisms (because monies), and I'm the one to suck this up and shut-up (because their monies)? Makes no sense whatsoever.

cellis 1y ago

Easy "By Appointment only" or "rate limited to authenticated users" done.

p3rls 1y ago

That is not the breaking point at all of the analogy-- that literally happens to my custom CMS/wiki/image host I built for my niche, kpopping.com. We are constantly attacked by crawlers. Meanwhile google rewards wordpress slop that buys backlinks with #1 pageranks for years. Welcome to the internet.