4 ms·

I'm curious whether we'll hit a point where some sites will want to be as friendly to bots as possible (hopefully they'll just shift to having sane, rate-limite

by xp84 2mo ago

I'm curious whether we'll hit a point where some sites will want to be as friendly to bots as possible (hopefully they'll just shift to having sane, rate-limited public APIs so everyone can save time and bandwidth), while the rest want the reverse. If you're providing a service, and AI agents truly take off to where people say "Claude, go buy me a new shower hose about 5' long, brushed nickel. Use my usual evaluation criteria and don't spend over $20." then a lot of ecommerce stores are very very incentivized to be bot friendly. And of course every "content" site is the reverse, wanting to not waste their bandwidth and hoping to score exclusive paid content deals with any entity that might want to scrape them.

ssl-3 2mo ago
I've already seen a small amount of transition in that direction. For instance: Adafruit, last year: They blocked bots very heavy-handedly, including those that were there to gather information at the behest of a present-at-the-keyboard human. This made their products nearly invisible for those who were using ChatGPT instead of Google to find particular things to buy. Even when they were discoverable through forum postings, this added friction meant that I was unable to make a swift comparison betwixt their wares and those of others, and I wasn't buying anything from them because they were making deliberate choices that I felt were hostile to my process. But that's changed. Adafruit, at least a week or so ago: The bot was able to get in and find details about the stuff they sell. This lack of friction meant that I was able to make a swift comparison between their wares and those of others. I still bought the particular sensor-chip-on-a-board widget that I wanted elsewhere, but at least Adafruit's offerings were in the running this time.
- ptorrone 2mo ago
  ssl-3 - phil from adafruit, pt@adafruit.com if email is easier, ladyada and i will both read it. the blocking of automated attacks or purchase attempts wasn't aimed at you. when a scarce part is in stock here cheaper than everywhere else, scripts that some human decided to go after us with, show up and try to clear the store before a human person can click buy when it's in stock, first fix is a blunt instrument, then filing it down. your bot getting through is the filing working, good to know, most people only post when it's broken. most people are only complaining, most people. but you, us, and others, are not most people - we work here. you bought the widget elsewhere, fair, that's what comparison shopping is for, is there is a personal axe to grind, could probably help with that too. we'd still like to know where the price, stock, or a spec buried three clicks deep. we can fix just about anything, you can help us. if you've got a project that keeps stalling on tooling instead of the idea, send that too and ladyada and i will look at it as a problem to solve, and share the results, like we always do. if the answer is nothing adafruit makes, that's a useful answer. and next time the site snags your setup, tell us exactly where. today is circuitpython day too, it's taken almost 10 years for being able to do anything i want in less than 5 minutes to get started, and then i can spend as much time on the best parts.
  - ssl-3 2mo ago
    Hi Phil. Well-met. I remember the days of the Great Pi Shortage very well. You guys were trying very hard to sell Raspberry Pis at sensible prices to real people instead of to computer programs, which was/is a commendable effort. The part I was trying to find last year was a USB canbus interface to do some automotive hacking with. That was a frustrating experience because trying to find such a widget with Google retrieves endless pages of impossible-to-exclude OBD-II interfaces and "CANBUS" replacement automotive LED lamps -- stuff that obviously wasn't applicable to my hacking project at all. So I used the bot (ChatGPT) to help sort the good stuff that I might want to buy from the irrelevant stuff that I did not want to buy. It got stalled hard when trying to access adafruit.com, which did not help my process. But, like I said: It's working now. Last week, when I set forth on a small project to characterize daylight for home automation, the same bot worked well with finding your pages on the AS7341 boards you list. No issues. It all flowed very well and Adafruit was listed prominently in the bot's output as a potential place to buy a board from. :) Anyway, for full disclosure: I see that this board is out of stock right now and I don't remember if that was the case last week or not. I was nearly broke at that time (insert "in this economy" joke here), so I wound up looking on Aliexpress so that maybe I could move the project forward and also buy a cheeseburger. A seller there had a little board which is supposed to have an AS7341 and appears to have the right interface glue brought out to sensible pins, for like $2.50 -- delivered to my door in the US. You guys can't compete with that. There's no way. Someone lost money on that transaction. But! The bot wasn't able to find this on Aliexpress. It got stuck, and didn't recommend that as a potential source at all. So score one for the team: Your stuff is showing up in the bot's results, while your (ruthless) competition is not! That exposes more people to the things that you guys work so hard to accomplish, and more exposure tends to lead to more sales, which tends to help keep the lights on. :) It's good stuff. Right now, you're ahead of the game. And thanks. I will shoot you an email directly if I find an aspect that's broken or buried in the future.
hackernud3s 2mo ago
Yeah, as soon as site operators implement that paid traffic scheme the Cloudflare has been pushing. It's all about being to monetize their compute/content.
rixed 2mo ago
Very few "user facing" applications will have to deal with real users any longer. From an author's point of view let's admit that it will make things easier.
mzajc 2mo ago
My cgit instance is incredibly friendly to bots; I go out of my way to support git's fast "smart" HTTP protocol, and the clone URL is linked on every project's root page. I even included the full clone command on 429 error pages. The end result was that I got millions of requests daily trying to download every HTML page (every variation of action, commit hash, branch, whatever else cgit allows) and basically no traffic to the proper clone route. Turns out the bots just don't care about your bot-friendly access method and will instead throw more illicitly-proxied residential IPs at the problem.
mxmilkiib 2mo ago
the best economic use-case for the semantic web (text/turtle, text/n3); businesses providing manifests of their stocks and services plus, AI would also skip search result ads, and, if those have to legally be displayed as such, then there goes the market for them, I'd think
- xp84 2mo ago
  Ngl - Can’t wait for advertising to just fully die if you’re right. If I need to pay a few bucks here and there for content, or email, or whatever that’s FINE. Just end the ad insanity and its associated pathologies like the “engagement optimizer” algos that are eating out fucking society!

by xp84 2mo ago

ssl-3 2mo ago

I've already seen a small amount of transition in that direction. For instance: Adafruit, last year: They blocked bots very heavy-handedly, including those that were there to gather information at the behest of a present-at-the-keyboard human. This made their products nearly invisible for those who were using ChatGPT instead of Google to find particular things to buy. Even when they were discoverable through forum postings, this added friction meant that I was unable to make a swift comparison betwixt their wares and those of others, and I wasn't buying anything from them because they were making deliberate choices that I felt were hostile to my process. But that's changed. Adafruit, at least a week or so ago: The bot was able to get in and find details about the stuff they sell. This lack of friction meant that I was able to make a swift comparison between their wares and those of others. I still bought the particular sensor-chip-on-a-board widget that I wanted elsewhere, but at least Adafruit's offerings were in the running this time.

ptorrone 2mo ago
ssl-3 - phil from adafruit, pt@adafruit.com if email is easier, ladyada and i will both read it. the blocking of automated attacks or purchase attempts wasn't aimed at you. when a scarce part is in stock here cheaper than everywhere else, scripts that some human decided to go after us with, show up and try to clear the store before a human person can click buy when it's in stock, first fix is a blunt instrument, then filing it down. your bot getting through is the filing working, good to know, most people only post when it's broken. most people are only complaining, most people. but you, us, and others, are not most people - we work here. you bought the widget elsewhere, fair, that's what comparison shopping is for, is there is a personal axe to grind, could probably help with that too. we'd still like to know where the price, stock, or a spec buried three clicks deep. we can fix just about anything, you can help us. if you've got a project that keeps stalling on tooling instead of the idea, send that too and ladyada and i will look at it as a problem to solve, and share the results, like we always do. if the answer is nothing adafruit makes, that's a useful answer. and next time the site snags your setup, tell us exactly where. today is circuitpython day too, it's taken almost 10 years for being able to do anything i want in less than 5 minutes to get started, and then i can spend as much time on the best parts.
- ssl-3 2mo ago
  Hi Phil. Well-met. I remember the days of the Great Pi Shortage very well. You guys were trying very hard to sell Raspberry Pis at sensible prices to real people instead of to computer programs, which was/is a commendable effort. The part I was trying to find last year was a USB canbus interface to do some automotive hacking with. That was a frustrating experience because trying to find such a widget with Google retrieves endless pages of impossible-to-exclude OBD-II interfaces and "CANBUS" replacement automotive LED lamps -- stuff that obviously wasn't applicable to my hacking project at all. So I used the bot (ChatGPT) to help sort the good stuff that I might want to buy from the irrelevant stuff that I did not want to buy. It got stalled hard when trying to access adafruit.com, which did not help my process. But, like I said: It's working now. Last week, when I set forth on a small project to characterize daylight for home automation, the same bot worked well with finding your pages on the AS7341 boards you list. No issues. It all flowed very well and Adafruit was listed prominently in the bot's output as a potential place to buy a board from. :) Anyway, for full disclosure: I see that this board is out of stock right now and I don't remember if that was the case last week or not. I was nearly broke at that time (insert "in this economy" joke here), so I wound up looking on Aliexpress so that maybe I could move the project forward and also buy a cheeseburger. A seller there had a little board which is supposed to have an AS7341 and appears to have the right interface glue brought out to sensible pins, for like $2.50 -- delivered to my door in the US. You guys can't compete with that. There's no way. Someone lost money on that transaction. But! The bot wasn't able to find this on Aliexpress. It got stuck, and didn't recommend that as a potential source at all. So score one for the team: Your stuff is showing up in the bot's results, while your (ruthless) competition is not! That exposes more people to the things that you guys work so hard to accomplish, and more exposure tends to lead to more sales, which tends to help keep the lights on. :) It's good stuff. Right now, you're ahead of the game. And thanks. I will shoot you an email directly if I find an aspect that's broken or buried in the future.

hackernud3s 2mo ago

Yeah, as soon as site operators implement that paid traffic scheme the Cloudflare has been pushing. It's all about being to monetize their compute/content.

rixed 2mo ago

Very few "user facing" applications will have to deal with real users any longer. From an author's point of view let's admit that it will make things easier.

mzajc 2mo ago

My cgit instance is incredibly friendly to bots; I go out of my way to support git's fast "smart" HTTP protocol, and the clone URL is linked on every project's root page. I even included the full clone command on 429 error pages. The end result was that I got millions of requests daily trying to download every HTML page (every variation of action, commit hash, branch, whatever else cgit allows) and basically no traffic to the proper clone route. Turns out the bots just don't care about your bot-friendly access method and will instead throw more illicitly-proxied residential IPs at the problem.

mxmilkiib 2mo ago

the best economic use-case for the semantic web (text/turtle, text/n3); businesses providing manifests of their stocks and services plus, AI would also skip search result ads, and, if those have to legally be displayed as such, then there goes the market for them, I'd think

xp84 2mo ago
Ngl - Can’t wait for advertising to just fully die if you’re right. If I need to pay a few bucks here and there for content, or email, or whatever that’s FINE. Just end the ad insanity and its associated pathologies like the “engagement optimizer” algos that are eating out fucking society!