4 ms·

I don't disagree, but there is a sliding scale here. For instance, I wanted to buy a piece of equipment the other day from a local company for a specific useca

I don't disagree, but there is a sliding scale here. For instance, I wanted to buy a piece of equipment the other day from a local company for a specific usecase. I wanted to find a specific price/weight/specs ratio, and asked an llm to loop through the 20 or so items, fetch their page and calculate and present some values for each. This then led to me going and buying the one I found. So the llm was mainly just an extension of me clicking into every page and making a spreadsheet myself. However, if it were to continuously poll, or just scrape or something with no intention of buying, I would be no better than a bot.

hluska 2mo ago
I don’t see the sliding scale - if the website didn’t want a bot they could have blocked you. They clearly don’t mind so what’s the issue? You still used a bot.
- deleted 2mo ago
  [deleted]
- baby_souffle 2mo ago
  > I don’t see the sliding scale - if the website didn’t want a bot they could have blocked you. The point is that not all bots are bad. Assuming as much / implementing policies to that effect won't block all bots but it _will_ block the portion of bots that represent users considering giving you money. I have a series of bots that monitor various eCom sites to monitor prices over time for big-ticket items I am considering as well as staples/groceries and everything in between. I have this little scrape/ingest pipeline because there's no other way to obtain this data... not even an API that I can pay for access to. One of the large appliance sellers that I have in the scrape queue has gotten _hyper_ aggressive with bot detection to the point where even my personal head-full chrome instance doesn't always get to load the page. Guess who I will never buy that ~$2000 appliance from.
  - compiler-guy 2mo ago
    The tradeoff here is a classic false-positive vs false-negative issue. If the cost of the bad-bot false positive (which blocks your bot out) is lower than the cost of the bad-bot false negatives (which allow bad bots in), then it is still a good tradeoff, if a suboptimal situation.
  - jbs789 2mo ago
    The fundamental issue here is the way we use the internet is changing… and what we are seeing is some friction as that works its way through. I have a business where I clearly don’t want any bots. A retailer, maybe they do.
  - volkl48 2mo ago
    If this is the case, they probably outright do not want you as a customer. Significant sales (beyond for excess stock clearance/discontinued items) exist largely to generate some excitement from potential customers and to lure them in to taking a look at the site/store and to hopefully get them to buy some extra stuff that's less discounted while they're there. The actual on-sale item is potentially losing money for the business or at break-even. ------ A person who's running a series of bots like this to track the exact lowest price on the internet for items over time and to always only buy at the lowest price it's ever been sold for....is probably the least likely kind of customer to buy anything extra that's actually profitable for the business.
    - baby_souffle 2mo ago
      > If this is the case, they probably outright do not want you as a customer. Almost certainly! No eComm site wants the cautious shopper that's done their comparisons and knows price trends; they want the impulsive people with a line of credit. I doubt that my local grocery store is happy that I index all of my receipts so I know what I buy, how often and when the prices change so I know when a discount on something I buy every once in a while is actually a discount. When I built this a few years ago, I had to write a lot of code to make all this work. Now that's a simple "hey $agent, check the price of https://some-site.com/appliances/that-new-fridge.html https://some-site.com/appliances/that-new-fridge.html and https://other-site.com/shop/appliances/that-new-fridge.aspx https://other-site.com/shop/appliances/that-new-fridge.aspx daily and notify me when it's on sale or has a promotion" I would expect that more and more people will discover that their LLM of choice can do this, that their LLM of choice can help them comparison shop and do analysis of when - historically - is the best time to buy that new TV or whatever (yes, pricing on this stuff is seasonal!). Maybe _most_ people will never do this, maybe I'm wrong. But if i'm not, blocking all bots "because bots are bad, mmmkay" is how you loose a sale. Ignoring the specific price data example, more and more and more people _are_ browsing the web through an LLM. Assuming that continues, blocking bots is going to have the same effect on your traffic as doing everything in the "what not to do" section of the google SEO guide book.
- deleted 2mo ago
  [deleted]
- psd1 2mo ago
  That sounds like "if they didn't want to get stabbed, they would have disarmed the attacker" But this level of competence is not a given. I wrote my first html when i was 9. Nowadays, you need to know about <squints at notes> buns and crosses and tailwinds and what-have-you. Dealing with scrapers? most web publishers can't even deal with javascript.
weego 2mo ago
No, there isn't, it's a bot. Justifying it with a strawman case study doesn't change it. This is giving "why is this plane full? What in doing is important but what is everyone else doing here?"
- matsemann 2mo ago
  No, the first case I used their website in a way that gave them a sale, which is why they have it. Why is it a straw man case when it's reality? Don't argue in bad faith, please. Express your reasoning, like I did, don't just dismiss me because you disagree..
  - vector_spaces 2mo ago
    You are assuming that because you are giving the operator money that you are entitled to use their website however you please -- that's not how it works. If you enter a brick & mortar establishment and break their rules, even as a paying customer, you might risk e.g. being kicked out and banned, depending on the behavior. This is not unique to e-commerce To be fair, I do not generally support wholesale banning of scrapers and non-browser clients, since this tends to have lots of ugly side effects. But regardless, I'm interpreting your last comment as saying that you should be entitled to behave as you please because you are a paying customer. If that's the case, then you are in for a rude awakening one day
    - AussieWog93 2mo ago
      I think the implicit assumption is that if you explained to the owner of the website what the scraper was doing, they would be more than happy to allow it. The bot was also being generally courteous, only scraping the 2-3 pages needed to get the information for the human who spun up the LLM. I don't think GP was at all suggesting he should be able to act however he wants, just because he's a paying customer. I run an ecommerce site and I'm actually more worried about bots being rejected than accepted because of stuff like GP. We get a decent amount of business through ChatGPT.
    - drknownuffin 2mo ago

hluska 2mo ago

I don’t see the sliding scale - if the website didn’t want a bot they could have blocked you. They clearly don’t mind so what’s the issue? You still used a bot.

deleted 2mo ago
[deleted]
baby_souffle 2mo ago
> I don’t see the sliding scale - if the website didn’t want a bot they could have blocked you. The point is that not all bots are bad. Assuming as much / implementing policies to that effect won't block all bots but it _will_ block the portion of bots that represent users considering giving you money. I have a series of bots that monitor various eCom sites to monitor prices over time for big-ticket items I am considering as well as staples/groceries and everything in between. I have this little scrape/ingest pipeline because there's no other way to obtain this data... not even an API that I can pay for access to. One of the large appliance sellers that I have in the scrape queue has gotten _hyper_ aggressive with bot detection to the point where even my personal head-full chrome instance doesn't always get to load the page. Guess who I will never buy that ~$2000 appliance from.
- compiler-guy 2mo ago
  The tradeoff here is a classic false-positive vs false-negative issue. If the cost of the bad-bot false positive (which blocks your bot out) is lower than the cost of the bad-bot false negatives (which allow bad bots in), then it is still a good tradeoff, if a suboptimal situation.
- jbs789 2mo ago
  The fundamental issue here is the way we use the internet is changing… and what we are seeing is some friction as that works its way through. I have a business where I clearly don’t want any bots. A retailer, maybe they do.
- volkl48 2mo ago
  If this is the case, they probably outright do not want you as a customer. Significant sales (beyond for excess stock clearance/discontinued items) exist largely to generate some excitement from potential customers and to lure them in to taking a look at the site/store and to hopefully get them to buy some extra stuff that's less discounted while they're there. The actual on-sale item is potentially losing money for the business or at break-even. ------ A person who's running a series of bots like this to track the exact lowest price on the internet for items over time and to always only buy at the lowest price it's ever been sold for....is probably the least likely kind of customer to buy anything extra that's actually profitable for the business.
  - baby_souffle 2mo ago
    > If this is the case, they probably outright do not want you as a customer. Almost certainly! No eComm site wants the cautious shopper that's done their comparisons and knows price trends; they want the impulsive people with a line of credit. I doubt that my local grocery store is happy that I index all of my receipts so I know what I buy, how often and when the prices change so I know when a discount on something I buy every once in a while is actually a discount. When I built this a few years ago, I had to write a lot of code to make all this work. Now that's a simple "hey $agent, check the price of https://some-site.com/appliances/that-new-fridge.html https://some-site.com/appliances/that-new-fridge.html and https://other-site.com/shop/appliances/that-new-fridge.aspx https://other-site.com/shop/appliances/that-new-fridge.aspx daily and notify me when it's on sale or has a promotion" I would expect that more and more people will discover that their LLM of choice can do this, that their LLM of choice can help them comparison shop and do analysis of when - historically - is the best time to buy that new TV or whatever (yes, pricing on this stuff is seasonal!). Maybe _most_ people will never do this, maybe I'm wrong. But if i'm not, blocking all bots "because bots are bad, mmmkay" is how you loose a sale. Ignoring the specific price data example, more and more and more people _are_ browsing the web through an LLM. Assuming that continues, blocking bots is going to have the same effect on your traffic as doing everything in the "what not to do" section of the google SEO guide book.
deleted 2mo ago
[deleted]
psd1 2mo ago
That sounds like "if they didn't want to get stabbed, they would have disarmed the attacker" But this level of competence is not a given. I wrote my first html when i was 9. Nowadays, you need to know about <squints at notes> buns and crosses and tailwinds and what-have-you. Dealing with scrapers? most web publishers can't even deal with javascript.

weego 2mo ago

No, there isn't, it's a bot. Justifying it with a strawman case study doesn't change it. This is giving "why is this plane full? What in doing is important but what is everyone else doing here?"

matsemann 2mo ago
No, the first case I used their website in a way that gave them a sale, which is why they have it. Why is it a straw man case when it's reality? Don't argue in bad faith, please. Express your reasoning, like I did, don't just dismiss me because you disagree..
- vector_spaces 2mo ago
  You are assuming that because you are giving the operator money that you are entitled to use their website however you please -- that's not how it works. If you enter a brick & mortar establishment and break their rules, even as a paying customer, you might risk e.g. being kicked out and banned, depending on the behavior. This is not unique to e-commerce To be fair, I do not generally support wholesale banning of scrapers and non-browser clients, since this tends to have lots of ugly side effects. But regardless, I'm interpreting your last comment as saying that you should be entitled to behave as you please because you are a paying customer. If that's the case, then you are in for a rude awakening one day
  - AussieWog93 2mo ago
    I think the implicit assumption is that if you explained to the owner of the website what the scraper was doing, they would be more than happy to allow it. The bot was also being generally courteous, only scraping the 2-3 pages needed to get the information for the human who spun up the LLM. I don't think GP was at all suggesting he should be able to act however he wants, just because he's a paying customer. I run an ecommerce site and I'm actually more worried about bots being rejected than accepted because of stuff like GP. We get a decent amount of business through ChatGPT.
  - drknownuffin 2mo ago