3 ms·

What we need is some legal teeth behind robots.txt. It won't stop everyone, but Big Corp would be a tasty target for lawsuits.

notatoad 1y ago
It wouldn’t stop anyone. The bots you want to block already operate out of places where those laws wouldn’t be enforced.
- qbane 1y ago
  Then that is a good reason to deny the requests from those IPs
  - literalAardvark 1y ago
    I've run a few hundred small domains for various online stores with an older backend that didn't scale very well for crawlers and at some point we started blocking by continent. It's getting really, really ugly out there.
  - deleted 1y ago
    [deleted]
- account42 1y ago
  If that was the case then I am I getting buttflare-blocked here in the EU.
stronglikedan 1y ago
It should have the same protections as an EULA, where the crawler is the end user, and crawlers should be required to read it and apply it.
- immibis 1y ago
  So none at all? EULAs are mostly just meant to intimidate you so you won't exercise your inalienable rights.
  - majorchord 1y ago
    I find that extremely hard to believe. Do you have a source?
quectophoton 1y ago
I don't know about this. This means I'd get sued for using a feed reader on Codeberg[1], or for mirroring repositories from there (e.g. with Forgejo), since both are automated actions that are not caused directly by a user interaction (i.e. bots, rather than user agents). [1]: https://codeberg.org/robots.txt#:~:text=Disallow:%20/.git/,.rss https://codeberg.org/robots.txt#:~:text=Disallow:%20/.git/,....
- blibble 1y ago
  > This means I'd get sued for using a feed reader on Codeberg you think codeberg would sue you?
  - quectophoton 1y ago
    Probably not. But it's the same thing with random software from a random nobody that has no license, or has a license that's not open-source: If I use those libraries or programs, do I think they would sue me? Probably not.
- gausswho 1y ago
  To be more specific, if we assume good faith upon our fine congresspeople to craft this well... ok yeah, well for hypothetical case I'll continue... What legal teeth I would advocate would be targeted to crawlers (a subset of bot) and not include your usage. It would mandate that Big Corp crawlers (for search indexing, AI data harvesting, etc.) be registered and identify themselves in their requests. This would allow serverside tools to efficiently reject them. Failure to comply would result in fines large enough to change behavior. Now that I write that out, if such a thing were to come to pass, and it was well received, I do worry that congress would foam at the mouth to expand it to bots more generally, Microsoft-Uncertified-Devices, etc.
  - quectophoton 1y ago
    Yeah, my main worry here is how we define the unwanted traffic, and how that definition could be twisted by bigcorp lawyers. If it's too loose and similar to "wanted traffic is how the authors intend the website to be accessed, unwanted traffic is anything else", that's an argument that can be used against adblocks, or in favor of very specific devices like you mention. Might even give slightly more teeth to currently-unenforceable TOS. If it's too strict, it's probably easier to find loopholes and technicalities that just lets them say "technically it doesn't match the definition of unwanted traffic". Even if it's something balanced, I bet bigcorp lawyers will find a way to twist the definitions in their favor and set a precedent that's convenient for them. I know this is a mini-rant rather than a helpful comment that tries to come up with a solution, it's just that I'm pessimistic because it seems the internet becomes a bit worse day by day no matter what we try to do :c
Galanwe 1y ago
- Moral rules are never really effective - Legal threats are never really effective Effective solutions are: - Technical - Monetary I like the idea of web as a blockchain of content. If you want to pull some data, you have to pay for it with some kind of token. You either buy that token to consume information if you're of the leecher type, or get some by doing contributions that gain back tokens. It's more or less the same concept as torrents back in the day. This should be applied to emails too. The regular person send what, 20 emails per day max ? Say it costs $0.01 per mail, anyone could pay that. But if you want to spam 1,000,000 everyday that becomes prohibitive.
- edm0nd 1y ago
  >This should be applied to emails too. The regular person send what, 20 emails per day max ? Say it costs $0.01 per mail, anyone could pay that. This seems flawed. Poor people living in 3rd world countries that make like $2.00/day wouldn't be able to afford this. >But if you want to spam 1,000,000 everyday that becomes prohibitive. Companies and people with $ can easily pay this with no issues. If it costs $10,000 to send 1M emails that inbox but you profit $50k, its a non issue.
qwerty456127 1y ago
What we need is stop fighting robots and start welcoming and helping them. I se zero reasons to oppose robots visiting any website I would build. The only purpose I ever tried disallowed robots for was preventing search engines from indexing incomplete versions or going the paths which really make no sense for them to go. Now I think we should write separate instructions for different kinds of robots: a search engine indexer shouldn't open pages which have serious side-effects (e.g. place an order) or display semi-realtime technical details but an LLM agent may be on a legitimate mission involving this.
- vkou 1y ago
  > What we need is stop fighting robots and start welcoming and helping them. I se zero reasons to oppose robots visiting any website I would build. Well, I'm glad you speak for the entire Internet. Pack it in folks, we've solved the problem. Tomorrow, I'll give us the solution to wealth inequality (just stop fighting efforts to redistribute wealth and political power away from billionaires hoarding it), and next week, we'll finally get to resolve the old question of software patents.
- Symbiote 1y ago
  > I see zero reasons to oppose robots visiting any website I would build. > preventing search engines from indexing incomplete versions or going the paths which really make no sense for them to go. What will you do when the bots ignore your instructions, and send a million requests a day to these URLs from half a million different IP addresses?
  - ianbutler 1y ago
    Let my site go down and then restart my server a few hours later. I'm a dude with a blog I'm not making uptime guarantees. I think you're overestimating the harm and how often this happens. Misbehaving scrapers have been a problem for years not just from AI. I've written posts on how to properly handle scraping and the legal grey area it puts you in and how to be a responsible one. If companies don't want to be responsible the solution isn't abdicate an open web. It's make better law and enforcement of said law.
edm0nd 1y ago
No we dont
ctoth 1y ago
The funny thing about the good old WWW is the first two W's stand for world-wide. So Which legal teeth?
- account42 1y ago
  Try hosting some illegal-in-the-US content and find out.
jopsen 1y ago
I have the feeling that it's the small players that cause problems. Dumb bots that don't respect robot.txt or nofollow are the ones trying all combinations of the filters available in your search options and requesting all pages for each such combination. The number of search pages can easily be exponential in the number of filters you offer. Bots walking around in these traps, do it because they are dumb. But even a small degenerate bot can send more requests than 1M MAUs. At least that's my impression of the problem we're sometimes facing. Signed agents seems like a horrific solution. And many serving the traffic is just better.

notatoad 1y ago

It wouldn’t stop anyone. The bots you want to block already operate out of places where those laws wouldn’t be enforced.

qbane 1y ago
Then that is a good reason to deny the requests from those IPs
- literalAardvark 1y ago
  I've run a few hundred small domains for various online stores with an older backend that didn't scale very well for crawlers and at some point we started blocking by continent. It's getting really, really ugly out there.
- deleted 1y ago
  [deleted]
account42 1y ago
If that was the case then I am I getting buttflare-blocked here in the EU.

stronglikedan 1y ago

It should have the same protections as an EULA, where the crawler is the end user, and crawlers should be required to read it and apply it.

immibis 1y ago
So none at all? EULAs are mostly just meant to intimidate you so you won't exercise your inalienable rights.
- majorchord 1y ago
  I find that extremely hard to believe. Do you have a source?

quectophoton 1y ago

I don't know about this. This means I'd get sued for using a feed reader on Codeberg[1], or for mirroring repositories from there (e.g. with Forgejo), since both are automated actions that are not caused directly by a user interaction (i.e. bots, rather than user agents). [1]: https://codeberg.org/robots.txt#:~:text=Disallow:%20/.git/,.rss https://codeberg.org/robots.txt#:~:text=Disallow:%20/.git/,....

blibble 1y ago
> This means I'd get sued for using a feed reader on Codeberg you think codeberg would sue you?
- quectophoton 1y ago
  Probably not. But it's the same thing with random software from a random nobody that has no license, or has a license that's not open-source: If I use those libraries or programs, do I think they would sue me? Probably not.
gausswho 1y ago
To be more specific, if we assume good faith upon our fine congresspeople to craft this well... ok yeah, well for hypothetical case I'll continue... What legal teeth I would advocate would be targeted to crawlers (a subset of bot) and not include your usage. It would mandate that Big Corp crawlers (for search indexing, AI data harvesting, etc.) be registered and identify themselves in their requests. This would allow serverside tools to efficiently reject them. Failure to comply would result in fines large enough to change behavior. Now that I write that out, if such a thing were to come to pass, and it was well received, I do worry that congress would foam at the mouth to expand it to bots more generally, Microsoft-Uncertified-Devices, etc.
- quectophoton 1y ago
  Yeah, my main worry here is how we define the unwanted traffic, and how that definition could be twisted by bigcorp lawyers. If it's too loose and similar to "wanted traffic is how the authors intend the website to be accessed, unwanted traffic is anything else", that's an argument that can be used against adblocks, or in favor of very specific devices like you mention. Might even give slightly more teeth to currently-unenforceable TOS. If it's too strict, it's probably easier to find loopholes and technicalities that just lets them say "technically it doesn't match the definition of unwanted traffic". Even if it's something balanced, I bet bigcorp lawyers will find a way to twist the definitions in their favor and set a precedent that's convenient for them. I know this is a mini-rant rather than a helpful comment that tries to come up with a solution, it's just that I'm pessimistic because it seems the internet becomes a bit worse day by day no matter what we try to do :c

Galanwe 1y ago

- Moral rules are never really effective - Legal threats are never really effective Effective solutions are: - Technical - Monetary I like the idea of web as a blockchain of content. If you want to pull some data, you have to pay for it with some kind of token. You either buy that token to consume information if you're of the leecher type, or get some by doing contributions that gain back tokens. It's more or less the same concept as torrents back in the day. This should be applied to emails too. The regular person send what, 20 emails per day max ? Say it costs $0.01 per mail, anyone could pay that. But if you want to spam 1,000,000 everyday that becomes prohibitive.

edm0nd 1y ago
>This should be applied to emails too. The regular person send what, 20 emails per day max ? Say it costs $0.01 per mail, anyone could pay that. This seems flawed. Poor people living in 3rd world countries that make like $2.00/day wouldn't be able to afford this. >But if you want to spam 1,000,000 everyday that becomes prohibitive. Companies and people with $ can easily pay this with no issues. If it costs $10,000 to send 1M emails that inbox but you profit $50k, its a non issue.

qwerty456127 1y ago

What we need is stop fighting robots and start welcoming and helping them. I se zero reasons to oppose robots visiting any website I would build. The only purpose I ever tried disallowed robots for was preventing search engines from indexing incomplete versions or going the paths which really make no sense for them to go. Now I think we should write separate instructions for different kinds of robots: a search engine indexer shouldn't open pages which have serious side-effects (e.g. place an order) or display semi-realtime technical details but an LLM agent may be on a legitimate mission involving this.

vkou 1y ago
> What we need is stop fighting robots and start welcoming and helping them. I se zero reasons to oppose robots visiting any website I would build. Well, I'm glad you speak for the entire Internet. Pack it in folks, we've solved the problem. Tomorrow, I'll give us the solution to wealth inequality (just stop fighting efforts to redistribute wealth and political power away from billionaires hoarding it), and next week, we'll finally get to resolve the old question of software patents.
Symbiote 1y ago
> I see zero reasons to oppose robots visiting any website I would build. > preventing search engines from indexing incomplete versions or going the paths which really make no sense for them to go. What will you do when the bots ignore your instructions, and send a million requests a day to these URLs from half a million different IP addresses?
- ianbutler 1y ago
  Let my site go down and then restart my server a few hours later. I'm a dude with a blog I'm not making uptime guarantees. I think you're overestimating the harm and how often this happens. Misbehaving scrapers have been a problem for years not just from AI. I've written posts on how to properly handle scraping and the legal grey area it puts you in and how to be a responsible one. If companies don't want to be responsible the solution isn't abdicate an open web. It's make better law and enforcement of said law.

edm0nd 1y ago

No we dont

ctoth 1y ago

The funny thing about the good old WWW is the first two W's stand for world-wide. So Which legal teeth?

account42 1y ago
Try hosting some illegal-in-the-US content and find out.

jopsen 1y ago

I have the feeling that it's the small players that cause problems. Dumb bots that don't respect robot.txt or nofollow are the ones trying all combinations of the filters available in your search options and requesting all pages for each such combination. The number of search pages can easily be exponential in the number of filters you offer. Bots walking around in these traps, do it because they are dumb. But even a small degenerate bot can send more requests than 1M MAUs. At least that's my impression of the problem we're sometimes facing. Signed agents seems like a horrific solution. And many serving the traffic is just better.