3 ms·

If they don't respect robots.txt then block them using a firewall or other server config. All of these companies are parasites.

by cdme 3y ago

pksebben 3y ago
I don't think this message is about "protecting the site's data" quite so much as "hey guys, you're wasting a ton of time and network connect to make your model worse. Might wanna do something 'bout that"
- cdme 3y ago
  I suppose in that case, let them keep wasting their time.
jeremyjh 3y ago
The entire purpose of this website is to identify bad actors who do not respect robots.txt, so that they can be publicly shamed.
- cdme 3y ago
  Well, we know where OpenAI lands then.
m3047 3y ago
No. I've run 'bot motels myself. I've got better things to do than curating a block list when they can just switch or renumber their infrastructure. Most notably I ran a 'bot motel on a compute-intensive web app; it was cheaper to burn bandwidth (and I slow-rolled that) than CPU cycles. Poisoning the datasets was just lulz. I block ping from virtually all of Amazon; there are a few providers out there for which I block every naked SYN coming to my environment except port 25, and a smattering I block entirely. I can't prove that the pings even come from Amazon, even if the pongs are supposed to go there (although I have my suspicions that even if the pings don't come from the host receiving the pongs the pongs are monitored by the generator of the pings). The point I'm making is that e.g. Amazon doesn't have the right to sell access to my compute and tragedy of the commons applies, folks. I offered them a live feed of the worst offenders, but all they want is pcaps. (I've got a list of 50 prefixes, small enough to be a separate specialty firewall table. It misses a few things and picks up some dust bunnies. But contrast that to the 8,000 prefixes they publish in that JSON file. Spoiler alert: they won't admit in that JSON file that they own the entirety of 3.0.0.0/8. I'm willing to share the list TLP:RED/YELLOW, hunt me down and introduce yourself.)

If they don't respect robots.txt then block them using a firewall or other server config. All of these companies are parasites.

by cdme 3y ago

If they don't respect robots.txt then block them using a firewall or other server config. All of these companies are parasites.

pksebben 3y ago

I don't think this message is about "protecting the site's data" quite so much as "hey guys, you're wasting a ton of time and network connect to make your model worse. Might wanna do something 'bout that"

cdme 3y ago
I suppose in that case, let them keep wasting their time.

jeremyjh 3y ago

The entire purpose of this website is to identify bad actors who do not respect robots.txt, so that they can be publicly shamed.

cdme 3y ago
Well, we know where OpenAI lands then.

m3047 3y ago

No. I've run 'bot motels myself. I've got better things to do than curating a block list when they can just switch or renumber their infrastructure. Most notably I ran a 'bot motel on a compute-intensive web app; it was cheaper to burn bandwidth (and I slow-rolled that) than CPU cycles. Poisoning the datasets was just lulz. I block ping from virtually all of Amazon; there are a few providers out there for which I block every naked SYN coming to my environment except port 25, and a smattering I block entirely. I can't prove that the pings even come from Amazon, even if the pongs are supposed to go there (although I have my suspicions that even if the pings don't come from the host receiving the pongs the pongs are monitored by the generator of the pings). The point I'm making is that e.g. Amazon doesn't have the right to sell access to my compute and tragedy of the commons applies, folks. I offered them a live feed of the worst offenders, but all they want is pcaps. (I've got a list of 50 prefixes, small enough to be a separate specialty firewall table. It misses a few things and picks up some dust bunnies. But contrast that to the 8,000 prefixes they publish in that JSON file. Spoiler alert: they won't admit in that JSON file that they own the entirety of 3.0.0.0/8. I'm willing to share the list TLP:RED/YELLOW, hunt me down and introduce yourself.)