5 ms·

Google scrapes like a maniac. And for profit. Many others do the same. A website can put up a TOS prohibiting such use, but my understanding is that is essenti

by bicx 2y ago

Google scrapes like a maniac. And for profit. Many others do the same. A website can put up a TOS prohibiting such use, but my understanding is that is essentially unenforceable if the site is publicly accessible. The recent Meta v Bright Data case highlights how extreme it can get without being technically illegal. https://techcrunch.com/2024/02/26/meta-drops-lawsuit-against-web-scraping-firm-bright-data-that-sold-millions-of-instagram-records/ https://techcrunch.com/2024/02/26/meta-drops-lawsuit-against... If you’re trying to prevent scraping of your data, your best option is to not make it public.

herbst 2y ago
I have a website that has around 5k Bing visits every single day. Basically my most expensive user. Compared to Google with about 70 visits daily. I randomly block their IPs, tried some stuff with robots.txt and even completely banned it in the past as I thought this must be something else. It would just show up with new IPs and proceed. The few times I checked it looked like official IPs. If I knew how I would sue Microsoft. They have no business in scraping my website 3-5 times a day when they send me basically no traffic Edit:// it's also not my only website where bing goes crazy. And it's not new, this is going on for several years now (so no AI scraping I guess)
- tetris11 2y ago
  They absolutely should be paying you if they're single-handedly abusing your resources
  - themoonisachees 2y ago
    There is a YouTube video about a content creator who got a huge AWS bill out of byte spider (bytedance's scraper) indexing his site incredible numbers of times.
    - Nextgrid 2y ago
      Or maybe just don't host your stuff in such a way that any internet user can bankrupt you just from sending GET requests?
      - matthews2 2y ago
        If GET requests increase my AWS bill, does that make them not idempotent any more?
        qup 2y ago
        This sounds like a scoping issue
      - herbst 2y ago
        This so much. While it's annoying to waste resources for nothing. All I really had to do to make this a non issue is upgrading my VPS to a slightly larger box. If I would pay per requests I couldn't sleep anymore
  - Nextgrid 2y ago
    On the other hand the amount of traffic he's mentioning is about 3.4 requests per minute which is effectively background noise. They're probably getting much more hits from actually malicious scanners than Bing's misbehaving spider.
- Nextgrid 2y ago
  > The few times I checked it looked like official IPs Considering Microsoft now runs a cloud service, it may very well be their cloud provider users and not official Bing scrapers.
wildrhythms 2y ago
Usually it's robots.txt that 'prohibits' such use, but you're right it's not enforceable.

by bicx 2y ago

herbst 2y ago

I have a website that has around 5k Bing visits every single day. Basically my most expensive user. Compared to Google with about 70 visits daily. I randomly block their IPs, tried some stuff with robots.txt and even completely banned it in the past as I thought this must be something else. It would just show up with new IPs and proceed. The few times I checked it looked like official IPs. If I knew how I would sue Microsoft. They have no business in scraping my website 3-5 times a day when they send me basically no traffic Edit:// it's also not my only website where bing goes crazy. And it's not new, this is going on for several years now (so no AI scraping I guess)

tetris11 2y ago
They absolutely should be paying you if they're single-handedly abusing your resources
- themoonisachees 2y ago
  There is a YouTube video about a content creator who got a huge AWS bill out of byte spider (bytedance's scraper) indexing his site incredible numbers of times.
  - Nextgrid 2y ago
    Or maybe just don't host your stuff in such a way that any internet user can bankrupt you just from sending GET requests?
    - matthews2 2y ago
      If GET requests increase my AWS bill, does that make them not idempotent any more?
      - qup 2y ago
        This sounds like a scoping issue
    - herbst 2y ago
      This so much. While it's annoying to waste resources for nothing. All I really had to do to make this a non issue is upgrading my VPS to a slightly larger box. If I would pay per requests I couldn't sleep anymore
- Nextgrid 2y ago
  On the other hand the amount of traffic he's mentioning is about 3.4 requests per minute which is effectively background noise. They're probably getting much more hits from actually malicious scanners than Bing's misbehaving spider.
Nextgrid 2y ago
> The few times I checked it looked like official IPs Considering Microsoft now runs a cloud service, it may very well be their cloud provider users and not official Bing scrapers.

wildrhythms 2y ago

Usually it's robots.txt that 'prohibits' such use, but you're right it's not enforceable.