5 ms·

Is cloudflare becoming a mob outfit? Because they are selling scraping countermeasures but are now selling scraping too. And they can pull it off because of th

by ljm 7mo ago

Is cloudflare becoming a mob outfit? Because they are selling scraping countermeasures but are now selling scraping too. And they can pull it off because of their reach over the internet with the free DNS.

rrr_oh_man 7mo ago
[flagged]
- stri8ted 7mo ago
  Do you have any evidence to support this view?
  - rolymath 7mo ago
    Read who and how it was founded. It's not a secret at all.
    - rrr_oh_man 7mo ago
      It’s funny how I got immediately downvoted and flagged
  - pocksuppet 7mo ago
    Who else would MITM 30% of the internet?
- mtmail 7mo ago
  Any kind of source for the claim?
Retr0id 7mo ago
For a long time cloudflare has proudly protected DDoS-as-a-service sites (but of course, they claim they don't "host" them)
- Dylan16807 7mo ago
  Are you using the word "claim" to call them wrong or for a more confusing reason? Because I'm pretty sure they are not in fact wrong.
  - Retr0id 7mo ago
    The distinction between a caching proxy and an origin server is pretty meaningless when you're serving static content, if you ask me.
    - Dylan16807 7mo ago
      There's a blurry line there, true. On the other hand when a page is small and static enough that it's basically just a flyer, I also care a lot less about who hosts it.
theamk 7mo ago
no? it takes 10 seconds to check: > The /crawl endpoint respects the directives of robots.txt files, including crawl-delay. All URLs that /crawl is directed not to crawl are listed in the response with "status": "disallowed". You don't need any scraping countermeasures for crawlers like those.
- Macha 7mo ago
  So what’s the user agent for their bot? They don’t seem to specify the default in the docs and it looks like it’s user configurable. So yet another opt out bot which you need your web server to match on special behaviour to block
  - gruez 7mo ago
    >So yet another opt out bot which you need your web server to match on special behaviour to block Given that malicious bots are allegedly spoofing real user agents, "another user agent you have to add to your list" seems like the least of your problems.
    - AdamN 7mo ago
      Not 'allegedly' - it's just a fact. Even if you're not malicious however it's still sometimes necessary because the server may have different sites for different browsers and check user agents for the experience they deliver. So then even for legitimate purposes you need to at least use the prefix of the user agent that the server expects.
    - Macha 7mo ago
      It is cloudflare who made the claim that they are well behaved unlike those other bots and that their behaviour can be controlled by robots.txt If I need to treat cloudflare bots the same as malicious bots, that undermines their claim.
  - flanksteak20 7mo ago
    Isn't this covered here? https://developers.cloudflare.com/browser-rendering/reference/robots-txt/#identifying-browser-rendering-requests https://developers.cloudflare.com/browser-rendering/referenc...
iso-logi 7mo ago
Their free DNS is only a small piece of the pie. The fact that 30%+ of the web relies on their caching services, routablility services and DDoS protection services is the main pull. Their DNS is only really for data collection and to front as "good will"
- jen729w 7mo ago
  > The fact that 30%+ of the web relies on their caching services 30% of the web might use their caching services. 'Relies on' implies that it wouldn't work without them, which I doubt is the case. It might be the case for the biggest 1% of that 30%. But not the whole lot.
  - reddalo 7mo ago
    >'Relies on' implies that it wouldn't work without them Last time Cloudflare went down, their dashboard was also unavailable, so you couldn't turn off their proxy service anyway.
QuantumNoodle 7mo ago
Cloudflare has been trying to mediate publishers & AI companies. If publishers are behind Cloudflare and Cloudflare's bot detection stops scrapers at the request of publishers, the publishers can allow their data to be scraped (via this end point) for a price. It creates market scarcity. I don't believe the target audience is you and me. Unless you own a very popular blog that AI companies would pay you for.
- PeterStuer 7mo ago
  Next step will be their default "free" anti-bot denying all but their own bot. They know full well nearly nobody changes the default.
shadowfiend 7mo ago
No: https://developers.cloudflare.com/browser-rendering/rest-api/crawl-endpoint/#robotstxt-and-bot-protection https://developers.cloudflare.com/browser-rendering/rest-api...
- oefrha 7mo ago
  That's not the perfect defense you think it is. Plenty of robots.txts[1] technically allow scraping their main content pages as long as your user-agent isn't explicitly disallowed, but in practice they're behind Cloudflare so they still throw up Cloudflare bot check if you actually attempt to crawl. And forget about crawling. If you have a less reputable IP (basically every IP in third world countries are less reputable, for instance), you can be CAPTCHA'ed to no end by Cloudflare even as a human user, on the default setting, so plenty of site owners with more reputable home/office IPs don't even know what they subject a subset of their users to. [1] E.g. https://www.wired.com/robots.txt https://www.wired.com/robots.txt to pick an example high up on HN front page.
  - account42 7mo ago
    > If you have a less reputable IP (basically every IP in third world countries are less reputable, for instance), you can be CAPTCHA'ed to no end by Cloudflare even as a human user, on the default setting, so plenty of site owners with more reputable home/office IPs don't even know what they subject a subset of their users to. Or if you have a less common browser like Firefox with some moderate privacy settings/extensions.
giancarlostoro 7mo ago
If they ever sell or the CEO shifts, yes. For the meantime, they have not given any strong indication that they're trying to bully anybody. I could see things changing drastically if the people in charge are swapped out.
subscribed 7mo ago
I think there's some space being absolutely snuffed by the countless bots of everyone, ignoring everything, pulling from residential proxies, and this, supposedly slower, well behavior, smarter bot. Like there's a difference between dozens of drunk teenagers thrashing the city streets in the illegal street race vs a taxi driver.
pocksuppet 7mo ago
Was it ever not one? They protect a lot of DDoS-for-hire sites from DDoS by their competitors. In return they increase the quantity of DDoS on the internet. They offer you a service for $150, then months later suddenly demand $150k in 24 hours or they shut down your business. If you use them as a DNS registrar they will hold your domain hostage.
- Sebguer 7mo ago
  yeah, GP completely fails to realize that Cloudflare has always played both sides. that is their entire business model, and it was transparent from the beginning that they would absolutely do the same here.
- azinman2 7mo ago
  Where can I learn more about the 150k in 24h?
  - caffeinewriter 7mo ago
    I imagine it's referencing this story: https://robindev.substack.com/p/cloudflare-took-down-our-website https://robindev.substack.com/p/cloudflare-took-down-our-web... HN Discussion: https://news.ycombinator.com/item?id=40481808 https://news.ycombinator.com/item?id=40481808
isodev 7mo ago
They always have been. They also use their dominant position to apply political pressure when they don’t like how a country chooses to run things. So yeah, we’ve created another mega corp monster that will hurt for years to come.
rendaw 7mo ago
I think the simple explanation is that they weren't selling scraping countermeasures, they were selling web-based denial of service protection (which may be caused by scrapers).
- PeterStuer 7mo ago
  Ask yourself, why would a scraper ddos? Why would a ddos-protection vendor ddos?
  - wongarsu 7mo ago
    Because the scraper is either impatient, careless or indifferent; and if they scrape for training data they don't plan to come back. If they don't plan to come back they don't care if you tighten up crawling protections after they have moved on. In fact they are probably happy that they got their data and their competition won't
    - wiether 7mo ago
      > they don't plan to come back To me the current behavior of those scrapers tells me that "they don't plan", period. Looks like they hired a bunch of excavators and are digging 2 meters deep on whole fields, looking for nuggets of gold, and pilling the dirt on a huge mountain. Once they realize the field was bereft of any gold but full of silver? Or that the gold was actually 2.5 meters deep? They have to go through everything again.
  - c0balt 7mo ago
    The number of git forges behind Anubis et al and the numerous public announcements should be enough. Scrappers seem to be exceedingly careless in using public resources. The problem is often not even DDOS (as in overwhelming bandwidth usage) but rather DOS through excessive hits on expensive routes.
  - junaru 7mo ago
    > Ask yourself, why would a scraper ddos? Don't need to ask anything i can tell you exactly - because they have no regard for anything but their own profit. Let me give you an example of this mom and pop shop known as anthropic. You see they have this thing called claudebot and at least initially it scraped iterating through IP's. Now you have these things called shared hosting servers, typically running 1000-10000 domains of actual low volume websites on 1-50 or so IPs. Guess what happens when it is your networks time to bend over? Whole hosting company infrastructure going down as each server has hundreds of claudebots crawling hundreds of vhosts at the same time. This happened for months. Its the reason they are banned in WAFs by half the hosting industry.
andrepd 7mo ago
Well this scraper honours robots.txt so I'm sure most AI crawlers will find it useless.

by ljm 7mo ago

rrr_oh_man 7mo ago

[flagged]

stri8ted 7mo ago
Do you have any evidence to support this view?
- rolymath 7mo ago
  Read who and how it was founded. It's not a secret at all.
  - rrr_oh_man 7mo ago
    It’s funny how I got immediately downvoted and flagged
- pocksuppet 7mo ago
  Who else would MITM 30% of the internet?
mtmail 7mo ago
Any kind of source for the claim?

Retr0id 7mo ago

For a long time cloudflare has proudly protected DDoS-as-a-service sites (but of course, they claim they don't "host" them)

Dylan16807 7mo ago
Are you using the word "claim" to call them wrong or for a more confusing reason? Because I'm pretty sure they are not in fact wrong.
- Retr0id 7mo ago
  The distinction between a caching proxy and an origin server is pretty meaningless when you're serving static content, if you ask me.
  - Dylan16807 7mo ago
    There's a blurry line there, true. On the other hand when a page is small and static enough that it's basically just a flyer, I also care a lot less about who hosts it.

theamk 7mo ago

no? it takes 10 seconds to check: > The /crawl endpoint respects the directives of robots.txt files, including crawl-delay. All URLs that /crawl is directed not to crawl are listed in the response with "status": "disallowed". You don't need any scraping countermeasures for crawlers like those.

Macha 7mo ago
So what’s the user agent for their bot? They don’t seem to specify the default in the docs and it looks like it’s user configurable. So yet another opt out bot which you need your web server to match on special behaviour to block
- gruez 7mo ago
  >So yet another opt out bot which you need your web server to match on special behaviour to block Given that malicious bots are allegedly spoofing real user agents, "another user agent you have to add to your list" seems like the least of your problems.
  - AdamN 7mo ago
    Not 'allegedly' - it's just a fact. Even if you're not malicious however it's still sometimes necessary because the server may have different sites for different browsers and check user agents for the experience they deliver. So then even for legitimate purposes you need to at least use the prefix of the user agent that the server expects.
  - Macha 7mo ago
    It is cloudflare who made the claim that they are well behaved unlike those other bots and that their behaviour can be controlled by robots.txt If I need to treat cloudflare bots the same as malicious bots, that undermines their claim.
- flanksteak20 7mo ago
  Isn't this covered here? https://developers.cloudflare.com/browser-rendering/reference/robots-txt/#identifying-browser-rendering-requests https://developers.cloudflare.com/browser-rendering/referenc...

iso-logi 7mo ago

Their free DNS is only a small piece of the pie. The fact that 30%+ of the web relies on their caching services, routablility services and DDoS protection services is the main pull. Their DNS is only really for data collection and to front as "good will"

jen729w 7mo ago
> The fact that 30%+ of the web relies on their caching services 30% of the web might use their caching services. 'Relies on' implies that it wouldn't work without them, which I doubt is the case. It might be the case for the biggest 1% of that 30%. But not the whole lot.
- reddalo 7mo ago
  >'Relies on' implies that it wouldn't work without them Last time Cloudflare went down, their dashboard was also unavailable, so you couldn't turn off their proxy service anyway.

QuantumNoodle 7mo ago

Cloudflare has been trying to mediate publishers & AI companies. If publishers are behind Cloudflare and Cloudflare's bot detection stops scrapers at the request of publishers, the publishers can allow their data to be scraped (via this end point) for a price. It creates market scarcity. I don't believe the target audience is you and me. Unless you own a very popular blog that AI companies would pay you for.

PeterStuer 7mo ago
Next step will be their default "free" anti-bot denying all but their own bot. They know full well nearly nobody changes the default.

shadowfiend 7mo ago

No: https://developers.cloudflare.com/browser-rendering/rest-api/crawl-endpoint/#robotstxt-and-bot-protection https://developers.cloudflare.com/browser-rendering/rest-api...

oefrha 7mo ago
That's not the perfect defense you think it is. Plenty of robots.txts[1] technically allow scraping their main content pages as long as your user-agent isn't explicitly disallowed, but in practice they're behind Cloudflare so they still throw up Cloudflare bot check if you actually attempt to crawl. And forget about crawling. If you have a less reputable IP (basically every IP in third world countries are less reputable, for instance), you can be CAPTCHA'ed to no end by Cloudflare even as a human user, on the default setting, so plenty of site owners with more reputable home/office IPs don't even know what they subject a subset of their users to. [1] E.g. https://www.wired.com/robots.txt https://www.wired.com/robots.txt to pick an example high up on HN front page.
- account42 7mo ago
  > If you have a less reputable IP (basically every IP in third world countries are less reputable, for instance), you can be CAPTCHA'ed to no end by Cloudflare even as a human user, on the default setting, so plenty of site owners with more reputable home/office IPs don't even know what they subject a subset of their users to. Or if you have a less common browser like Firefox with some moderate privacy settings/extensions.

giancarlostoro 7mo ago

If they ever sell or the CEO shifts, yes. For the meantime, they have not given any strong indication that they're trying to bully anybody. I could see things changing drastically if the people in charge are swapped out.

subscribed 7mo ago

I think there's some space being absolutely snuffed by the countless bots of everyone, ignoring everything, pulling from residential proxies, and this, supposedly slower, well behavior, smarter bot. Like there's a difference between dozens of drunk teenagers thrashing the city streets in the illegal street race vs a taxi driver.

pocksuppet 7mo ago

Was it ever not one? They protect a lot of DDoS-for-hire sites from DDoS by their competitors. In return they increase the quantity of DDoS on the internet. They offer you a service for $150, then months later suddenly demand $150k in 24 hours or they shut down your business. If you use them as a DNS registrar they will hold your domain hostage.

Sebguer 7mo ago
yeah, GP completely fails to realize that Cloudflare has always played both sides. that is their entire business model, and it was transparent from the beginning that they would absolutely do the same here.
azinman2 7mo ago
Where can I learn more about the 150k in 24h?
- caffeinewriter 7mo ago
  I imagine it's referencing this story: https://robindev.substack.com/p/cloudflare-took-down-our-website https://robindev.substack.com/p/cloudflare-took-down-our-web... HN Discussion: https://news.ycombinator.com/item?id=40481808 https://news.ycombinator.com/item?id=40481808

isodev 7mo ago

They always have been. They also use their dominant position to apply political pressure when they don’t like how a country chooses to run things. So yeah, we’ve created another mega corp monster that will hurt for years to come.

rendaw 7mo ago

I think the simple explanation is that they weren't selling scraping countermeasures, they were selling web-based denial of service protection (which may be caused by scrapers).

PeterStuer 7mo ago
Ask yourself, why would a scraper ddos? Why would a ddos-protection vendor ddos?
- wongarsu 7mo ago
  Because the scraper is either impatient, careless or indifferent; and if they scrape for training data they don't plan to come back. If they don't plan to come back they don't care if you tighten up crawling protections after they have moved on. In fact they are probably happy that they got their data and their competition won't
  - wiether 7mo ago
    > they don't plan to come back To me the current behavior of those scrapers tells me that "they don't plan", period. Looks like they hired a bunch of excavators and are digging 2 meters deep on whole fields, looking for nuggets of gold, and pilling the dirt on a huge mountain. Once they realize the field was bereft of any gold but full of silver? Or that the gold was actually 2.5 meters deep? They have to go through everything again.
- c0balt 7mo ago
  The number of git forges behind Anubis et al and the numerous public announcements should be enough. Scrappers seem to be exceedingly careless in using public resources. The problem is often not even DDOS (as in overwhelming bandwidth usage) but rather DOS through excessive hits on expensive routes.
- junaru 7mo ago
  > Ask yourself, why would a scraper ddos? Don't need to ask anything i can tell you exactly - because they have no regard for anything but their own profit. Let me give you an example of this mom and pop shop known as anthropic. You see they have this thing called claudebot and at least initially it scraped iterating through IP's. Now you have these things called shared hosting servers, typically running 1000-10000 domains of actual low volume websites on 1-50 or so IPs. Guess what happens when it is your networks time to bend over? Whole hosting company infrastructure going down as each server has hundreds of claudebots crawling hundreds of vhosts at the same time. This happened for months. Its the reason they are banned in WAFs by half the hosting industry.

andrepd 7mo ago

Well this scraper honours robots.txt so I'm sure most AI crawlers will find it useless.