12 ms·

Cleaning up bad bots

98 points by m1guelpf 7y ago

kylehotchkiss 7y ago
If any cloudflare staff happen to see this... is Cloudflare Warp coming soon? I was pretty excited to give that a try (and it seemed like something I would be willing to pay for)
- jgrahamc 7y ago
  https://blog.cloudflare.com/birthday-week-2019/ https://blog.cloudflare.com/birthday-week-2019/
  - cj 7y ago
    Sounds like it'll come on Wednesday: "The wait is over. Our product and engineering teams have been working round-the-clock to build a new experience that makes the Internet faster and safer for everyone. If I were to say anything more, it would surely give it away so I’ll leave it at that — you’ll just need to tune in to the blog on Wednesday to find out."
  - kylehotchkiss 7y ago
    I asked at the perfect time :D Thanks for responding.
- _eht 7y ago
  Found the shill bot...
- nullbyte 7y ago
  Wednesday.
bonerman69 7y ago
> for Bandwidth Alliance partners, we’re going to hand the IP of the bot to the partner and get the bot kicked offline; What's that mean, 'kicked offline'? Isn't scraping 'legal'?
- jgrahamc 7y ago
  There's a lot more to bots than just scraping... DDoS bots, bots that buy hot sneakers before the public gets a chance and drive up the price, bots that go credential stuffing, bots that play nasty tricks with airline seats ...
  - vorpalhex 7y ago
    They don't seem to specify that on this post. I hope it's merely miscommunication...
  - ikeboy 7y ago
    Only two of those are malicious. Buying products at the price offered is perfectly legitimate, regardless of all the scaremongering to the contrary.
    - eli 7y ago
      Not sure what "legitimate" means here? Legal? Running aggressive web crawlers is in many instances against the rules for consumer cloud servers. For example, AWS requires that you obey robots.txt if you run a crawler there. https://aws.amazon.com/premiumsupport/knowledge-center/report-aws-resource-crawling/ https://aws.amazon.com/premiumsupport/knowledge-center/repor... In my experience a lot of bots seem to be running on hacked servers or through hacked/insecure proxies. I'd imagine tracking down the owner or someone upstream of those boxes could be effective in taking them offline.
      - ikeboy 7y ago
        What does that have to with my point? Bots used to purchase inventory (and that aren't otherwise commiting fraud by using stolen credit cards or something) are not malicious.
        eli 7y ago
        Are they following the sneaker website's robots.txt while doing that? If not, they are probably violating the AWS terms regardless of whether you believe that activity is "malicious."
acolytic 7y ago
I'm not very familiar with the concept of tarpitting. How do they get the bot to run CPU intensive code? By passing in extra Javascript? Can this affect a bot that doesn't run any JS?
- s09dfhks 7y ago
  Was also curious about this step. I assume they're not going to reveal the nitty gritty details for fear of botters coding around it, but I am curious as to how you can "make them use more CPU" while crawling a website
  - nielsbot 7y ago
    Maybe it's JavaScript?
  - jsnell 7y ago
    Serve the suspected bots a page with Javascript that computes a proof of work, submits it, and gets the real page in return. Steamspy.com seems to trigger one of these basically every time when loaded with a fresh cookie.
- opwieurposiu 7y ago
  One way is to use a special very slow TCP handler. Imagine a TCP stack that only lets through one 10 byte packet every 10 seconds. This wastes a connection slot on target machine. https://en.wikipedia.org/wiki/Slowloris_(computer_security) https://en.wikipedia.org/wiki/Slowloris_(computer_security)
- AznHisoka 7y ago
  I've been running many non-JS crawlers for the past few years, and there were a few pages that kept pushing the CPU load of my servers to a halt. When I dug into the source code, I saw that the HTML was a convoluted text of tables inside tables inside tables inside more tables, thus making it incredibly time-consuming + CPU-intensive for my DOM parser to parse (I was using Nokogiri, a Ruby gem at the time). Thus Cloudflare could be serving these types of "fake" pages to bad bots. They could also be doing things like serving fake streaming audio that never ends, or anything that might make it seem like the web page is just a huge page that needs time to load.
comex 7y ago
> Our bot detection breaks down into four large components: > - Identification of well known legitimate bots; What about non-well-known legitimate bots? If I run my own web crawler, am I at risk of falling into the tarpit (and having my IP address reported)?
- avip 7y ago
  What makes your crawler legitimate?
  - AznHisoka 7y ago
    This builds up to a bigger question, with anti-competitive implications. What if you want to build a Google competitor? Is Google the only one that's allowed to crawl web pages now, and anyone else is automatically deemed a bad bot? Is CloudFlare the one that decides who can be a search engine, and who can't?
    - RcouF1uZ4gsC 7y ago
      >What if you want to build a Google competitor? Use CommonCrawl https://commoncrawl.org/ https://commoncrawl.org/ Then, after you can use that corpus and provide useful results and have people coming to your search site, then you can have a discussion with CloudFlare that you will be creating your own crawler.
      - 4gotunameagain 7y ago
        What if for whatever reason the innovation is on the crawler itself
        kordlessagain 7y ago
        An AI crawler.
      - AznHisoka 7y ago
        Not sure how it is now, but when I looked into CommonCrawl, they had a significant delay in storing newer pages. It took at least a month before a web page made it into their index. Which makes creating any type of Google-like search engine very tough.
mfontani 7y ago
I've toyed with the idea of a tarpit service for badly behaved crawlers which just don't get other hints. I'm glad to see that if my services are behind cloudflare, I could just turn something on and let _them_ deal with it.
- dclusin 7y ago
  What is does it mean to tarpit someone? Throttle their connection?
  - petre 7y ago
    Waste their time with a very slow server. Like OpenBSD’s spamd. Think one letter per second for text based protocols.
    - mfontani 7y ago
      Precisely that
  - snowwolf 7y ago
    Something like this https://github.com/die-net/http-tarpit https://github.com/die-net/http-tarpit
  - KineticLensman 7y ago
    To delay incoming connections https://en.wikipedia.org/wiki/Tarpit_(networking) https://en.wikipedia.org/wiki/Tarpit_(networking)
  - judge2020 7y ago
    Cloudflare's specific methods are described in the article, it's likely some web crypto stuff or an extreme response that takes a long time to parse or handle.
ikeboy 7y ago
>Another trend we have seen is the increase of the combination of bots with botnets, particularly in the world of inventory hoarding bots. The motivation and willingness to spend for these bot operators is quite high. >The targets are goods of generally of limited supply and high in demand and in value. Think sneakers, concert tickets, airline seats, and popular short run Broadway musicals. Bot operators who are able to purchase those items at retail can charge massive premiums in aftermarket sales. When the operator identifies a target site, such as an ecommerce retailer, and a specific item, such as a new pair of sneakers going on sale, they can purchase time on the new Residential Proxy as a Service market to gain access to end user machines and (relatively) clean IPs from which to launch their attack. They then go on to spout some economic nonsense about how such bots are harmful. Actually, resellers make the market more efficient, and cloudflare is doing a disservice by lumping legitimate bots in with malicious ones like their credential stuffing example.
- deadbunny 7y ago
  How do bots make the market more efficient?
  - ikeboy 7y ago
    By moving the goods to those willing to pay more for them, which increases consumer surplus, which means the market is more efficient. It's fairly basic economics.
    - comex 7y ago
      Increasing the price should reduce consumer surplus, not increase it. If the manufacturer of an item increases its price and thus their profit from it, then in principle the increased profit can drive them to manufacture more of the item, avoiding deadweight loss due to insufficient supply. However, that doesn’t apply if a reseller is the one making the profits! And in any case, some of the concrete examples cited (tickets to concerts and Broadway musicals) have an essentially fixed supply; it’s hard or impossible to simply increase production. Separately, moving goods toward those willing to pay more for them is to some extent equivalent to moving them toward those who care more about them, which could be said to improve efficiency in terms of how much personal happiness is gained per unit. However, it also moves the goods toward those who are simply wealthier. And there are other possible proxies for personal interest besides price, ones that aren’t as affected by wealth. One of them is a person’s willingness to research in advance when certain tickets will go on sale, and be online at that time to snag them before they sell out – in other words, the system that bots are used to subvert. This system has its own biases (favoring those who are organized, have more flexible schedules, or are simply in a more favorable timezone), but it‘s still considered to be reasonably fair, at least when it’s not being subverted.
tony 7y ago
Cloudflare, as a service, has it all. Nice UX, fast, free. Nice domain service if you transfer to them. Fast DNS management. DOS/bot mitigation. Caching. Quick SSL, and affordable upgrade options. 2FA via TOTP. With all the networking/domain stuff momentum build, it would be nice to be able to spin up servers for apps/db. Main language at cloudflare is golang? Rust? Any python over there? Request: Allow changing the super administrator for cloudflare account more easily. At least for early-stage accounts.
- jgrahamc 7y ago
  Lots of Go, lots of Rust, lots of Python. Also... Lua, PHP, C++.
- mfontani 7y ago
  Not yet U2F, which I'm hopeful they get real soon
  - judge2020 7y ago
    https://twitter.com/eastdakota/status/1175522433643601920?s=21 https://twitter.com/eastdakota/status/1175522433643601920?s=...
- judge2020 7y ago
  I’d still consider the registrar service a “maturing product” at best, similar to how CF was in its early years. Enough time and it’ll be polished to where I can say it’s “production ready”.
thegeomaster 7y ago
Great! Let's add it to the list of obstacles to accessing content for us savages in small, poor, out-of-sight-out-of-mind countries: - Harrassment with an extra verification step every god damn time you log in into a website you've been a paying customer at for years because some rando risk model thinks you're a bad guy - Harrassment by reCAPTCHA to respond to US-centric image challenges when it decides it's up for some sadistic fun (Mark all images with a store front or street light? Sure, let me just Google how they are "supposed" to look, because I sure as hell ain't seeing any I'm familiar with here.) - Blanket IP range bans which serve you a default 503 error page and call it a day, with ripple effects throghout tons of unrelated websites. I always know when my ISP's IP range is added to some new blacklist. - Harrassment by Cloudflare "verifying my browser", presumably burning my CPU cycles so they can be sure that my browser, which has been hitting their IP ranges for years with not as much as a cookie wipe, has not suddenly turned into a bot - "Your name is invalid": no, your regex is - "Our fetishism for credit prevents us from accepting your debit card payment unless you submit a scan of your passport signed in blood" - (NEW) CloudFlare's anti-bot measures which will surely not misfire because no one thought to test their shiny new model on the traffic patterns of some culture with < 50 million Internet users Apologize for the off-topic rant, but you wouldn't believe how using the Internet for basic things has gotten difficult in the past few years over here (and from what I hear, in many other "forgotten" geographies). No one cares. Long live colonialism!
- Meekro 7y ago
  I'm sorry you have to put up with all of that, and I don't want to be part of the problem for folks like you. But here's the other side of the coin: I ran a small VPS hosting company for many years, and the "small, poor, out-of-sight-out-of-mind countries" are a huge source of fraud. Someone signing up from an IP in Latvia, for example, was virtually guaranteed to be paying with a stolen card that would later get disputed. Debit card payments can get disputed, too, and disputes can happen as much as 120 days after the initial charge. For comparison purposes, the dispute rate for US-origin payments (US card, US origin IP, no evidence of proxy use) was under 1%. If I couldn't do things like "submit a scan of your passport", I would probably just decline any cards from high-risk countries. What would you suggest I do instead?
andrerm 7y ago
This is Claudfare saying that now that they arbitrated X and everybody said "OK" they are moving from infrastructure into infra and arbitration. My question is, what if you are starting an alternative search engine or something legit? Edit: my point is, the rules than make a legit or not bot, crawler, scrapper etc. are not clear at all.

98 points by m1guelpf 7y ago

If any cloudflare staff happen to see this... is Cloudflare Warp coming soon? I was pretty excited to give that a try (and it seemed like something I would be willing to pay for)

jgrahamc 7y ago
https://blog.cloudflare.com/birthday-week-2019/ https://blog.cloudflare.com/birthday-week-2019/
- cj 7y ago
  Sounds like it'll come on Wednesday: "The wait is over. Our product and engineering teams have been working round-the-clock to build a new experience that makes the Internet faster and safer for everyone. If I were to say anything more, it would surely give it away so I’ll leave it at that — you’ll just need to tune in to the blog on Wednesday to find out."
- kylehotchkiss 7y ago
  I asked at the perfect time :D Thanks for responding.
_eht 7y ago
Found the shill bot...
nullbyte 7y ago
Wednesday.

bonerman69 7y ago

> for Bandwidth Alliance partners, we’re going to hand the IP of the bot to the partner and get the bot kicked offline; What's that mean, 'kicked offline'? Isn't scraping 'legal'?

jgrahamc 7y ago
There's a lot more to bots than just scraping... DDoS bots, bots that buy hot sneakers before the public gets a chance and drive up the price, bots that go credential stuffing, bots that play nasty tricks with airline seats ...
- vorpalhex 7y ago
  They don't seem to specify that on this post. I hope it's merely miscommunication...
- ikeboy 7y ago
  Only two of those are malicious. Buying products at the price offered is perfectly legitimate, regardless of all the scaremongering to the contrary.
  - eli 7y ago
    Not sure what "legitimate" means here? Legal? Running aggressive web crawlers is in many instances against the rules for consumer cloud servers. For example, AWS requires that you obey robots.txt if you run a crawler there. https://aws.amazon.com/premiumsupport/knowledge-center/report-aws-resource-crawling/ https://aws.amazon.com/premiumsupport/knowledge-center/repor... In my experience a lot of bots seem to be running on hacked servers or through hacked/insecure proxies. I'd imagine tracking down the owner or someone upstream of those boxes could be effective in taking them offline.
    - ikeboy 7y ago
      What does that have to with my point? Bots used to purchase inventory (and that aren't otherwise commiting fraud by using stolen credit cards or something) are not malicious.
      - eli 7y ago
        Are they following the sneaker website's robots.txt while doing that? If not, they are probably violating the AWS terms regardless of whether you believe that activity is "malicious."

acolytic 7y ago

I'm not very familiar with the concept of tarpitting. How do they get the bot to run CPU intensive code? By passing in extra Javascript? Can this affect a bot that doesn't run any JS?

s09dfhks 7y ago
Was also curious about this step. I assume they're not going to reveal the nitty gritty details for fear of botters coding around it, but I am curious as to how you can "make them use more CPU" while crawling a website
- nielsbot 7y ago
  Maybe it's JavaScript?
- jsnell 7y ago
  Serve the suspected bots a page with Javascript that computes a proof of work, submits it, and gets the real page in return. Steamspy.com seems to trigger one of these basically every time when loaded with a fresh cookie.
opwieurposiu 7y ago
One way is to use a special very slow TCP handler. Imagine a TCP stack that only lets through one 10 byte packet every 10 seconds. This wastes a connection slot on target machine. https://en.wikipedia.org/wiki/Slowloris_(computer_security) https://en.wikipedia.org/wiki/Slowloris_(computer_security)
AznHisoka 7y ago
I've been running many non-JS crawlers for the past few years, and there were a few pages that kept pushing the CPU load of my servers to a halt. When I dug into the source code, I saw that the HTML was a convoluted text of tables inside tables inside tables inside more tables, thus making it incredibly time-consuming + CPU-intensive for my DOM parser to parse (I was using Nokogiri, a Ruby gem at the time). Thus Cloudflare could be serving these types of "fake" pages to bad bots. They could also be doing things like serving fake streaming audio that never ends, or anything that might make it seem like the web page is just a huge page that needs time to load.

comex 7y ago

> Our bot detection breaks down into four large components: > - Identification of well known legitimate bots; What about non-well-known legitimate bots? If I run my own web crawler, am I at risk of falling into the tarpit (and having my IP address reported)?

avip 7y ago
What makes your crawler legitimate?
- AznHisoka 7y ago
  This builds up to a bigger question, with anti-competitive implications. What if you want to build a Google competitor? Is Google the only one that's allowed to crawl web pages now, and anyone else is automatically deemed a bad bot? Is CloudFlare the one that decides who can be a search engine, and who can't?
  - RcouF1uZ4gsC 7y ago
    >What if you want to build a Google competitor? Use CommonCrawl https://commoncrawl.org/ https://commoncrawl.org/ Then, after you can use that corpus and provide useful results and have people coming to your search site, then you can have a discussion with CloudFlare that you will be creating your own crawler.
    - 4gotunameagain 7y ago
      What if for whatever reason the innovation is on the crawler itself
      - kordlessagain 7y ago
        An AI crawler.
    - AznHisoka 7y ago
      Not sure how it is now, but when I looked into CommonCrawl, they had a significant delay in storing newer pages. It took at least a month before a web page made it into their index. Which makes creating any type of Google-like search engine very tough.

mfontani 7y ago

I've toyed with the idea of a tarpit service for badly behaved crawlers which just don't get other hints. I'm glad to see that if my services are behind cloudflare, I could just turn something on and let _them_ deal with it.

dclusin 7y ago
What is does it mean to tarpit someone? Throttle their connection?
- petre 7y ago
  Waste their time with a very slow server. Like OpenBSD’s spamd. Think one letter per second for text based protocols.
  - mfontani 7y ago
    Precisely that
- snowwolf 7y ago
  Something like this https://github.com/die-net/http-tarpit https://github.com/die-net/http-tarpit
- KineticLensman 7y ago
  To delay incoming connections https://en.wikipedia.org/wiki/Tarpit_(networking) https://en.wikipedia.org/wiki/Tarpit_(networking)
- judge2020 7y ago
  Cloudflare's specific methods are described in the article, it's likely some web crypto stuff or an extreme response that takes a long time to parse or handle.

ikeboy 7y ago

>Another trend we have seen is the increase of the combination of bots with botnets, particularly in the world of inventory hoarding bots. The motivation and willingness to spend for these bot operators is quite high. >The targets are goods of generally of limited supply and high in demand and in value. Think sneakers, concert tickets, airline seats, and popular short run Broadway musicals. Bot operators who are able to purchase those items at retail can charge massive premiums in aftermarket sales. When the operator identifies a target site, such as an ecommerce retailer, and a specific item, such as a new pair of sneakers going on sale, they can purchase time on the new Residential Proxy as a Service market to gain access to end user machines and (relatively) clean IPs from which to launch their attack. They then go on to spout some economic nonsense about how such bots are harmful. Actually, resellers make the market more efficient, and cloudflare is doing a disservice by lumping legitimate bots in with malicious ones like their credential stuffing example.

deadbunny 7y ago
How do bots make the market more efficient?
- ikeboy 7y ago
  By moving the goods to those willing to pay more for them, which increases consumer surplus, which means the market is more efficient. It's fairly basic economics.
  - comex 7y ago
    Increasing the price should reduce consumer surplus, not increase it. If the manufacturer of an item increases its price and thus their profit from it, then in principle the increased profit can drive them to manufacture more of the item, avoiding deadweight loss due to insufficient supply. However, that doesn’t apply if a reseller is the one making the profits! And in any case, some of the concrete examples cited (tickets to concerts and Broadway musicals) have an essentially fixed supply; it’s hard or impossible to simply increase production. Separately, moving goods toward those willing to pay more for them is to some extent equivalent to moving them toward those who care more about them, which could be said to improve efficiency in terms of how much personal happiness is gained per unit. However, it also moves the goods toward those who are simply wealthier. And there are other possible proxies for personal interest besides price, ones that aren’t as affected by wealth. One of them is a person’s willingness to research in advance when certain tickets will go on sale, and be online at that time to snag them before they sell out – in other words, the system that bots are used to subvert. This system has its own biases (favoring those who are organized, have more flexible schedules, or are simply in a more favorable timezone), but it‘s still considered to be reasonably fair, at least when it’s not being subverted.

tony 7y ago

Cloudflare, as a service, has it all. Nice UX, fast, free. Nice domain service if you transfer to them. Fast DNS management. DOS/bot mitigation. Caching. Quick SSL, and affordable upgrade options. 2FA via TOTP. With all the networking/domain stuff momentum build, it would be nice to be able to spin up servers for apps/db. Main language at cloudflare is golang? Rust? Any python over there? Request: Allow changing the super administrator for cloudflare account more easily. At least for early-stage accounts.

jgrahamc 7y ago
Lots of Go, lots of Rust, lots of Python. Also... Lua, PHP, C++.
mfontani 7y ago
Not yet U2F, which I'm hopeful they get real soon
- judge2020 7y ago
  https://twitter.com/eastdakota/status/1175522433643601920?s=21 https://twitter.com/eastdakota/status/1175522433643601920?s=...
judge2020 7y ago
I’d still consider the registrar service a “maturing product” at best, similar to how CF was in its early years. Enough time and it’ll be polished to where I can say it’s “production ready”.

thegeomaster 7y ago

Great! Let's add it to the list of obstacles to accessing content for us savages in small, poor, out-of-sight-out-of-mind countries: - Harrassment with an extra verification step every god damn time you log in into a website you've been a paying customer at for years because some rando risk model thinks you're a bad guy - Harrassment by reCAPTCHA to respond to US-centric image challenges when it decides it's up for some sadistic fun (Mark all images with a store front or street light? Sure, let me just Google how they are "supposed" to look, because I sure as hell ain't seeing any I'm familiar with here.) - Blanket IP range bans which serve you a default 503 error page and call it a day, with ripple effects throghout tons of unrelated websites. I always know when my ISP's IP range is added to some new blacklist. - Harrassment by Cloudflare "verifying my browser", presumably burning my CPU cycles so they can be sure that my browser, which has been hitting their IP ranges for years with not as much as a cookie wipe, has not suddenly turned into a bot - "Your name is invalid": no, your regex is - "Our fetishism for credit prevents us from accepting your debit card payment unless you submit a scan of your passport signed in blood" - (NEW) CloudFlare's anti-bot measures which will surely not misfire because no one thought to test their shiny new model on the traffic patterns of some culture with < 50 million Internet users Apologize for the off-topic rant, but you wouldn't believe how using the Internet for basic things has gotten difficult in the past few years over here (and from what I hear, in many other "forgotten" geographies). No one cares. Long live colonialism!

Meekro 7y ago
I'm sorry you have to put up with all of that, and I don't want to be part of the problem for folks like you. But here's the other side of the coin: I ran a small VPS hosting company for many years, and the "small, poor, out-of-sight-out-of-mind countries" are a huge source of fraud. Someone signing up from an IP in Latvia, for example, was virtually guaranteed to be paying with a stolen card that would later get disputed. Debit card payments can get disputed, too, and disputes can happen as much as 120 days after the initial charge. For comparison purposes, the dispute rate for US-origin payments (US card, US origin IP, no evidence of proxy use) was under 1%. If I couldn't do things like "submit a scan of your passport", I would probably just decline any cards from high-risk countries. What would you suggest I do instead?

andrerm 7y ago

This is Claudfare saying that now that they arbitrated X and everybody said "OK" they are moving from infrastructure into infra and arbitration. My question is, what if you are starting an alternative search engine or something legit? Edit: my point is, the rules than make a legit or not bot, crawler, scrapper etc. are not clear at all.