11 ms·
How to detect web bots?
- Minor49er 7y agoWhile interesting and well-intentioned, the advice in the article can cause issues. The agent isn't downloading CSS or images? Could be a blind user. Pages being downloaded in quick succession? Could be browser prefetch. Lack of mouse movements during navigation? It could be a user on a mobile device or screen reader. What if the bot is making individual requests from unique IP addresses? What if it's scraping many pages over time rather than a single smash-and-grab approach? The article admits that this is a hard thing to solve. In most cases, it's probably not very worthwhile to try to detect bots in the ways that are suggested. Focusing on patterns in the server logs to find out what's being targeted might be more beneficial. Then slap a login or captcha around anything valuable that bots shouldn't have access to
- hinkley 7y agoI joke that I only have two real skills. Asking questions and identifying good ideas. In the latter category I have heterogenous load balancing, segregated by traffic type. For example, if you have a read-mostly web application, often the most expensive parts are in the administrative functions. If they run on separate hardware, then a regression doesn’t affect everyone, and a client side bug or flood of traffic doesn’t automatically lock people out of the admin workflow. Similarly, you have server side rendering for web crawlers and anyone with JavaScript disabled for Angular and other interactive frameworks, which generally involves running a small quantity of specialized services on a few machines. Shunting all users who smell like a bot to a small set of machines doesn’t lock them out, but it does preserve the experience for everybody else.
- zzzcpan 7y agoI somewhat agree, but those can be good signals though, you just have to gather statistical data on them from known good users and combine with other signals. Like what are the chances for an agent to be used by a real person if it doesn't download images, lacks mouse movements and, for example, is coming from a datacenter? Obviously pretty low. The same way access patterns alone don't give you much, but are good against scrapers when combined with other signals. And putting captchas or logins on anything you have to protect from bots also "protects" from real people, most of them simply leave and don't bother with those thing.
- heavyset_go 7y ago> Like what are the chances for an agent to be used by a real person if it doesn't download images, lacks mouse movements and, for example, is coming from a datacenter? Obviously pretty low. Congrats, you just identified a visually impaired user with a screen reader who uses a VPN through AWS/GCP/etc.
- zzzcpan 7y agoAnd your point is?
- bryanrasmussen 7y agothat depending on your sites purpose and where your servers are located / where your main customer base is you might be opening yourself to extremely negative public relations by blocking this person as a bot, or even legal repercussions.
- zzzcpan 7y agoThis is completely irrelevant to the conversation about technical solutions. My example is just about probabilities and how to use them for bot detection.
- gregoryl 7y agoIf that logic is extended, the true technical solution is to turn the site off entirely.
- zzzcpan 7y agoWhat exactly in that logic in any way leads to turning off the site entirely? It's literally the opposite: narrowing down the bots by calculating probability of them not being real people using multiple signals and like Bayesian inference or something like that, it's pretty much a primitive machine learning. I'm surprised there are so many trolling comments here.
- forgotmypw 7y agoI use a keyboard-driven browser to reduce wrist strain, and I sometimes get tagged as a bot, probably for lack of mouse movement.
- bryanrasmussen 7y agobasically I've written bots for scraping governmental sites, and fashion sites. I've never yet had a problem (although the code on one fashion site looked like all classes were randomly generated with implied some attempt to keep scraping from working), so what I'm wondering is - what kinds of sites actually go through the trouble of tagging you as a bot, since you've got experience care to share?
- edoceo 7y agoTicketmaster
- MonkeyIsNull 7y agoGood response, I definitely never though of that before. Serious case where you don't want Bots buying stuff up.
- edoceo 7y agoAges ago I was one of those bots. Most fun I've had playing cat-and-mouse. Their team was actually quite good. And I learned that Perl can solve any problem.
- GrayTextIsTruth 7y agoHave you written about this? Sounds like an entertaining story.
- edoceo 7y agoI have not. But I will and will post to HN.
- madamelic 7y ago> it’s safe to assume that attackers are likely to lie on their fingerprints I hate this sentiment that every one who doesn't want to be tracked is a criminal. It's the digital version of "You don't have anything to hide if you aren't doing bad". --- I don't understand the fear about bots or scraping. As long as bots are behaving nicely (not slamming servers), they are just as much web citizens as humans. The web is about sharing of information and having an entire company about exterminating the viability of bots is horrifying.
- dragonwriter 7y ago> > it’s safe to assume that attackers are likely to lie on their fingerprints > I hate this sentiment that every one who doesn't want to be tracked is a criminal. The statement you are referring to is more accurately paraphrased as “everyone who is a criminal does not want to be tracked”, which is very different from “everyone who does not want to be tracked is a criminal.” > It's the digital version of "You don't have anything to hide if you aren't doing bad". For the reason noted above, the statement at issue is actually the digital version of “if you are doing something bad, you do have something to hide,” which is a near-universal truth (I mean, it not true if you have actual or practical immunity from any accountability for your wrongdoing, but otherwise it's true.)
- a1369209993 7y agoThe stamement, yes; the sentiment is "and therefore we can use not wanting to be tracked as probable cause for treating them as a criminal".
- dragonwriter 7y agoThis makes, and implies, no claims about “probable cause”, and isn't about criminal process where “probable cause” would even be relevant, it's about security policy where personal rights are not involved, only private discretionary benefits. It is often entirely appropriate in such a content to apply heuristics which are effective because they minimize false negatives even when they might have a high false positive rate.
- koboll 7y agoThe message I'm getting from this article is that headless Chrome should offer an 'undetectable' mode where its unique, fingerprintable globals are replaced with those from the headful version.
- hinkley 7y agoI work for a SaaS company that provides a domain per customer, and one of the consequences of this that never occurred to me before is that even polite crawlers and bots can thrash your servers if you have enough TLDs. Every few months someone discovers the correlation between our log messages and user agents and we rehash the same discussion about a particular badly behaved crawler that produces enough log noise that it distracts from identifying new regressions. I coordinated an effort to fix a problem with bad encoding of query parameters ages ago, but we still see broken ones from this one bot.
- DailyHN 7y agoBots have a PR problem.
- DailyHN 7y agoWhat about users of RPA software like UiPath, Blue Prism, etc.?
- gesman 7y agoWe detecting web bots by analyzing behavior of all web traffic and via clustering we finding entities that are behaving unusually similar to each other. This way we detect "clusters" or families of bots.
- madamelic 7y agoInteresting! I wrote something that leverages AWS Lambdas to get around rate limiting, this solution would've tagged me instantly.
- zzo38computer 7y agoThere are many false positives, though. It might believe it a bot but is wrong. Changing the number in the URL is something I commonly do. Additionally, I often use curl when I want to download a file (rather than using the browser's download manager).
- to1y 7y agoOut of interest, why do you prefer curl?
- zzo38computer 7y agoI just don't like the graphical file management.
- mobilemidget 7y ago"..(the OS and its version, as well as whether it is a VM.." Could anybody link me up with something to read on how to detect it is a VM based on TLS fingerprint?
- dylz 7y agoOne of my services routinely bans abusive hosts that try repeatedly ban evading, but we notice via TCP fingerprinting that their machine has started up under a minute ago each time it rotates. Non-passive: It's also possible to read a GPU name like "VMware SVGA" from JS, or watch for mismatched/wrong hardwareConcurrency.
- mobilemidget 7y agothanks, will be googling bit more on these terms and techniques
- wittyusername 7y agoMy favorite thing to do was to detect a bot and then show the bot a parallel dataset I'd developed of bogus information. Instead of hitting a block and working its way around it, competitors scraped a bunch of slightly warbled nonsense data. Have fun with that gang! Also made it super duper easy to see who'd try to steal our data.
- ianandrich 7y agoI really like this approach to handling bots.
- madamelic 7y agoThat's really, really funny. I would absolutely love to see that as someone who scrapes sites. It would make me smile immensely to see the trolling.
- cdoxsey 7y agoAlso known as a honeypot: https://en.wikipedia.org/wiki/Honeypot_(computing) https://en.wikipedia.org/wiki/Honeypot_(computing).
- Can_Not 7y agoActually, better terminology would be Honeytoken, Trap Street, and/or Fictitious Entry. https://en.m.wikipedia.org/wiki/Honeytoken https://en.m.wikipedia.org/wiki/Honeytoken https://en.m.wikipedia.org/wiki/Fictitious_entry https://en.m.wikipedia.org/wiki/Fictitious_entry https://en.m.wikipedia.org/wiki/Trap_street https://en.m.wikipedia.org/wiki/Trap_street
- folkhack 7y agoAs someone on the other side of the game: this is incredibly effective and can be near-impossible for me to figure out you're feeding me BS data. I have also done this as an anti-botting tactic myself though and it can get pretty complicated to setup properly unless you're doing the low-tech "feed these networks BS data" list.
- brunoTbear 7y agoAnswering this question was recently a 1B acquisition for F5 with their purchase of Shape Security. Will be interesting to see what happens to other industry players like PerimeterX. Distil was eaten by the corpse of Imperva, and I don't see Akamai making strong headway with Botman. Google is going after this too with reCAPTCHA. The HN reaction to that has been interesting. It's interesting to me how many comments in these threads talk about scraping as the issue with bots. Every sale I saw when I worked on this problem was related to credential stuffing. Seems the enterprise dollars are in the fraud space, but the HN sentiment is in scraping. Funny how disconnected the community here can be from what I saw first-hand as the "real" issue. Makes me wonder what other topics it gets wrong. Surely my area of expertise isn't special.
- Razengan 7y agoI wonder if they could just make a new browser standard which lets the server put the browser into a mode where the browser itself ensures there’s a live human interacting with the page/app, like using the webcam + OS facial recognition to ensure the server that it’s not a bot.
- fluxsauce 7y agoThis makes my skin crawl. I understand the intent, but this is morally wrong on many different levels. I hate credential stuffing and malicious activity as much as the next person, but I would never sacrifice user privacy like that. Also, you can spoof a webcam (but that's not the real problem).
- Razengan 7y agoIt would be better than the unpaid labor we have to do for Captcha.
- realusername 7y agoCaptcha don't stop bots anymore, they only stop humans. I don't know why websites are still using those.
- missblit 7y agoUnless it was closed source / DRM-y, the scraper developer would just compiler their browser from source and fake the detection code. Scrapers that know what they're doing could implement this in a day. There are already easier ways to detect scrapers that don't know what they're doing. And that's just the start of a long cat & mouse game that'd inevitably end up even more user-hostile and unaccessible than requiring a webcam to browse a website.
- fouc 7y agoI think bots should have full rights to access the internet, just like humans. Discriminating against bots just makes the internet worse for everyone. In the future, everyone will have personal bots and agents to help automate many things.
- deleted 7y ago[deleted]
- fouc 7y agoI'm actually being serious about this. Hopefully others are worried about the trend to discriminate against bots. If corporations can have personhood rights, why can't non-human agents?
- WilliamEdward 7y agoCorporations should not have human rights. Bots should not either but they should be allowed on the internet, no need to bring rights into it.
- deleted 7y ago[deleted]
- hombre_fatal 7y agoNot sure what 'full rights' mean here. My rights involve zero obligation to serve every request to my server. It doesn't make much sense to me for someone to have the 'right' to demand me to respond to them nor their Python loop.
- dominickreyes 7y agoI just launched a site. I have not mentioned it anywhere and there are no links to it on the internet, yet the logs are full of bots looking for vulnerabilities. Judging by what they are looking for, it seems I could eliminate half of them by if (location.contains('php')).
- jesterson 7y agoThose are kiddies and most of them are dumb but filtering more complex ones is not that easy. Don't worry about backlinks - they know you've registered a domain so already checking A records in domain DNS. They are much more clever that 10 years ago when we could expect 0 requests to site that was just set up.
- cpt1138 7y agoSince 1/21 I have identified 153,423 attempts to gain access to some servers I run. All from unique ip addresses. It's one thing to identify bots, it's quite another what do about them.
- throwaway4392 7y agoAt the very least you can use these techniques to know what not to do when scraping, and therefore scrape better.