9 ms·

The /unblock API from Browserless: dodging bot detection as a service

176 points by keepamovin 3y ago

adsthrowaway99 3y ago
For a while I worked in ads, and the specific team I was in put us in a position of being on both sides of this. We were tasked with monitoring advertisements for fraud, which meant we both had to catch bot traffic, but also we had to check ads for scams/phishing/etc. So on the one hand we were trying to plug holes in our own bot detection for botnets trying to drive up impression numbers to get website operators paid more, but on the other hand we were trying to bypass bot detecting on the part of phishing page authors (who would do things like display a totally plausible e-commerce page if they think the ad viewer is a bot, but a bank of america phishing page if they think it's a real user).
- Avamander 3y ago
  > other hand we were trying to bypass bot detecting on the part of phishing page authors An approach I recently saw gated the phishing page behind a Google Account login page. User that was logged in wouldn't even notice the brief redirect. Scanners would just get stuck. I hope they've patched it by now though...
  - yjftsjthsd-h 3y ago
    A coincidence of how I work with my browser profiles is that the browser (profile) that does most of my web browsing isn't logged into Google. It's nice to see that this has extra benefits:) Now if you'll excuse me, I need to go move my entire computing environment into a VM so that viruses will think they're being analyzed and refuse to run... (Kidding but also not)
    - simcop2387 3y ago
      Check out qubes os, run each app in it's own vm
- spondylosaurus 3y ago
  Ha, I think we may have worked at the same company (or at least at competing companies)....
nhggfu 3y ago
someone got tweak we can use for puppeteer to achieve this effect? [kinda LOL @ asking people who SCRAPE DATA for free to pay for an API, no?]
- ohthatsnotright 3y ago
  You can use Firefox nightly and create a webextension using their internal APIs. Gives you access to injecting non-synthetic or faked events (they look indistinguishable since they are emitted in the same code path that the real input device uses) and scraping the DOM with the added advantage of not exposing the automation flags that Firefox has when you use Puppeteer and the like in headless, automated mode. edit: Firefox Nightly because it gives you much deeper access to Firefox internals.
  - nhggfu 3y ago
    interesting. would love to see some examples / code
    - ohthatsnotright 3y ago
      I can't give examples (I did the work for a company so I don't own it) but starting at [0] [1] should get you to the internals. It definitely carries some risk of Firefox Nightly just randomly breaking when it updates but this solution got us beyond quite a number of bot detectors so if you're willing to risk Firefox Nightly it's possible. We didn't need nearly the flexibility something like Puppeteer has since we were injecting rrweb into the target page to record the DOM mutations and sending those to other clients. That will give you a good start on where to even begin looking. web-ext is insanely handy for launching Firefox with a WebExtension already installed (but I did a lot of other work around this to productize the offering). [0] https://firefox-source-docs.mozilla.org/toolkit/components/extensions/webextensions/index.html https://firefox-source-docs.mozilla.org/toolkit/components/e... [1] https://firefox-source-docs.mozilla.org/overview/gecko.html https://firefox-source-docs.mozilla.org/overview/gecko.html
      - 3y ago
mmcclure 3y ago
I can't help but feel a little conflicted about things like this in the current climate. On one hand, as a developer trying to just get some updated stuff into a spreadsheet, this looks extremely useful. On the other hand, there are a lot of people right now that want to keep stuff accessible to humans and not have it scraped for models. I know the lid's completely off the box on that front so it's probably useless to fight it, but seeing products explicitly designed to circumvent bot prevention feels kinda bad.
- imathrowaway 3y ago
  Agreed, we've seen a lot of governing bodies _now_ start scraping to fight scam sites and "fake" sites, so there's a lot of legitimate use-cases popping up. It's not necessarily just about gathering e-commerce data or backfilling lack of an API anymore. Source: Founder@browserless.io
  - Matheus28 3y ago
    So how will you ensure your service is only used for "good"?
    - 67j67j67n 3y ago
      no services remain "good" so why ask the question. it gets ignored or answered but then a decade or two down the line the philosophy gets scrapped, a story as old as time.
      - chankstein38 3y ago
        I still don't understand why Google publicly removed "Do No Evil" from their company. It's one thing to not follow it but to actively remove it feels weird. Like they wanted to say "HEY Y'ALL GET READY FOR EVIL!"
        ziddoap 3y ago
        You can go to https://abc.xyz/investor/google-code-of-conduct/ https://abc.xyz/investor/google-code-of-conduct/ and ctrl+f "evil", and you'll see it's still there.
pkiv 3y ago
In the end, the best way to avoid being blocked is to be a good actor. All of these hacks won't stop someone who's determined to prevent access (ie: LinkedIn). That's actually one of the reasons why I started https://browserbase.com/ https://browserbase.com/. Maintaining headless browser infrastructure can be such a pain. I've spent a lot of time managing headless chrome fleets at scale, so happy to answer any questions.
- Etheryte 3y ago
  If I understand correctly, a lot of the issues you can run into with regards to blocking come from the fact that you're using a headless browser. Past a certain point, wouldn't it be less work to use a regular browser and drive with Selenium or similar solutions? Or does that not address the kind of problems you're facing?
  - djbusby 3y ago
    I created a dedicated chrome profile (--user-data-dir) signed in to a few sites and then drive it, with visible window from scripts. Does all my crawling, it goes very slow, it's never trigger the bot detectors.
  - pkiv 3y ago
    The newest version of headless chrome actually runs the same code as a "regular browser": https://developer.chrome.com/docs/chromium/new-headless https://developer.chrome.com/docs/chromium/new-headless
  - tzs 3y ago
    I used to semi-automate access to some sites by using Selenium with a non-headless browser. These were sites where there were just one or two pages where I wanted some automation to fill out a form or scrape some data, and they frequently made changes to the home page that made it hard to automate navigating from the home page to the pages I wanted to automate. The idea was to have a script use Selenium to launch non-headless Chrome and then wait: driver = Chrome() driver.get("https://example.org") input("Press enter when ready") I could then manually deal with logging in, answering any CAPTCHA that came up, and navigate to the page I wanted to run my automation. Then I could press "enter" in my terminal and my script would continue. That used to work fine, but then on sites using Cloudflare's CAPTCHA it stopped working. Solving the CAPTCHA would just result in another CAPTCHA. I tried an alternative Selenium Chrome driver that was supposed to be more stealthy, and tried setting various flags that were supposed to make it so JavaScript could not tell that Selenium was there, and those worked for a while, but then they stopped working. The results were similar using Selenium with Firefox. I also tried Puppeteer, with Chromium and Firefox, and they too could not get past the CAPTCHA loops. I then tried Playwright. With Chromium and Webkit that got the CAPTCHA loops. With Firefox it actually worked. I didn't even see the CAPTCHA. The non-interactive check for not being a bot passed. Still, the whole approach seems fragile. I don't know if Firefox/Playwright working was due to some fundamental difference between Firefox and the others or just Cloudflare having not yet gotten around to dealing with it.
Avamander 3y ago
This sounds a lot like Abuse as a Service. Trying to defend against malicious bots is tedious at best, impossible at the worst. I don't really see how lowering the bar would be a net positive. People will start using methods that will cause more collateral damage and just reduce user freedoms. My guess is that this only increases the push towards attestation and attestation-like approaches. Login walls and PAT/Privacy Pass are just the start.
- Spivak 3y ago
  Damned if you do, damned if you don't. The only way to mitigate the attestation future will be legislation so you might as well do while it lasts. If you can't bypass bot detection now because in response they'll make the bot detection harder then there's really no winning is there?
  - Avamander 3y ago
    Even if you or I might find attestation against user freedoms nobody is going to legislate attestation away. The alternatives to the vast majority are way worse.
    - notpushkin 3y ago
      Could you elaborate on the alternatives bit?
      - Avamander 3y ago
        If it's too difficult to guarantee end-users' devices are legitimate using other means then the alternative would just be more invasive fingerprinting and login/paywalls. In addition to that any access restrictions would probably end up being less precise and thus cause more collateral damage. Which would also mean an increased amount of false negatives, leading to an increase in spammy content.
        hooverd 3y ago
unnouinceput 3y ago
The never-ending game of walls and ladders
jastingo 3y ago
What are the legitimate (i.e. legal) use cases for a product such as this? I agree with another comment that called this "Abuse as a Service". It seems to me this product's design goal is nothing more than to circumvent measures site owners take to prevent abuse of their site and run a sustainable business.
- mrmanner 3y ago
  > What are the legitimate (i.e. legal) use cases for a product such as this? Data portability! Tools like this can be used to allow individuals to export their data from hostile web services trying to hold it hostage. Legal in the EU, with GDPR.
- BoorishBears 3y ago
  I used their previously available bot detection defeat to add an import feature to my website: Users could link to their creation on another site and my site would scrape the publicly available content so they wouldn't need to re-enter all their data I've used their product many times actually, and I'm shocked on Hacker News of all places no one's thinking of anything besides abuse. How often is it useful to get information from a webpage and apply it in a new context? Then think of how often said webpage is behind a Cloudflare bot detector.
  - daemonw 3y ago
    If it's the user's data, then under GDPR the other site is obligated to provide a way for them to download/transfer it, specifically with this use case in mind. They are completely in the right to block you though, you're not the owner of that data, you might be breaking their TOS.
    - mrmanner 3y ago
      “ In exercising his or her right to data portability pursuant to paragraph 1, the data subject shall have the right to have the personal data transmitted directly from one controller to another, where technically feasible.” They’re not necessarily in the right to block you, if you’re the data subject or acting on their behalf.
LewisJEllis 3y ago
I used to work for a bot mitigation vendor 8-10 years ago, researching / implementing signals for this cat and mouse game. This will get you past some very mundane bot detections, but really this is like, the very first baby step of a long rabbit hole. The people who are taking this game seriously are 5-10 years ahead of this step. Good luck ¯\_(ツ)_/¯
- danpalmer 3y ago
  Yeah that’s my reading. No way this is passing Akamai bot detection. There are lots of signals like timings, user tapping and scrolling behaviour, signed sessions cookies that represent browsing flows which may be legitimate or not. And that’s all assuming you’re on a good looking IP. To do this you need a large supply of residential IPs which then leads to the dodgy underworld of botnets. I’d be surprised if this works for anything but the most basic bot protection, this is an advanced space. If it does work for those cases, they should be either keeping it quiet and making bank, or boasting about having a secret sauce, not basic stuff like this. Edit: for apps, Akamai provides an SDK that uses things like your motion data to create a signature that suggests that you're a real user. This signature is either injected into API requests or into a webview session. I'm sure it's crackable if you dedicate significant reverse engineering resources to it, but then you've got to crack every version, crack every other implementation from other companies, etc. Non-starter.
  - PathfinderBot 3y ago
    I'm just an outsider, but I wonder if these sort of bot-blocker-bypass services can be done by employing people to go to those pages manually.
    - danpalmer 3y ago
      Sure, but the point of the services is to be cheaper, faster, or more accurate than doing this.
  - dns_snek 3y ago
    When it comes to "motion data", are you referring to the client's movement around the website, i.e. which URLs they access and in which order? Their taps, clicks, scroll events, and other inputs?
NorwegianDude 3y ago
I've been fighting tailor made bots for over 15 years, and I can't say I'm a fan of people trying to circumvent it. In some cases it might not matter, but in others it can actually ruin a lot of things. Let's say I have a few 100 Gbps connections that is mostly idle, is it fine if I direct them at browserless? No? Exactly, that kind of traffic is not wanted.
- codedokode 3y ago
  Browserless could implement limits on their side, for example, allow only 1 parallel request to IP/domain. This way there will be no significant load, but the data will be scraped.
  - thrwwycbr 3y ago
    As if IPs matter in the golden age of IoT botnets and residential app malware. You can't even block the amount of subnets that's coming for you in a DDoS attack, thinking a human is able to keep up with something like this is pretty blindsighted and naive. The differentiation of network protocols and relay attacks alone is way too slow to be mitigated in most systems.
    - meowface 3y ago
      They're talking about rate limiting based on destination, not source.
- iamacyborg 3y ago
  It’s all fun and games until you get a taste of your own medicine.
deleted 3y ago
[deleted]
reactordev 3y ago
I'm sorry if this offends you, but it's companies like you that I despise. WAFs are a good thing. Your bot is not authorized. We'll be blocking all IP ciders from you, your referrers, and your service. Thanks.
- yard2010 3y ago
  No. Long live the piracy. Good bye and thanks for the fish.
internetter 3y ago
90% of the problems solved with puppeteer could be instead solved by replaying a few API requests. Cheaper, less wasteful, faster, in some cases, harder to detect (WAF is rarely deployed on APIs because there are much less heuristics)
- leptons 3y ago
  APIs change all the time. I work with plenty of undocumented APIs from a service we actually pay for, and it's a hassle of playing whack-a-mole when something changes without notice.
  - internetter 3y ago
    I do to. Almost every project I've worked on involves an undocumented API. It's still cheaper to devote developer time to reacting to changes than paying to run headless chrome and evade WAFs
    - leptons 3y ago
      But you can't fully detect the changes without the front-end. When an API changes, so does the official front-end that is interacting with the API. We need to track changes to the HTTP API requests coming from the front-end. You can't detect those changes just by hitting the APIs directly. If the official front-end changes how it sends data to the back-end, we need to change to match it. We also need to detect and react to UI layout changes, and the headless browser is the only real way to do that.
iambateman 3y ago
A lot of negativity about scraped data, in comments…but both OpenAI and Google rely on scraping as their foundation for…everything. If smaller indie hackers are going to build useful competitors, they’re going to need to scrape. I understand that a lot of scraping is from spammers, but not all.
- bdcravens 3y ago
  > If smaller indie hackers are going to build useful competitors Any useful competitor will have to scale well beyond the point of indie hacker, where someone is (eventually) getting rich. That doesn't make for a great argument for bypassing consent.
  - troyvit 3y ago
    Depends on how you describe "useful." Kagi is a profitable search company that satisfies its customers while competing directly with Google. They aren't useful in the sense that they're taking marketshare from Google, but they are useful to their user base. I hope they continue being able to scrape sites with the same freedom as other search engines. Edit: As mmcclure points out Kagi actually doesn't index much. So that's a bad example that might invalidate my point.
    - mmcclure 3y ago
      I'm a happy Kagi user, but they might be somewhat a bad example. I suspect their actual indexing work is pretty limited, and they only seem to claim that it's for indexing "niche" stuff. > Our data includes anonymized API calls to traditional search indexes like Google, Yandex, Mojeek and Brave, specialized search engines like Marginalia, and sources of vertical information like Apple, Wikipedia, Open Meteo, Yelp, TripAdvisor and other APIs. Typically every search query on Kagi will call a number of different sources at the same time, all with the purpose of bringing the best possible search results to the user. source: https://help.kagi.com/kagi/search-details/search-sources.html https://help.kagi.com/kagi/search-details/search-sources.htm...
silentsanctuary 3y ago
I want to try Browserless but it doesn’t appear to be possible to do so without signing up for a free trial of a $200/month plan - have they especially hidden this or am I just blind?
- pkiv 3y ago
  They recently increased their pricing quite a bit. We're looking to offer a much more affordable pay-as-you-go pricing model at https://browserbase.com/ https://browserbase.com/ Feel free to shoot me an email if you're interested in trying it out! paul@browserbase.com
generalizations 3y ago
I'm surprised that puppeteer uses that default window size. Last time I did a scraping project, I assumed randomizing the window size would be table stakes.
- darepublic 3y ago
  Puppeteer is not created for scraping per se right. I mean it could be, but isn't puppeteer created/maintained by google? So it's not meant to fly under the radar and avoid detection, it's just a naive tool for automating websites that would never try to block it.
darepublic 3y ago
Personally I see a legit use of this as avoiding detection with small amounts of traffic. I just want a browser bot to do things in my name, and not DDOS, but just spare me from having to personally go and tap my phone screen.
Exuma 3y ago
Lol I put in a site protected by cloudflare and get a 'you have been blocked'. great service
hipadev23 3y ago
I'm thoroughly confused on their pricing model. What is a unit? Am I reading this correctly that $500/mo comes with 25GB? So $20 per GB transferred? Which we're talking about running a real browser downloading all the bloat of a target website, 25GB doesn't go very far. Is this just absurdly expensive?
simple10 3y ago
These two websites used to be useful for testing bots. They may be several years old at this point. https://bot.sannysoft.com/ https://bot.sannysoft.com/ https://antoinevastel.com/bots/datadome https://antoinevastel.com/bots/datadome The first link shows the tests and is the most educational. Does anyone know of more recent tools that show why bot detection was triggered? Versus just testing against Cloudflare bot detection and getting pass/fail.
- simple10 3y ago
  Just found two more recent bot detectors that show the tests run: https://www.browserscan.net/en/bot-detection https://www.browserscan.net/en/bot-detection https://bot.incolumitas.com/ https://bot.incolumitas.com/
smsm42 3y ago
So basically is this Evil as a Service, or is there a legitimate case for this that is not identical to "stress testing is a legitimate case for me selling botnet DDos"?
- leptons 3y ago
  IFTTT is a legitimate use. There are an unlimited number of valid non-evil use cases for automated tasks looking at websites and doing stuff based on the data scraped.
  - smsm42 3y ago
    Not sure how IFTTT is a legitimate use of bot detection evasion. I've been using IFTTT for many years and never needed that.
    - leptons 3y ago
      IFTTT is a legitimate use of scraping public data from websites. But forget about IFTTT, it was just used as an example. It's just an example of automation looking at public website content and then taking some action. You're too focused on IFTTT and not the problem with blockading publicly accessible data. For example, Zillow goes to great lengths to restrict bots, but I want to get updates if a property changes somehow on Zillow. It's publicly accessible data if you're not using a bot. "IFTTT" is just an example of a service that automates these things. I can write my own automation service, but if I'm using a bot, Zillow will block me (I've actually done this before).
      - smsm42 3y ago
        Doesn't Zillow have an API here? https://www.zillowgroup.com/developers/ https://www.zillowgroup.com/developers/
bradgessler 3y ago
I'm surprised Cloudflare hasn't tried to create a market for this yet. They could charge some sort of fee between the scrapers and the scrapees and create additional revenue streams for sites being scraped. Hell they could even give tools to sites that lets them manage who can scrape them and who can't. I'm not saying I like this, I'm just saying I'm surprised it hasn't happened yet.
- nullpt_rs 3y ago
  They have a similar feature labelled "Friendly Bots" https://blog.cloudflare.com/friendly-bots/ https://blog.cloudflare.com/friendly-bots/
dreamcompiler 3y ago
Cloudflare's WAF breaks the web. The web should be open, anonymous, and automatable on behalf of users. Any web service that doesn't like this idea should not get my business. Period. What we really need is a browser that absolutely cannot access sites "protected" by Cloudflare, and this browser needs to become standard. Sites that put themselves behind Cloudflare need to pay a price for their hostility, and that price is the loss of our business. If I want to program a bot to get around the horrendous lossage of the terrible UI and multi-megabyte bloat you have created by farming out your web server programming to the lowest bidder, I have the right to do so. In turn my bot will respect robots.txt and will throttle itself appropriately. This is the deal you signed up for when you built a web server. If you don't like it, GTF off the web. And take your attestation and WAF with you.
- jfil 3y ago
  I tend to agree. I'm working on a web scraping project that allows an ordinary person to spot misbehaviour by well-resourced corporations. Cloudflare is absolutely getting in the way, and helping the bigger guys escape accountability by making them hard to track.
cebert 3y ago
It’s not too difficult to run playwright in a container or from an AWS Lambda with a layer. What’s the advantage to Browserless vs. rolling your own?
hamoodhabibi 3y ago
im not sure what the legal liabilities are for offering this type of service but couldn't somebody potentially use this to scrape a highly litigious website and end up bringing heat to browserless's company and its founders I wonder "we just provide the tool" defence no longer works in court like there's even a 7 day trial opening this up to anyone with bad intentions

176 points by keepamovin 3y ago

For a while I worked in ads, and the specific team I was in put us in a position of being on both sides of this. We were tasked with monitoring advertisements for fraud, which meant we both had to catch bot traffic, but also we had to check ads for scams/phishing/etc. So on the one hand we were trying to plug holes in our own bot detection for botnets trying to drive up impression numbers to get website operators paid more, but on the other hand we were trying to bypass bot detecting on the part of phishing page authors (who would do things like display a totally plausible e-commerce page if they think the ad viewer is a bot, but a bank of america phishing page if they think it's a real user).

Avamander 3y ago
> other hand we were trying to bypass bot detecting on the part of phishing page authors An approach I recently saw gated the phishing page behind a Google Account login page. User that was logged in wouldn't even notice the brief redirect. Scanners would just get stuck. I hope they've patched it by now though...
- yjftsjthsd-h 3y ago
  A coincidence of how I work with my browser profiles is that the browser (profile) that does most of my web browsing isn't logged into Google. It's nice to see that this has extra benefits:) Now if you'll excuse me, I need to go move my entire computing environment into a VM so that viruses will think they're being analyzed and refuse to run... (Kidding but also not)
  - simcop2387 3y ago
    Check out qubes os, run each app in it's own vm
spondylosaurus 3y ago
Ha, I think we may have worked at the same company (or at least at competing companies)....

nhggfu 3y ago

someone got tweak we can use for puppeteer to achieve this effect? [kinda LOL @ asking people who SCRAPE DATA for free to pay for an API, no?]

ohthatsnotright 3y ago
You can use Firefox nightly and create a webextension using their internal APIs. Gives you access to injecting non-synthetic or faked events (they look indistinguishable since they are emitted in the same code path that the real input device uses) and scraping the DOM with the added advantage of not exposing the automation flags that Firefox has when you use Puppeteer and the like in headless, automated mode. edit: Firefox Nightly because it gives you much deeper access to Firefox internals.
- nhggfu 3y ago
  interesting. would love to see some examples / code
  - ohthatsnotright 3y ago
    I can't give examples (I did the work for a company so I don't own it) but starting at [0] [1] should get you to the internals. It definitely carries some risk of Firefox Nightly just randomly breaking when it updates but this solution got us beyond quite a number of bot detectors so if you're willing to risk Firefox Nightly it's possible. We didn't need nearly the flexibility something like Puppeteer has since we were injecting rrweb into the target page to record the DOM mutations and sending those to other clients. That will give you a good start on where to even begin looking. web-ext is insanely handy for launching Firefox with a WebExtension already installed (but I did a lot of other work around this to productize the offering). [0] https://firefox-source-docs.mozilla.org/toolkit/components/extensions/webextensions/index.html https://firefox-source-docs.mozilla.org/toolkit/components/e... [1] https://firefox-source-docs.mozilla.org/overview/gecko.html https://firefox-source-docs.mozilla.org/overview/gecko.html
    - 3y ago

mmcclure 3y ago

I can't help but feel a little conflicted about things like this in the current climate. On one hand, as a developer trying to just get some updated stuff into a spreadsheet, this looks extremely useful. On the other hand, there are a lot of people right now that want to keep stuff accessible to humans and not have it scraped for models. I know the lid's completely off the box on that front so it's probably useless to fight it, but seeing products explicitly designed to circumvent bot prevention feels kinda bad.

imathrowaway 3y ago
Agreed, we've seen a lot of governing bodies _now_ start scraping to fight scam sites and "fake" sites, so there's a lot of legitimate use-cases popping up. It's not necessarily just about gathering e-commerce data or backfilling lack of an API anymore. Source: Founder@browserless.io
- Matheus28 3y ago
  So how will you ensure your service is only used for "good"?
  - 67j67j67n 3y ago
    no services remain "good" so why ask the question. it gets ignored or answered but then a decade or two down the line the philosophy gets scrapped, a story as old as time.
    - chankstein38 3y ago
      I still don't understand why Google publicly removed "Do No Evil" from their company. It's one thing to not follow it but to actively remove it feels weird. Like they wanted to say "HEY Y'ALL GET READY FOR EVIL!"
      - ziddoap 3y ago
        You can go to https://abc.xyz/investor/google-code-of-conduct/ https://abc.xyz/investor/google-code-of-conduct/ and ctrl+f "evil", and you'll see it's still there.

pkiv 3y ago

In the end, the best way to avoid being blocked is to be a good actor. All of these hacks won't stop someone who's determined to prevent access (ie: LinkedIn). That's actually one of the reasons why I started https://browserbase.com/ https://browserbase.com/. Maintaining headless browser infrastructure can be such a pain. I've spent a lot of time managing headless chrome fleets at scale, so happy to answer any questions.

Etheryte 3y ago
If I understand correctly, a lot of the issues you can run into with regards to blocking come from the fact that you're using a headless browser. Past a certain point, wouldn't it be less work to use a regular browser and drive with Selenium or similar solutions? Or does that not address the kind of problems you're facing?
- djbusby 3y ago
  I created a dedicated chrome profile (--user-data-dir) signed in to a few sites and then drive it, with visible window from scripts. Does all my crawling, it goes very slow, it's never trigger the bot detectors.
- pkiv 3y ago
  The newest version of headless chrome actually runs the same code as a "regular browser": https://developer.chrome.com/docs/chromium/new-headless https://developer.chrome.com/docs/chromium/new-headless
- tzs 3y ago
  I used to semi-automate access to some sites by using Selenium with a non-headless browser. These were sites where there were just one or two pages where I wanted some automation to fill out a form or scrape some data, and they frequently made changes to the home page that made it hard to automate navigating from the home page to the pages I wanted to automate. The idea was to have a script use Selenium to launch non-headless Chrome and then wait: driver = Chrome() driver.get("https://example.org") input("Press enter when ready") I could then manually deal with logging in, answering any CAPTCHA that came up, and navigate to the page I wanted to run my automation. Then I could press "enter" in my terminal and my script would continue. That used to work fine, but then on sites using Cloudflare's CAPTCHA it stopped working. Solving the CAPTCHA would just result in another CAPTCHA. I tried an alternative Selenium Chrome driver that was supposed to be more stealthy, and tried setting various flags that were supposed to make it so JavaScript could not tell that Selenium was there, and those worked for a while, but then they stopped working. The results were similar using Selenium with Firefox. I also tried Puppeteer, with Chromium and Firefox, and they too could not get past the CAPTCHA loops. I then tried Playwright. With Chromium and Webkit that got the CAPTCHA loops. With Firefox it actually worked. I didn't even see the CAPTCHA. The non-interactive check for not being a bot passed. Still, the whole approach seems fragile. I don't know if Firefox/Playwright working was due to some fundamental difference between Firefox and the others or just Cloudflare having not yet gotten around to dealing with it.

Avamander 3y ago

This sounds a lot like Abuse as a Service. Trying to defend against malicious bots is tedious at best, impossible at the worst. I don't really see how lowering the bar would be a net positive. People will start using methods that will cause more collateral damage and just reduce user freedoms. My guess is that this only increases the push towards attestation and attestation-like approaches. Login walls and PAT/Privacy Pass are just the start.

Spivak 3y ago
Damned if you do, damned if you don't. The only way to mitigate the attestation future will be legislation so you might as well do while it lasts. If you can't bypass bot detection now because in response they'll make the bot detection harder then there's really no winning is there?
- Avamander 3y ago
  Even if you or I might find attestation against user freedoms nobody is going to legislate attestation away. The alternatives to the vast majority are way worse.
  - notpushkin 3y ago
    Could you elaborate on the alternatives bit?
    - Avamander 3y ago
      If it's too difficult to guarantee end-users' devices are legitimate using other means then the alternative would just be more invasive fingerprinting and login/paywalls. In addition to that any access restrictions would probably end up being less precise and thus cause more collateral damage. Which would also mean an increased amount of false negatives, leading to an increase in spammy content.
      - hooverd 3y ago

unnouinceput 3y ago

The never-ending game of walls and ladders

jastingo 3y ago

What are the legitimate (i.e. legal) use cases for a product such as this? I agree with another comment that called this "Abuse as a Service". It seems to me this product's design goal is nothing more than to circumvent measures site owners take to prevent abuse of their site and run a sustainable business.

mrmanner 3y ago
> What are the legitimate (i.e. legal) use cases for a product such as this? Data portability! Tools like this can be used to allow individuals to export their data from hostile web services trying to hold it hostage. Legal in the EU, with GDPR.
BoorishBears 3y ago
I used their previously available bot detection defeat to add an import feature to my website: Users could link to their creation on another site and my site would scrape the publicly available content so they wouldn't need to re-enter all their data I've used their product many times actually, and I'm shocked on Hacker News of all places no one's thinking of anything besides abuse. How often is it useful to get information from a webpage and apply it in a new context? Then think of how often said webpage is behind a Cloudflare bot detector.
- daemonw 3y ago
  If it's the user's data, then under GDPR the other site is obligated to provide a way for them to download/transfer it, specifically with this use case in mind. They are completely in the right to block you though, you're not the owner of that data, you might be breaking their TOS.
  - mrmanner 3y ago
    “ In exercising his or her right to data portability pursuant to paragraph 1, the data subject shall have the right to have the personal data transmitted directly from one controller to another, where technically feasible.” They’re not necessarily in the right to block you, if you’re the data subject or acting on their behalf.

LewisJEllis 3y ago

I used to work for a bot mitigation vendor 8-10 years ago, researching / implementing signals for this cat and mouse game. This will get you past some very mundane bot detections, but really this is like, the very first baby step of a long rabbit hole. The people who are taking this game seriously are 5-10 years ahead of this step. Good luck ¯\_(ツ)_/¯

danpalmer 3y ago
Yeah that’s my reading. No way this is passing Akamai bot detection. There are lots of signals like timings, user tapping and scrolling behaviour, signed sessions cookies that represent browsing flows which may be legitimate or not. And that’s all assuming you’re on a good looking IP. To do this you need a large supply of residential IPs which then leads to the dodgy underworld of botnets. I’d be surprised if this works for anything but the most basic bot protection, this is an advanced space. If it does work for those cases, they should be either keeping it quiet and making bank, or boasting about having a secret sauce, not basic stuff like this. Edit: for apps, Akamai provides an SDK that uses things like your motion data to create a signature that suggests that you're a real user. This signature is either injected into API requests or into a webview session. I'm sure it's crackable if you dedicate significant reverse engineering resources to it, but then you've got to crack every version, crack every other implementation from other companies, etc. Non-starter.
- PathfinderBot 3y ago
  I'm just an outsider, but I wonder if these sort of bot-blocker-bypass services can be done by employing people to go to those pages manually.
  - danpalmer 3y ago
    Sure, but the point of the services is to be cheaper, faster, or more accurate than doing this.
- dns_snek 3y ago
  When it comes to "motion data", are you referring to the client's movement around the website, i.e. which URLs they access and in which order? Their taps, clicks, scroll events, and other inputs?

NorwegianDude 3y ago

I've been fighting tailor made bots for over 15 years, and I can't say I'm a fan of people trying to circumvent it. In some cases it might not matter, but in others it can actually ruin a lot of things. Let's say I have a few 100 Gbps connections that is mostly idle, is it fine if I direct them at browserless? No? Exactly, that kind of traffic is not wanted.

codedokode 3y ago
Browserless could implement limits on their side, for example, allow only 1 parallel request to IP/domain. This way there will be no significant load, but the data will be scraped.
- thrwwycbr 3y ago
  As if IPs matter in the golden age of IoT botnets and residential app malware. You can't even block the amount of subnets that's coming for you in a DDoS attack, thinking a human is able to keep up with something like this is pretty blindsighted and naive. The differentiation of network protocols and relay attacks alone is way too slow to be mitigated in most systems.
  - meowface 3y ago
    They're talking about rate limiting based on destination, not source.
iamacyborg 3y ago
It’s all fun and games until you get a taste of your own medicine.

deleted 3y ago

[deleted]

reactordev 3y ago

I'm sorry if this offends you, but it's companies like you that I despise. WAFs are a good thing. Your bot is not authorized. We'll be blocking all IP ciders from you, your referrers, and your service. Thanks.

yard2010 3y ago
No. Long live the piracy. Good bye and thanks for the fish.

internetter 3y ago

90% of the problems solved with puppeteer could be instead solved by replaying a few API requests. Cheaper, less wasteful, faster, in some cases, harder to detect (WAF is rarely deployed on APIs because there are much less heuristics)

leptons 3y ago
APIs change all the time. I work with plenty of undocumented APIs from a service we actually pay for, and it's a hassle of playing whack-a-mole when something changes without notice.
- internetter 3y ago
  I do to. Almost every project I've worked on involves an undocumented API. It's still cheaper to devote developer time to reacting to changes than paying to run headless chrome and evade WAFs
  - leptons 3y ago
    But you can't fully detect the changes without the front-end. When an API changes, so does the official front-end that is interacting with the API. We need to track changes to the HTTP API requests coming from the front-end. You can't detect those changes just by hitting the APIs directly. If the official front-end changes how it sends data to the back-end, we need to change to match it. We also need to detect and react to UI layout changes, and the headless browser is the only real way to do that.

iambateman 3y ago

A lot of negativity about scraped data, in comments…but both OpenAI and Google rely on scraping as their foundation for…everything. If smaller indie hackers are going to build useful competitors, they’re going to need to scrape. I understand that a lot of scraping is from spammers, but not all.

bdcravens 3y ago
> If smaller indie hackers are going to build useful competitors Any useful competitor will have to scale well beyond the point of indie hacker, where someone is (eventually) getting rich. That doesn't make for a great argument for bypassing consent.
- troyvit 3y ago
  Depends on how you describe "useful." Kagi is a profitable search company that satisfies its customers while competing directly with Google. They aren't useful in the sense that they're taking marketshare from Google, but they are useful to their user base. I hope they continue being able to scrape sites with the same freedom as other search engines. Edit: As mmcclure points out Kagi actually doesn't index much. So that's a bad example that might invalidate my point.
  - mmcclure 3y ago
    I'm a happy Kagi user, but they might be somewhat a bad example. I suspect their actual indexing work is pretty limited, and they only seem to claim that it's for indexing "niche" stuff. > Our data includes anonymized API calls to traditional search indexes like Google, Yandex, Mojeek and Brave, specialized search engines like Marginalia, and sources of vertical information like Apple, Wikipedia, Open Meteo, Yelp, TripAdvisor and other APIs. Typically every search query on Kagi will call a number of different sources at the same time, all with the purpose of bringing the best possible search results to the user. source: https://help.kagi.com/kagi/search-details/search-sources.html https://help.kagi.com/kagi/search-details/search-sources.htm...

silentsanctuary 3y ago

I want to try Browserless but it doesn’t appear to be possible to do so without signing up for a free trial of a $200/month plan - have they especially hidden this or am I just blind?

pkiv 3y ago
They recently increased their pricing quite a bit. We're looking to offer a much more affordable pay-as-you-go pricing model at https://browserbase.com/ https://browserbase.com/ Feel free to shoot me an email if you're interested in trying it out! paul@browserbase.com

generalizations 3y ago

I'm surprised that puppeteer uses that default window size. Last time I did a scraping project, I assumed randomizing the window size would be table stakes.

darepublic 3y ago
Puppeteer is not created for scraping per se right. I mean it could be, but isn't puppeteer created/maintained by google? So it's not meant to fly under the radar and avoid detection, it's just a naive tool for automating websites that would never try to block it.

darepublic 3y ago

Personally I see a legit use of this as avoiding detection with small amounts of traffic. I just want a browser bot to do things in my name, and not DDOS, but just spare me from having to personally go and tap my phone screen.

Exuma 3y ago

Lol I put in a site protected by cloudflare and get a 'you have been blocked'. great service

hipadev23 3y ago

I'm thoroughly confused on their pricing model. What is a unit? Am I reading this correctly that $500/mo comes with 25GB? So $20 per GB transferred? Which we're talking about running a real browser downloading all the bloat of a target website, 25GB doesn't go very far. Is this just absurdly expensive?

simple10 3y ago

These two websites used to be useful for testing bots. They may be several years old at this point. https://bot.sannysoft.com/ https://bot.sannysoft.com/ https://antoinevastel.com/bots/datadome https://antoinevastel.com/bots/datadome The first link shows the tests and is the most educational. Does anyone know of more recent tools that show why bot detection was triggered? Versus just testing against Cloudflare bot detection and getting pass/fail.

simple10 3y ago
Just found two more recent bot detectors that show the tests run: https://www.browserscan.net/en/bot-detection https://www.browserscan.net/en/bot-detection https://bot.incolumitas.com/ https://bot.incolumitas.com/

smsm42 3y ago

So basically is this Evil as a Service, or is there a legitimate case for this that is not identical to "stress testing is a legitimate case for me selling botnet DDos"?

leptons 3y ago
IFTTT is a legitimate use. There are an unlimited number of valid non-evil use cases for automated tasks looking at websites and doing stuff based on the data scraped.
- smsm42 3y ago
  Not sure how IFTTT is a legitimate use of bot detection evasion. I've been using IFTTT for many years and never needed that.
  - leptons 3y ago
    IFTTT is a legitimate use of scraping public data from websites. But forget about IFTTT, it was just used as an example. It's just an example of automation looking at public website content and then taking some action. You're too focused on IFTTT and not the problem with blockading publicly accessible data. For example, Zillow goes to great lengths to restrict bots, but I want to get updates if a property changes somehow on Zillow. It's publicly accessible data if you're not using a bot. "IFTTT" is just an example of a service that automates these things. I can write my own automation service, but if I'm using a bot, Zillow will block me (I've actually done this before).
    - smsm42 3y ago
      Doesn't Zillow have an API here? https://www.zillowgroup.com/developers/ https://www.zillowgroup.com/developers/

bradgessler 3y ago

I'm surprised Cloudflare hasn't tried to create a market for this yet. They could charge some sort of fee between the scrapers and the scrapees and create additional revenue streams for sites being scraped. Hell they could even give tools to sites that lets them manage who can scrape them and who can't. I'm not saying I like this, I'm just saying I'm surprised it hasn't happened yet.

nullpt_rs 3y ago
They have a similar feature labelled "Friendly Bots" https://blog.cloudflare.com/friendly-bots/ https://blog.cloudflare.com/friendly-bots/

dreamcompiler 3y ago

Cloudflare's WAF breaks the web. The web should be open, anonymous, and automatable on behalf of users. Any web service that doesn't like this idea should not get my business. Period. What we really need is a browser that absolutely cannot access sites "protected" by Cloudflare, and this browser needs to become standard. Sites that put themselves behind Cloudflare need to pay a price for their hostility, and that price is the loss of our business. If I want to program a bot to get around the horrendous lossage of the terrible UI and multi-megabyte bloat you have created by farming out your web server programming to the lowest bidder, I have the right to do so. In turn my bot will respect robots.txt and will throttle itself appropriately. This is the deal you signed up for when you built a web server. If you don't like it, GTF off the web. And take your attestation and WAF with you.

jfil 3y ago
I tend to agree. I'm working on a web scraping project that allows an ordinary person to spot misbehaviour by well-resourced corporations. Cloudflare is absolutely getting in the way, and helping the bigger guys escape accountability by making them hard to track.

cebert 3y ago

It’s not too difficult to run playwright in a container or from an AWS Lambda with a layer. What’s the advantage to Browserless vs. rolling your own?

hamoodhabibi 3y ago

im not sure what the legal liabilities are for offering this type of service but couldn't somebody potentially use this to scrape a highly litigious website and end up bringing heat to browserless's company and its founders I wonder "we just provide the tool" defence no longer works in court like there's even a 7 day trial opening this up to anyone with bad intentions