4 ms·
How do you get by when every major sites starts blocking headless browsers? A good example right now is Zillow, but I foresee a world where big chunks of the in
by cjonas 2y ago
How do you get by when every major sites starts blocking headless browsers? A good example right now is Zillow, but I foresee a world where big chunks of the internet are behind captcha and bot detection
- andrewmcwatters 2y agoThat's not really a problem for Stagehand. It's a problem for Selenium, Playwright, Puppeteer and others at the browser automation library level.
- cjonas 2y agoit's not really a problem for Playwrite, because Playwrite is really intended to be run by the owners of the website, not as a webscraper. It may become a real problem for the usefulness of this style of LLM driven browsing.
- andrewmcwatters 2y agoRight, but I mean it's a plugin issue outside of Stagehand. That's currently how the space treats the issue. There are plugins that extend the major browser automation libraries to do this sort of thing. And undetectable browser automation doesn't use Playwright, Puppeteer, or Selenium at all.
- keyserj 2y agoPlaywright does have some docs on scraping, which makes it seem like they do want to support that as a use case https://playwright.dev/docs/docker#crawling-and-scraping https://playwright.dev/docs/docker#crawling-and-scraping. I'm unfamiliar with that though and wouldn't be able to speak to addressing issues with scraping