4 ms·
It's increasingly difficult these days to write scrapers that don't at some point need to execute JavaScript on a page - so you need to have a good browser auto
by simonw 4y ago
It's increasingly difficult these days to write scrapers that don't at some point need to execute JavaScript on a page - so you need to have a good browser automation tool on hand.
I'm really impressed by Playwright. It feels like it has learned all of the lessons from systems like Selenium that came before it - it's very well designed and easy to apply to problems.
I wrote my own CLI scraping tool on top of Playwright a few months ago, which has been a fun way to explore Playwright's capabilities: https://simonwillison.net/2022/Mar/14/scraping-web-pages-shot-scraper/ https://simonwillison.net/2022/Mar/14/scraping-web-pages-sho...
- latchkey 4y agoTotally separate question, but I'm wondering why you put 'Mar' in your url instead of the month number?
- simonw 4y agoIt's a decision from 2003 I think. It's mainly because I'm from the UK, so I'm extremely sensitive to the risk of people confusing DD-MM-YYYY and MM-DD-YYYY - the least ambiguous format is to use DD-Mon-YYYY, so I picked that for my URLs. If I was designing my blog today I'd probably drop the day and month entirely, and go with /yyyy/unique-text-slug for the URLs.
- randomcarbloke 4y agoleast ambiguous is ISO 86 since it is standardised, but I agree that alpha month is unambiguous.
- latchkey 4y agoThanks simon! Makes perfect sense.
- brianzelip 4y ago> the least ambiguous format is to use DD-Mon-YYYY I consider YYYY-MM-DD to be the least ambiguous. But now that I look at the above, I guess the author is saying that since MM-DD could possibly be considered as DD-MM? Yuck.
- Terretta 4y agoISO 8601 is least ambiguous, as there's no question of "endian-ness". https://en.wikipedia.org/wiki/ISO_8601 https://en.wikipedia.org/wiki/ISO_8601 YYYY-MM-DD Yes, with year last, most of the world does it one way, but most of the audience are often from a part of the world that (a) thinks it's most of the world, and (b) does it the wrong way by shuffling endian-ness.
- contingencies 4y agoThis. Two additional benefits of this approach are it sorts correctly and it's already standard in China, which is effectively a whole heap of the world's internet population.
- marginalia_nu 4y agoThe killer feature of ISO-8601 style dates is that you don't need to parse the dates to sort them. Lexicographical order is chronological order. That's a pretty huge deal.
- shubhamjain 4y agoMy $0.02, but in most cases, I have seen you don't need to emulate a browser to scrape even if it's an SPA. The data has to be coming from somewhere. You can play around devtools to reverse engineer the API requests and get the data you need. I understand companies can put roadblocks to hinder this, but my point is, browser emulation is slow and expensive resource-wise. It should be the last resort.
- simonw 4y agoYeah, my first step in trying to scrape an SPA is always to hit the network tab in the browser, filter by type=JSON and then sort by size. The largest responses are often the most useful, and can then be grabbed with curl. Sometimes though that's not enough - particularly on older sites that might use weirder concepts like ASP.NET View state. For those I find having Playwright around is a big benefit. Generally the things I have the most trouble with for non-browser-automation scraping are things with complex state stored in cookies and URL fragments (and maybe even localStorage these days).
- real-dino 4y agoI'm struggling with a site at the moment for exactly this reason, it requires an auth key to return the products. Puppeteer had it working, but if I need to do this in a google firebase function, it would be so much better to get it to simple fetch requests. For the e-commerce sites I'm looking at though, most just are server side rendered, which is so much easier. In that case i use `cheerio` which has jQuery like syntax for crawling the raw text DOM.
- benbristow 4y agoMy £0.02 It's usually easier to use an Android emulator like GenyMotion or a rooted Android phone and use HTTPToolkit and/or some certificate bypassing method using Frida or other and then explore APIs through their official apps. I've scraped loads of stuff through unofficial APIs before this way. Most developers don't ever expect people to do this so they're often a bit less secure too. Alternatively sometimes doing a Global GitHub / Sourcegraph search you might find someone else who's done the hard work to reverse engineer an API and open-sourced it.
- 1vuio0pswjnm7 4y ago"It's increasingly difficult these days to write scrapers that don't at some point need to execute Javascript on a page - so you need to have a good browser automation tool on hand." But doesn't this assume which sites are being "scraped". How would anyone know what sites someone else needs to "scrape" unless people name the sites (and the specific pages at the sites as this is not "crawling"). For example, none of the websites with webpages I extract data from require me to use Javascript, i.e., I can retrieve and extract data without using JS. Also, it is possible to automate text-only browsers that do not run Javascript. "Browser automation" is not necessarily just for Javascript. Maybe we should have a "scraping challenge" in an effort to provide some evidence on this question. The challenge could be to "scrape" every webpage currently submitted to HN,^1 without using Javascript.^2 If someone manages to scrape a majority of the pages submitted to HN without JS, then we have some evidence that, for HN readers, JS and therefore Javascript-enabled browser automation is generally _not_ required for "scraping". 1. The problem with I see using something more generic like majestic_million.csv is that it is a list of domain names not webpages. 2. We would likely need to agree on what data would need to be extracted from each submitted page.
- simonw 4y agoI'll rephrase: "It's increasingly difficult these days to regularly write scrapers for a large range of different websites without eventually hitting a situation where you need to execute JavaScript on a page"
- tzs 4y agoWhat happens in Playwright with sites using CAPTCHAs? I had been occasionally scraping a site via curl, but then they started using Cloudflare's anti-bot stuff. I switched to Selenium and that worked for a while--my Selenium script would navigate to the site, pause to let me manually deal with Cloudflare, and then automatically grab the data I wanted. But then that stopped working. I found a Stack Overflow answer that gave some settings in Selenium to make it not tell the site's JavaScript that the browser was being automated and that briefly made things happy, but not too long afterwards that broke. There's a Selenium Chrome drive available that is meant for scraping which apparently tries to hide all evidence that the browser is being automated, but it didn't fool Cloudflare. What I want is a browser-based automation tool that to the site is indistinguishable from a human browsing, except possibly by the timing of user actions. E.g., if the site can deduce it is being automated because the client responds faster than human reaction time, or with too little variation in response time, that's fine.