4 ms·
I strongly recommend adding Playwright to your set of tools for Python web scraping. It's by far the most powerful and best designed browser automation tool I'v
by simonw 3y ago
I strongly recommend adding Playwright to your set of tools for Python web scraping. It's by far the most powerful and best designed browser automation tool I've ever worked with.
I use it for my shot-scraper CLI tool: https://shot-scraper.datasette.io/ https://shot-scraper.datasette.io/ - which lets you scrape web pages directly from the command line by running JavaScript against pages to extract JSON data: https://shot-scraper.datasette.io/en/stable/javascript.html https://shot-scraper.datasette.io/en/stable/javascript.html
- sakisv 3y agoCame here to write about Playwright. I've been using it for the last ~13 months to scrape supermarket prices and it's been a great experience.
- mrtimo 3y agowould love to learn more about what you are doing with supermarket prices
- dommer 3y ago+1 Interested in this area as well.
- sakisv 3y agoMy main drive was to document the crazy price hikes that's been going on in my home country, Greece, so I'm scraping its 3 biggest supermarkets and keep track of the prices of their products* Had a lot of fun building and automating the scraping, especially in order to get around some bot catching rules that they have. For example one of them blocks all the requests originating from non-residential IPs, so I had to use tailscale to route the scraper's traffic through my home connection and take advantage of my ISP's CGNAT. You can take a look here: https://pricewatcher.gr/en/ https://pricewatcher.gr/en/ * I'm not doing any deduplication or price comparisons between the supermarkets, I only show historical prices of the same product, to showcase the changes.
- deanishe 3y agoI can't get prices on my local supermarket's website (in Germany) without selecting a specific branch. Seems kinda suspicious to me.
- sakisv 3y agoYeah in this case between these 3 supermarkets there are 2 options: 1. You choose your general location or 2. You don't choose In the first option you get one less general category to choose from (for example they may not have fresh fish) As far as I can tell, in both cases the supermarket closest to the delivery address is responsible for filling out your order and usually what happens is that they call you to let you know that they don't have something and suggest substitutions.
- samstave 3y agoI too choose this guys supermaket scraper! What Ive long wanted was the the ability to map prices to SCUs by having folks simply take a pic of the UPC + price, just like gasbuddy or what not - in addition to scraping from grocery posting their coupon sheets online for scraping, in addition to people just scanning (non-PII) portions of receipts. Can you share what you've made thus far? * could it be used as an automated "price matching" finder? (for those companies that do a "we price match!" thing
- Oras 3y ago+1 for playwright. The codegen is a brilliant way to simplify scraping.
- BeetleB 3y agoI actually use your shot-scraper tool (coupled with Mozilla's Readability) to extract the main text of a site (to convert to audio and listen via a podcast player). I love it! Some caveats though: - It does fail on some sites. I think the value of scrapy is you get more fine grained control. Although I guess if you can use any JS with shot-scraper you could also get that fine grained control. - It's slow and uses up a lot of CPU (because Playwright is slow and uses up a lot of CPU). I recently used shot-scraper to extract the text of about 90K sites (long story). Ran it on 22 cores, and the room got very hot. I suspect Scrapy would use an order of magnitude less power. On the plus side, of course, is the fact that it actually executes JS, so you can get past a lot of JS walls.
- simonw 3y agoWow, you're really putting it through its paces! When you ran it against 90,000 sites were you running the "shot-scraper" command 90,000 times? If so, my guess is that most of that CPU time is spent starting and stopping the process - shot-scraper wasn't designed for efficient start/stop times. I wonder if that could be fixed? For the moment I'd suggest writing Playwright code for 90,000 site scraping directly in Python or JavaScript, to avoid that startup overhead.
- BeetleB 3y agoYes, indeed I launched shot-scraper command 90K times. Because it's convenient :-) I didn't realize starting/stopping was that expensive. I thought it was mostly the fact that you're practically running a whole browser engine (along with a JS engine). If I do this again, I'll look into writing the playwright code directly (I've never used it).
- ravenstine 3y agoReadability is great, and I use it, but it's odd how half-assed the maintenance for it has been. I've haven't seen any noticeable improvements to it in quite some time, and when I've looked for alternatives, it usually turns out they're using it under the hood in some capacity. Perhaps it's already being made obsolete by LLM technologies? I'd be curious to hear from anyone who's used a locally running LLM to extract written content, especially if it's been built specifically for that task.
- thrdbndndn 3y agoKinda tangent, but Playwright's doc (specifically, the intro https://playwright.dev/python/docs/intro https://playwright.dev/python/docs/intro ) confuses me. It asks you to write a test and then run `pytest`, instead of just letting you to use the library directly (which exists, but is buried in the main text: https://playwright.dev/python/docs/library https://playwright.dev/python/docs/library). I understand that using Playwright in tests is probably the most common use case (it's even in their tagline) but ultimately the introduction section of a lib should be about the lib itself, not certain scenario to use it with a 3rd-party lib B (`pytest`). Especially when it may cause side effect (I wasn't "bitten" by it but surely was confusing: when I was learning it before, I created test_example.py as said in a minefield folder which has batch of other test_xxxx.py files. And running `pytest` causes all of them to run, and gives confusing outputs. And it's not obvious to me at all, since I've never used pytest before and this is not a documentation about pytest, so no additional context was given.) > tagline
- simonw 3y agoHah yeah that's confusing. https://playwright.dev/python/docs/intro https://playwright.dev/python/docs/intro is actually the documentation for pytest-playwright - their pytest plugin. https://playwright.dev/python/docs/library https://playwright.dev/python/docs/library is the documentation for their automation library. I just filed an issue pointing out that this is confusing. https://github.com/microsoft/playwright/issues/29579 https://github.com/microsoft/playwright/issues/29579
- PaulHoule 3y agoBack in the day I used to use HTMLUnit https://htmlunit.sourceforge.io/ https://htmlunit.sourceforge.io/ to crawl Javascript-based sites from Java. I think it was originally intended for integration tests but it sure works well for webcrawlers. I just wrote a Python-based webcrawler this weekend for a small set of sites that is connected to a bookmark manager (you bookmark a page, it crawls related pages, builds database records, copies images, etc.) and had a very easy time picking out relevant links, text and images w/ CSS selectors and beautifulsoup. This time I used a database to manage the frontier because the system is interactive (you add a new link and it ought to get crawled quickly) but for a long time my habit was writing crawlers that read the frontier for pass N from a text file which is one URL per line and then write the frontier for pass N+1 to another text file because this kind of crawler is not only simple to write but it doesn't get stuck in web traps. I have a few of these systems that do very heterogenous processing of mostly scraped content and something think about setting up a celery server to break work up into tasks .
- tnolet 3y ago100%. Playwright (which does have Python support) is completely owning this scene. The robustness is amazing.
- thundergolfer 3y agoWe use shot-scraper internally to automate keeping screenshots in our documentation up-to-date. Thanks for the tool![1] Agree that Playwright is great. It's super easy to run on Modal.[2] 1. https://modal.com/docs/guide/workspaces#dashboard https://modal.com/docs/guide/workspaces#dashboard 2. https://modal.com/docs/examples/web-scraper#a-simple-web-scraper https://modal.com/docs/examples/web-scraper#a-simple-web-scr...
- 3abiton 3y agoHow does it compare to selenium or puppeteer?
- black3r 3y agoPlaywright is a rewrite of puppeteer by people who worked on puppeteer before, but now under Microsoft instead of Github. Not sure if it reached feature parity yet, but all the things we used to do with puppeteer work with playwright, and it seems to be more actively developed.
- sam2426679 3y agoIme playwright is selenium plus some, e.g. you can inspect network activity without having a separately configured proxy.
- sam2426679 3y agoCan anyone recommend a good methodology for writing tests against a Playwright scraping project? I have a relatively sophisticated scraping operation going, but I haven’t found a great way to test methods that are dependent on JavaScript interaction behind a login. I’ve used Playwright’s har recording to great effect for writing tests that don’t require login, but I’ve found that har recording doesn’t get me there for post-login because the har playback keeps serving the content from pre-login (even though it includes the relevant assets from both pre and post login.)