3 ms·

I’ve found Playwright to be a really great tool for scraping

• derac 4y ago
  Playwright is easily the best for browser automation. I still use requests + beautiful soup often as well.
• samwillis 4y ago
  Headless scraping is in the region of 10x slower and more resource intensive, even when carefully blocking requests such as images. It should always be a second choice. Other than that Playwright is incredible, by far the best browser automation api.
  • move-on-by 4y ago
    For sure it’s a heavy approach, but if you need a full blown browser with JS, then that’s just what you’ll have to do. Use the right tool for the job.
• malshe 4y ago
  This sounds interesting. Any resources for a beginner? I use Selenium regularly.
  • simonw 4y ago
    Playwright for Python has really good documentation: https://playwright.dev/python/ https://playwright.dev/python/ I used it for my https://shot-scraper.datasette.io/ https://shot-scraper.datasette.io/ tool, and wrote a bit about CLI-driven scraping using that tool here: https://simonwillison.net/2022/Mar/14/scraping-web-pages-shot-scraper/ https://simonwillison.net/2022/Mar/14/scraping-web-pages-sho...
    • malshe 4y ago
      Thanks!
  • 8n4vidtmkvmk 4y ago
    selenium stinks in comparison to playwright and puppeteer