3 ms·

Is Beautiful Soup still the best way to scrape the web with python? IIRC, Beautiful Soup doesn't handle javascript, so at least for JS you're forced to use som

Is Beautiful Soup still the best way to scrape the web with python? IIRC, Beautiful Soup doesn't handle javascript, so at least for JS you're forced to use something else. I'm also looking forward to seeing how people scrape the web once Web Assembly becomes prevalent.

NegativeLatency 4y ago
It can be useful if it fits your case, the more recent scrapers run a whole browser with automation which can make scraping stuff a lot easier, since js will run etc
diarrhea 4y ago
For JS, I've used Selenium with a Chrome driver, then parsed the HTML with BeautifulSoup. I know nothing about web development, so this might be an outdated way, but it worked. BS was nice to deal with.
- wildrhythms 4y ago
  I used Puppeteer for this to great success. Very easy to set up, and you control the whole browser + access to everything you would have access to normally through Chrome dev tools.
driscoll42 4y ago
If you don't need JS, I think it's still the best. Sure, that doesn't always work for you, but once you start needing to render JS the scraping slows down tremendously.
move-on-by 4y ago
I’ve found Playwright to be a really great tool for scraping
- derac 4y ago
  Playwright is easily the best for browser automation. I still use requests + beautiful soup often as well.
- samwillis 4y ago
  Headless scraping is in the region of 10x slower and more resource intensive, even when carefully blocking requests such as images. It should always be a second choice. Other than that Playwright is incredible, by far the best browser automation api.
  - move-on-by 4y ago
    For sure it’s a heavy approach, but if you need a full blown browser with JS, then that’s just what you’ll have to do. Use the right tool for the job.
- malshe 4y ago
  This sounds interesting. Any resources for a beginner? I use Selenium regularly.
  - simonw 4y ago
    Playwright for Python has really good documentation: https://playwright.dev/python/ https://playwright.dev/python/ I used it for my https://shot-scraper.datasette.io/ https://shot-scraper.datasette.io/ tool, and wrote a bit about CLI-driven scraping using that tool here: https://simonwillison.net/2022/Mar/14/scraping-web-pages-shot-scraper/ https://simonwillison.net/2022/Mar/14/scraping-web-pages-sho...
    - malshe 4y ago
      Thanks!
  - 8n4vidtmkvmk 4y ago
    selenium stinks in comparison to playwright and puppeteer

NegativeLatency 4y ago

It can be useful if it fits your case, the more recent scrapers run a whole browser with automation which can make scraping stuff a lot easier, since js will run etc

diarrhea 4y ago

For JS, I've used Selenium with a Chrome driver, then parsed the HTML with BeautifulSoup. I know nothing about web development, so this might be an outdated way, but it worked. BS was nice to deal with.

wildrhythms 4y ago
I used Puppeteer for this to great success. Very easy to set up, and you control the whole browser + access to everything you would have access to normally through Chrome dev tools.

driscoll42 4y ago

If you don't need JS, I think it's still the best. Sure, that doesn't always work for you, but once you start needing to render JS the scraping slows down tremendously.

move-on-by 4y ago

I’ve found Playwright to be a really great tool for scraping

derac 4y ago
Playwright is easily the best for browser automation. I still use requests + beautiful soup often as well.
samwillis 4y ago
Headless scraping is in the region of 10x slower and more resource intensive, even when carefully blocking requests such as images. It should always be a second choice. Other than that Playwright is incredible, by far the best browser automation api.
- move-on-by 4y ago
  For sure it’s a heavy approach, but if you need a full blown browser with JS, then that’s just what you’ll have to do. Use the right tool for the job.
malshe 4y ago
This sounds interesting. Any resources for a beginner? I use Selenium regularly.
- simonw 4y ago
  Playwright for Python has really good documentation: https://playwright.dev/python/ https://playwright.dev/python/ I used it for my https://shot-scraper.datasette.io/ https://shot-scraper.datasette.io/ tool, and wrote a bit about CLI-driven scraping using that tool here: https://simonwillison.net/2022/Mar/14/scraping-web-pages-shot-scraper/ https://simonwillison.net/2022/Mar/14/scraping-web-pages-sho...
  - malshe 4y ago
    Thanks!
- 8n4vidtmkvmk 4y ago
  selenium stinks in comparison to playwright and puppeteer