2 ms·
Does it also include logic to download JS-driven sites properly or is this out of scope?
by oezi 2y ago
Does it also include logic to download JS-driven sites properly or is this out of scope?
- simonw 2y agoIt doesn't. For that you would need to execute a full headless browser first, extract the HTML (document.body.innerHTML after the page has finished loading can work) and process the result. If you're already running a headless browser you may as well run the conversion in JavaScript though - I use this recipe pretty often with my shot-scraper tool: https://shot-scraper.datasette.io/en/stable/javascript.html#example-extracting-page-content-with-readability-js https://shot-scraper.datasette.io/en/stable/javascript.html#... - adding https://github.com/mixmark-io/turndown https://github.com/mixmark-io/turndown to the mix will get you Markdown conversion as well.
- JohannesKauf 2y agoThat is unfortunately out of scope. I like the philosophy of doing one thing really well. But nowadays—with Playwright and Puppeteer—there are great choices for Browser automation.
- jot 2y agoWe do that with Urlbox’s markdown feature: https://urlbox.com/extracting-text https://urlbox.com/extracting-text
- bni 2y agoI used https://github.com/mozilla/readability https://github.com/mozilla/readability for this