I built a similar thing in python using Playwright and Pandoc [0]. It's used by aider's `/web <url>` command that lets you paste a markdown version of any webpa

I built a similar thing in python using Playwright and Pandoc [0]. It's used by aider's `/web <url>` command that lets you paste a markdown version of any webpage into your AI coding chat. This helps if you want to include docs for an obscure or non-public package/api/etc with the LLM while coding. I really value dependencies which are easy for all users to install, cross-platform. Playwright is nice because it has a simple way to install its dependencies on most platforms. And the `pypandoc` module provides a seamless install of pandoc across platforms. The result turns most web pages into nice markdown, without requiring users to solve some painful platform specific chromium dependency nightmare. For example, it produced this markdown version [1] of the pyinfra getting started doc [2]. [0] https://github.com/paul-gauthier/aider/blob/main/aider/scrape.py https://github.com/paul-gauthier/aider/blob/main/aider/scrap... [1] https://gist.github.com/paul-gauthier/6c8dd16a6bceb9fd287494437c51317e https://gist.github.com/paul-gauthier/6c8dd16a6bceb9fd287494... [2] https://docs.pyinfra.com/en/2.x/getting-started.html https://docs.pyinfra.com/en/2.x/getting-started.html