3 ms·
Reader – web scraping that outputs clean Markdown for LLMs
- nihalwashere 9mo agoI built an open-source web scraping engine for LLMs, self-hostable, Docker ready I kept rebuilding web scraping infrastructure for AI projects, so I open-sourced it. Two functions: - `scrape()` – any URL to markdown - `crawl()` – entire sites with depth/page limits Self-hosting: - Docker image available - Manages its own browser pool - Proxy support built-in - No external dependencies Deployment guide: https://docs.reader.dev/documentation/guides/deployment https://docs.reader.dev/documentation/guides/deployment GitHub: https://github.com/vakra-dev/reader https://github.com/vakra-dev/reader Built with TypeScript, runs anywhere you can run Docker. Happy to answer questions about the architecture or deployment.
- jauntywundrkind 9mo agoMade before cloudflare launched text/markdown, nice! If the timestamps are to be believed! https://blog.cloudflare.com/markdown-for-agents/ https://blog.cloudflare.com/markdown-for-agents/ https://news.ycombinator.com/item?id=46997526 https://news.ycombinator.com/item?id=46997526