4 ms·
I'm working on pure.md[1], which lets your scripts, APIs, apps, agents, etc reliably access web content in markdown format. Simply prefix any URL with `pure.md/
by andrethegiant 2y ago
I'm working on pure.md[1], which lets your scripts, APIs, apps, agents, etc reliably access web content in markdown format. Simply prefix any URL with `pure.md/` and you get the unblocked markdown content of that webpage. It avoids bot detection and renders JavaScript-heavy websites, and can convert HTML, PDFs, images, and more into pure markdown.
pure.md acts as a global caching layer between LLMs and web content. I like to think of it like a CDN for LLMs, similar to how Cloudinary is a CDN for images.
[1] https://pure.md https://pure.md
- wild_egg 2y agoThanks for sharing. I was planning on building something like this in April after hitting too many issues with Jina and Tavily but it looks like you've already done the hard work!
- andrethegiant 2y agoThanks! Still a work in progress :-)
- wanderingbit 2y agoWhat a great idea, I will soon be a paying customer. This solves a problem of an app I'm using that I was hesitant to try to develop myself.
- andrethegiant 2y agoMuch appreciated!
- metadat 2y agoCool project! Recently discussed, too: https://news.ycombinator.com/item?id=43462894 https://news.ycombinator.com/item?id=43462894 (10 comments)
- hardlyfun 2y agoVery nice, how did you manage to bypass sites with cloudflare turnstile setup?
- udev4096 2y agoFlaresolverr most probably
- udev4096 2y ago[flagged]
- NationOfJoe 2y agoi have no skin in the game and honestly i am wondering how this idea contributes to enshittifying the web more? this idea just seems like it provides the same content as visiting the site in a different view, like reader mode?
- elric 2y agoI think the parent is referring to the goal of making the web more "LLM friendly".
- hbsbsbsndk 2y agoThe service seems designed to bypass anti-scraping measures. If site owners don't want their content scraped by AI this is subverting their will. It also obfuscates responsibility between the AI vendor and the scraping service. One can imagine unethical AI providers using a series of ephemeral "gateways" to access content while avoiding any legal or reputational harm.
- shoebham 2y agoLove the recursion redirect at pure.md/pure.md
- andrethegiant 2y agoYou found the easter egg!
- m0rde 2y agoIs there an example we can see?
- 27theo 2y agohttps://pure.md/https://news.ycombinator.com/item?id=43533233 https://pure.md/https://news.ycombinator.com/item?id=4353323...
- WillAdams 2y agoIt seems to miss URLs? At: https://willadams.gitbook.io/design-into-3d/2d-drawing https://willadams.gitbook.io/design-into-3d/2d-drawing the links for: https://mathcs.clarku.edu/~djoyce/java/elements/elements.html https://mathcs.clarku.edu/~djoyce/java/elements/elements.htm... https://mathcs.clarku.edu/~djoyce/java/elements/bookI/bookI.html https://mathcs.clarku.edu/~djoyce/java/elements/bookI/bookI.... https://mathcs.clarku.edu/~djoyce/java/elements/bookI/defI1.html https://mathcs.clarku.edu/~djoyce/java/elements/bookI/defI1.... are rendered as: _Elements_ _:_ _Book I_ _:_ _Definition 1_ Maybe detect when a page is on gitbook or some other site where there is .md source on github or some other site and grab the original instead?
- andrethegiant 2y agoBy default, href values of <a> tags are removed, because they add significant token length without adding more context. Coming soon, you can specify a request header to set whether or not you want links removed from the response. Those underscores you mentioned are from the italics.
- sharpshadow 2y agoWorks great on mobile thanks, helpful tool to bypass flaky websites, js and even some paywalls.
- erekp 2y agohow do you exactly fallback to common crawl? isn't the cost to even hold and query common crawl insane?
- andrethegiant 2y agoWith AWS Athena, you can query the contents of someone else’s public S3 bucket. You pay per read, but if you craft your query the right way then it’s very inexpensive. Each query I run only scans about 1MB of data.
- wfn 2y agoSince I was just looking at this accidentally, here are some examples of how to query at a ~cent-per-query cost level (just examples but quite illustrative): https://commoncrawl.org/blog/index-to-warc-files-and-urls-in-columnar-format https://commoncrawl.org/blog/index-to-warc-files-and-urls-in...