6 ms·

2markdown – Transform Websites into Markdown

17 points by smartmic 2y ago

MilStdJunkie 2y ago
Wow, "pay as you go html to markdown" seems like a very hard sell in this year of our lord 2024. I've got a brace of converters for various parts of this problem, some of them are pushing a decade in age. You have products like Zapier or the open source ActivePieces to restrain extraction to specific elements. Finally, several pre-wrapped AI models for this specific purpose.
franze 2y ago
Should definitely use a try 1st one free without login demo (maybe with a strong captcha if missuse costs are an issue)
- eddd-ddde 2y ago
  The markup on this seems insane. A Cloudflare worker could easily do this and you would make 10K on 30 cents.
hinkley 2y ago
> empower your AI models pardon?
- nextaccountic 2y ago
  I suppose that AI models may take markdown as input? But they can take html as input as well..
- LordDragonfang 2y ago
  I saw that and immediately clicked off. I do sometimes want documents in markdown format (though Obsidian is pretty good at just converting whatever you paste in automatically), but it's clear this is a lazy $current_buzzword-hype cash grab. Not worth HN front page space.
  - swyx 2y ago
    no, its a real need. llms need to ingest websites and markdown is more efficient than html.
disembiggen 2y ago
pandoc as a service, mad times we live in, best of luck I suppose
jonplackett 2y ago
Why not just use something like https://github.com/mozilla/readability https://github.com/mozilla/readability And not pay $0.01 per request? There’s a node version too https://www.npmjs.com/package/@mozilla/readability https://www.npmjs.com/package/@mozilla/readability
- anotherpaulg 2y ago
  Not the OP, but I tested readability for this purpose. My use case was converting web pages of API documentation into markdown for AI coding. Readability works pretty well on "newspaper article" or "blog post" type content. But fails badly on the more structured webpages that I was interested in like API documentation, etc.
anotherpaulg 2y ago
I built a similar thing in python using Playwright and Pandoc [0]. It's used by aider's `/web <url>` command that lets you paste a markdown version of any webpage into your AI coding chat. This helps if you want to include docs for an obscure or non-public package/api/etc with the LLM while coding. I really value dependencies which are easy for all users to install, cross-platform. Playwright is nice because it has a simple way to install its dependencies on most platforms. And the `pypandoc` module provides a seamless install of pandoc across platforms. The result turns most web pages into nice markdown, without requiring users to solve some painful platform specific chromium dependency nightmare. For example, it produced this markdown version [1] of the pyinfra getting started doc [2]. [0] https://github.com/paul-gauthier/aider/blob/main/aider/scrape.py https://github.com/paul-gauthier/aider/blob/main/aider/scrap... [1] https://gist.github.com/paul-gauthier/6c8dd16a6bceb9fd287494437c51317e https://gist.github.com/paul-gauthier/6c8dd16a6bceb9fd287494... [2] https://docs.pyinfra.com/en/2.x/getting-started.html https://docs.pyinfra.com/en/2.x/getting-started.html
billybuckwheat 2y ago
I'll stick with using Markdownload (https://github.com/deathau/markdownload https://github.com/deathau/markdownload) when I need to do something like this.
low_tech_punk 2y ago
Not interesting in the service but fascinated by this new wave of re-modularization and re-integration of services through the lens of Ben Thompson aggregation theory. Modularizers like 2markdown are building single purpose services optimized for LLM workloads Aggregators like ChatGPT fights for end-to-end AI experience. It's just exciting to watch.
swyx 2y ago
> From this > This content displays when the browser does not support custom elements. > To this > This content displays when the browser does not support custom elements. this happens with ublock origin on
low_tech_punk 2y ago
These two blob posts by the author seems to be revealing: How to Build a Startup Without Funding https://andinfinity.eu/post/2024-04-02-how-to-build-a-startup-without-funding/ https://andinfinity.eu/post/2024-04-02-how-to-build-a-startu... What's the minimal set of user stories for a Micro-SaaS MVP? https://andinfinity.eu/post/2022-06-06-whats-part-of-an-mvp-for-a-micro-saas/ https://andinfinity.eu/post/2022-06-06-whats-part-of-an-mvp-...
meatjuice 2y ago
It might be helpful when accessing news websites outdoor (since my internet provider rate limits hard). It’s basically a proxy, right? I’m not even going to try though because there’s a signup wall.
cpeffer 2y ago
If you’re looking for an open core version of this check out firecrawl.dev
reagle 2y ago
I have a wrapper for the many ways you might convert something to a text document in the terminal. https://github.com/reagle/pandoc-wrappers/blob/main/doc2txt.py https://github.com/reagle/pandoc-wrappers/blob/main/doc2txt....
CyberFoxar 2y ago
I used to be intrested in converting/saving websites to my own archives. The web is rotting away and saving a local copy is the easiest way to make sure I keep my references. This, this ain't it chief. If you're looking for local saving, use firefox extensions. MarkDownload and SinglePage are both very good for backing up single web pages.
mariocesar 2y ago
I have been using lynx to integrate a chatbot with web content. Here's the Python function that converts a website to plain text: def website_to_text(url): output = subprocess.check_output(["lynx", "-dump", "-nolist", url], stderr=subprocess.PIPE, universal_newlines=True) return output.strip() While a hosted service can be convenient, I find that using a text-based browser like Lynx gives almost the same results and consistent results.

17 points by smartmic 2y ago

Wow, "pay as you go html to markdown" seems like a very hard sell in this year of our lord 2024. I've got a brace of converters for various parts of this problem, some of them are pushing a decade in age. You have products like Zapier or the open source ActivePieces to restrain extraction to specific elements. Finally, several pre-wrapped AI models for this specific purpose.

franze 2y ago

Should definitely use a try 1st one free without login demo (maybe with a strong captcha if missuse costs are an issue)

eddd-ddde 2y ago
The markup on this seems insane. A Cloudflare worker could easily do this and you would make 10K on 30 cents.

hinkley 2y ago

> empower your AI models pardon?

nextaccountic 2y ago
I suppose that AI models may take markdown as input? But they can take html as input as well..
LordDragonfang 2y ago
I saw that and immediately clicked off. I do sometimes want documents in markdown format (though Obsidian is pretty good at just converting whatever you paste in automatically), but it's clear this is a lazy $current_buzzword-hype cash grab. Not worth HN front page space.
- swyx 2y ago
  no, its a real need. llms need to ingest websites and markdown is more efficient than html.

disembiggen 2y ago

pandoc as a service, mad times we live in, best of luck I suppose

jonplackett 2y ago

Why not just use something like https://github.com/mozilla/readability https://github.com/mozilla/readability And not pay $0.01 per request? There’s a node version too https://www.npmjs.com/package/@mozilla/readability https://www.npmjs.com/package/@mozilla/readability

anotherpaulg 2y ago
Not the OP, but I tested readability for this purpose. My use case was converting web pages of API documentation into markdown for AI coding. Readability works pretty well on "newspaper article" or "blog post" type content. But fails badly on the more structured webpages that I was interested in like API documentation, etc.

anotherpaulg 2y ago

I built a similar thing in python using Playwright and Pandoc [0]. It's used by aider's `/web <url>` command that lets you paste a markdown version of any webpage into your AI coding chat. This helps if you want to include docs for an obscure or non-public package/api/etc with the LLM while coding. I really value dependencies which are easy for all users to install, cross-platform. Playwright is nice because it has a simple way to install its dependencies on most platforms. And the `pypandoc` module provides a seamless install of pandoc across platforms. The result turns most web pages into nice markdown, without requiring users to solve some painful platform specific chromium dependency nightmare. For example, it produced this markdown version [1] of the pyinfra getting started doc [2]. [0] https://github.com/paul-gauthier/aider/blob/main/aider/scrape.py https://github.com/paul-gauthier/aider/blob/main/aider/scrap... [1] https://gist.github.com/paul-gauthier/6c8dd16a6bceb9fd287494437c51317e https://gist.github.com/paul-gauthier/6c8dd16a6bceb9fd287494... [2] https://docs.pyinfra.com/en/2.x/getting-started.html https://docs.pyinfra.com/en/2.x/getting-started.html

billybuckwheat 2y ago

I'll stick with using Markdownload (https://github.com/deathau/markdownload https://github.com/deathau/markdownload) when I need to do something like this.

low_tech_punk 2y ago

Not interesting in the service but fascinated by this new wave of re-modularization and re-integration of services through the lens of Ben Thompson aggregation theory. Modularizers like 2markdown are building single purpose services optimized for LLM workloads Aggregators like ChatGPT fights for end-to-end AI experience. It's just exciting to watch.

swyx 2y ago

> From this > This content displays when the browser does not support custom elements. > To this > This content displays when the browser does not support custom elements. this happens with ublock origin on

low_tech_punk 2y ago

These two blob posts by the author seems to be revealing: How to Build a Startup Without Funding https://andinfinity.eu/post/2024-04-02-how-to-build-a-startup-without-funding/ https://andinfinity.eu/post/2024-04-02-how-to-build-a-startu... What's the minimal set of user stories for a Micro-SaaS MVP? https://andinfinity.eu/post/2022-06-06-whats-part-of-an-mvp-for-a-micro-saas/ https://andinfinity.eu/post/2022-06-06-whats-part-of-an-mvp-...

meatjuice 2y ago

It might be helpful when accessing news websites outdoor (since my internet provider rate limits hard). It’s basically a proxy, right? I’m not even going to try though because there’s a signup wall.

cpeffer 2y ago

If you’re looking for an open core version of this check out firecrawl.dev

reagle 2y ago

I have a wrapper for the many ways you might convert something to a text document in the terminal. https://github.com/reagle/pandoc-wrappers/blob/main/doc2txt.py https://github.com/reagle/pandoc-wrappers/blob/main/doc2txt....

CyberFoxar 2y ago

I used to be intrested in converting/saving websites to my own archives. The web is rotting away and saving a local copy is the easiest way to make sure I keep my references. This, this ain't it chief. If you're looking for local saving, use firefox extensions. MarkDownload and SinglePage are both very good for backing up single web pages.

mariocesar 2y ago

I have been using lynx to integrate a chatbot with web content. Here's the Python function that converts a website to plain text: def website_to_text(url): output = subprocess.check_output(["lynx", "-dump", "-nolist", url], stderr=subprocess.PIPE, universal_newlines=True) return output.strip() While a hosted service can be convenient, I find that using a text-based browser like Lynx gives almost the same results and consistent results.