4 ms·

I feel like I am going insane. Who in their right mind would feed an LLM raw HTML in the first place?? HTML is a markup language. User agents present it in a w

I feel like I am going insane. Who in their right mind would feed an LLM raw HTML in the first place?? HTML is a markup language. User agents present it in a way that makes sense for the user—visually, or through assistive technology. Nothing about adding LLM “users” should change this—their user agent, the harness, should present HTML in a way they can natively understand, by translating it to Markdown. We are not going to rewrite the entire web because harness developers are too lazy or stupid to pull an HTML to markdown package from npm. If some sites want to do so anyway, good for them, in many cases I’d love to skip the CSS/JS and read the markdown directly (or better, nicely formatted). But don’t blame website authors for your harness wasting your tokens.

alsetmusic 1mo ago
> in many cases I’d love to skip the CSS/JS and read the markdown directly I've been enjoying https://defuddle.md https://defuddle.md since learning about it. Works great. Not affiliated.
- usef- 1mo ago
  Exa also has an API for it that has worked well for me, returning markdown for a URL, which means you don't need to render js or anything yourself. It doesn't need an account for up to 1k requests/month, which is more than I've ever needed. Most harnesses have a "zero config" plugin that works, eg: https://pi.dev/packages/pi-exa https://pi.dev/packages/pi-exa (no affiliation to either)
k1m 1mo ago
Agree. I think many people forget that not long ago, HTML markup on many sites was a lot richer than it is today. Making it trivial to produce a good trimmed down markdown version. The reason it may be more difficult today is because we've lost a lot of that. Some of it because of modern JS frameworks, but some also because publishers simply don't want to make it easy for the useful stuff to be scraped and extracted easily. I'm not convinced that's changing because of AI agents (it's getting worse in many ways with anti-agent rules). Maybe improving for documentation pages intended for agents. But if it is changing, I think it'd be far easier to improve the HTML and let the agent take care of the rest.
Semiapies 1mo ago
Who in their right mind would feed any random LLM crawler their website's content?
- joshum97 1mo ago
  marketers
0xbadcafebee 1mo ago
I think you are operating under the misconception that software developers are in their right mind
pjerem 1mo ago
> Who in their right mind would feed an LLM raw HTML in the first place?? I'd even say, is that even a problem ? If your website isn't a front end blob soup and just a relatively clean HTML file, your LLM is totally capable to handle it anyway. And I'm still wondering myself which incentive do I have to serve my content to LLMs anyway.
- joshum97 1mo ago
  There's nothing wrong with "dirty" HTML. Modern layouts require lots of wrapper elements--certainly some sites use them unnecessarily, but it's not reasonable to expect a nice modern layout to use the same number of elements as a basic document. And utility CSS has merits. That's a totally separate concern from inaccessible div soup--you can still use sections, navs, aria- attributes, etc. If you're working with a marketing team, you'll find that they are desperate to serve their content to LLMs, to counteract the rapidly shrinking number of people actually visiting websites. What I don't want is for that to mean "build the website twice, one for humans and one for LLMs". Websites are for people—bots can adapt.
tigrandza 1mo ago
i have built 2 html to document pipelines and read it back with pdftotext, and I can say that "just convert it yourself" step is the place where things are getting wrong. For example Markdown has no support for the subscript and superscript in CommonMark or GFM and if you have 43 becomes 43 and water formula becomes H2O so in the output you get not the ugly but the wrong one. And no converter can fix it, because that's a target format which can't express it. extractors like Defuddle don't save from this also. and non of those extractors or converter throws you just get wrong text

alsetmusic 1mo ago

> in many cases I’d love to skip the CSS/JS and read the markdown directly I've been enjoying https://defuddle.md https://defuddle.md since learning about it. Works great. Not affiliated.

usef- 1mo ago
Exa also has an API for it that has worked well for me, returning markdown for a URL, which means you don't need to render js or anything yourself. It doesn't need an account for up to 1k requests/month, which is more than I've ever needed. Most harnesses have a "zero config" plugin that works, eg: https://pi.dev/packages/pi-exa https://pi.dev/packages/pi-exa (no affiliation to either)

k1m 1mo ago

Agree. I think many people forget that not long ago, HTML markup on many sites was a lot richer than it is today. Making it trivial to produce a good trimmed down markdown version. The reason it may be more difficult today is because we've lost a lot of that. Some of it because of modern JS frameworks, but some also because publishers simply don't want to make it easy for the useful stuff to be scraped and extracted easily. I'm not convinced that's changing because of AI agents (it's getting worse in many ways with anti-agent rules). Maybe improving for documentation pages intended for agents. But if it is changing, I think it'd be far easier to improve the HTML and let the agent take care of the rest.

Semiapies 1mo ago

Who in their right mind would feed any random LLM crawler their website's content?

joshum97 1mo ago
marketers

0xbadcafebee 1mo ago

I think you are operating under the misconception that software developers are in their right mind

pjerem 1mo ago

> Who in their right mind would feed an LLM raw HTML in the first place?? I'd even say, is that even a problem ? If your website isn't a front end blob soup and just a relatively clean HTML file, your LLM is totally capable to handle it anyway. And I'm still wondering myself which incentive do I have to serve my content to LLMs anyway.

joshum97 1mo ago
There's nothing wrong with "dirty" HTML. Modern layouts require lots of wrapper elements--certainly some sites use them unnecessarily, but it's not reasonable to expect a nice modern layout to use the same number of elements as a basic document. And utility CSS has merits. That's a totally separate concern from inaccessible div soup--you can still use sections, navs, aria- attributes, etc. If you're working with a marketing team, you'll find that they are desperate to serve their content to LLMs, to counteract the rapidly shrinking number of people actually visiting websites. What I don't want is for that to mean "build the website twice, one for humans and one for LLMs". Websites are for people—bots can adapt.

tigrandza 1mo ago

i have built 2 html to document pipelines and read it back with pdftotext, and I can say that "just convert it yourself" step is the place where things are getting wrong. For example Markdown has no support for the subscript and superscript in CommonMark or GFM and if you have 43 becomes 43 and water formula becomes H2O so in the output you get not the ugly but the wrong one. And no converter can fix it, because that's a target format which can't express it. extractors like Defuddle don't save from this also. and non of those extractors or converter throws you just get wrong text