4 ms·
The trick there is to use a subagent to read the HTML page and extract the relevant information, than dumping all that HTML into your top-level session. That'
by simonw 2mo ago
The trick there is to use a subagent to read the HTML page and extract the relevant information, than dumping all that HTML into your top-level session.
That's effectively using an LLM as an HTML to markdown converter, which is both absurdly wasteful and also surprisingly inexpensive (if you use a model like GPT-5.6 Luna.)
- honr 2mo agoI found HTML itself works [slightly] better with some LLMs (that I happen to frequent). So, when scraping, I parse the resulting html, simplify it, and produce simpler html contents (closer to semantics of what I guessed the real content was). Given processing tools for html are far more mature, I tend to keep content in semantic, low structure html.