52 ms·
Ask HN: A translation pipeline that doesn't chew up long HTML pages
My site has 1000+ HTML pages. Most of the content is AI-drafted and
poured into templates, so pages tend to be long and full of repeated
structural markup instead of hand-written one-offs.
Feed a long HTML page straight to an LLM for translation and it doesn't
just mistranslate. It silently drops content. Whole sections vanish,
paragraphs get summarized instead of translated, and the longer the page,
the worse it gets. You get back something that looks like a translation
but is missing half the page, with no warning that anything was lost.
Naively splitting the page into smaller chunks, by line or by character
count, doesn't fix this. It just cuts through the middle of tags and
sentences, breaking the structure and handing the model fragments that
don't mean anything on their own.
What actually works is parsing the page into its real structure with
tree-sitter, then chunking along structural boundaries so each chunk is a
complete, meaningful unit and nothing gets silently cut or dropped.
Anything non-translatable, like template logic or layout attributes,
stays out of the model's hands entirely. A structural check after
translation catches anything that still slips through, mostly as a
safety net at this point.
Ran it for real on 1000+ pages, six languages, output checked by hand.
Not pitching anything. Just want to know if this content-loss problem
bites other people translating long pages with LLMs, or if it's specific
to how my site is built.
- i18nagentai 3mo ago[flagged]