3 ms·
> So it breaks all of the morphemes? Not really. TeX uses Knuth-Liang algorithm which does not do any morphological or semantical analysis. You basically feed
by ilammy 6y ago
> So it breaks all of the morphemes?
Not really. TeX uses Knuth-Liang algorithm which does not do any morphological or semantical analysis.
You basically feed it a corpus of words, it learns substrings that contain word breaks more often, condenses this information, and uses that to guess where an arbitrary word could be hyphenated.
In particular, TeX's hyphenation data contains lines
y3po
5po4g
which strongly suggest that when TeX sees substrings “ypo” and “?pog” they can be hyphenated as “y-po” and “?-pog”. (Odd weights allow hyphenation, even weights discourage it.)
That's why “typograher” is hyphenated like that, since it's not in the exception list.
- neolog 6y agoIs there a morpheme-aware version?
- dhosek 6y agoYes, with different patterns it's possible and the en-gb patterns do this (see https://www.tug.org/tex-hyphen/ https://www.tug.org/tex-hyphen/). Morpheme-aware hyphenation does have some oddities, though, like hyphenating helicopter as helico-pter.
- neolog 6y agoI saw your webpage on programming language choice, but I couldn't figure out how to use the comment box. So here's my comment. Rust seems like a good candidate. The runebender/druid/xi project has an active chatroom [1] full of people [2,3,...] with deep expertise in typesetting, text rendering, etc. [1] https://xi.zulipchat.com/ https://xi.zulipchat.com/ [2] https://raphlinus.github.io/ https://raphlinus.github.io/ [3] https://joy.recurse.com/~cmyr https://joy.recurse.com/~cmyr
- dhosek 6y agoI'm not sure why there was a problem with comments for you—it looks like it's correct for me, although no one else has commented so there might be an issue. Rust is intriguing and it looks like some of the issues that are of concern to me are addressed by it (cross-platform, ability to target iOS and JNI)... I'll have to investigate a bit more. If it's got a good PDF writing library I may have a winner, although C++ does offer some really solid abilities with ICU and harfbuzz that appeal to me.
- neolog 6y agoI couldn't figure out what all the boxes were for or why the big box truncated my comment at around 20 characters.
- dhosek 6y agoCertainly, the pain around library usage in C++ is something that would be nice to be able to avoid. A quick google search has turned up Rust bindings for harfbuzz and the print_pdf library (the latter would be superior to dealing with the scant open source C/C++ options most of which seem to have lost their primary maintainers).
- dhosek 6y agoOne of my minor claims to fame is contributing the word Wertherian to the hyphanation exception list. I only knew it was an issue because I saw it mis-hyphenated in a Trollope novel.