3 ms·
"Text normalization in Go" https://news.ycombinator.com/item?id=6806062 https://news.ycombinator.com/item?id=6806062 was here yesterday and made me wonder: how
by willvarfar 13y ago
"Text normalization in Go" https://news.ycombinator.com/item?id=6806062 https://news.ycombinator.com/item?id=6806062 was here yesterday and made me wonder: how can you handle ligatures, accents, digraphs etc in fixed-width?
Perhaps text parsing is just a hard problem?
- qznc 13y agoYes it is hard. Instead of code points, humans usually handle text by graphemes, which have an arbitrary length in edge cases [0] no matter what encoding is used. An O(1) solution for graphemes would be an array of pointers to grapheme objects, which is slower and more memory hungry than necessary in most cases. [0] http://stackoverflow.com/questions/6579844/how-does-zalgo-text-work http://stackoverflow.com/questions/6579844/how-does-zalgo-te...
- deleted 13y ago[deleted]