3 ms·
That "café" -> "cafeś" replacement is pretty scary. It looks like the built in strings.Replace function makes the same mistake: fmt.Println(strings.Replace
by Nogwater 13y ago
That "café" -> "cafeś" replacement is pretty scary. It looks like the built in strings.Replace function makes the same mistake:
fmt.Println(strings.Replace("multiple cafe\u0301", "cafe", "cafes", 1)) // multiple cafeś
- rsc 13y agofmt.Println(strings.Replace("multiple cafeterias", "cafe", "cafes", 1))
- Nogwater 13y agoYeah, I get that. It's just that you might assume that the strings functions would operate on character boundaries (as defined in the blog post) and not based on runes (code points). Leaky abstractions and all that...
- enneff 13y agoThe purpose of the normalization package is to help you work with text under these constraints. I can't imagine many situations where strings.Replace would be sufficient for reliably manipulating natural language. The cafe example is to demonstrate the why you might need the package.
- Nogwater 13y agoI wasn't thinking that I'd really want to pluralize text like this, but maybe you'd want to turn people's names into links in HTML source or something. If someone's name ends with an accent, and if the unicode isn't normalized, strange things are bound to happen. The blog post is great at pointing this out, and it sounds like people are working on a go.text/search package to help, so that's good. I'm not saying Go is broken, just that this kind of stuff can be really surprising.
- enneff 13y agoYep, working with natural languages is scary. :-)
- lelf 13y agoIt's not for bashing the parent comment, but I find it funny. From the beginning of time, 99.99% languages have horrific unicode support (and 99.999% programmers have not got a bit of clue in this area) and then suddenly…