4 ms·
Rust's string type doesn't have the mentioned problems: https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=7b00bb63f816d8a1fbf71aff85d34a21
by brundolf 5y ago
Rust's string type doesn't have the mentioned problems: https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=7b00bb63f816d8a1fbf71aff85d34a21 https://play.rust-lang.org/?version=stable&mode=debug&editio...
Though I would bet this mainly has to do with being a language born in the age of the web. It seems like something that would be very hard to back-port (as opposed to designing for it from the beginning), but proper handling of Unicode is practically table-stakes for a new language in today's world. I would guess (I don't know) that Swift, Go, etc also do a good job with this stuff.
- unknown_error 5y agoWhat does "table stakes" mean? Never heard that before.
- brundolf 5y ago"Baseline expectation". I think it comes from poker or something
- bombcar 5y agoTable stakes are a required minimum bet in a game (often poker) - the meaning being if you can’t provide at least this you can’t play. So if a new language doesn’t support Unicode well it won’t get any adoption.
- deleted 5y ago[deleted]
- billforsternz 5y agoIt's a gambling expression meaning the minimum [money] you need to even sit at the table. To be a really competitive player you need more.
- thurn 5y ago.chars() iterates over unicode scalar values, not actual grapheme clusters, so this code would also be incorrect in many circumstances. Ultimately it is incumbent on the developer to understand what their actual unicode-processing requirements are.
- brundolf 5y agoI'm not familiar with what exactly falls into each category, though apparently there's an (official?) crate which extends the core str type to support those as well: https://stackoverflow.com/questions/58770462/how-to-iterate-over-unicode-grapheme-clusters-in-rust/58770681#58770681 https://stackoverflow.com/questions/58770462/how-to-iterate-...
- natanbc 5y agoIt does for multi-codepoint characters (such as the rainbow flag): https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=a85e284cee100cdca7b3b6a7c229cf1e https://play.rust-lang.org/?version=stable&mode=debug&editio... Elixir gets it right: https://replit.com/@natanbc/reverse https://replit.com/@natanbc/reverse
- brundolf 5y agoYeah, another commenter mentioned grapheme-clusters, though apparently there's a readily available crate that adds support: https://stackoverflow.com/questions/58770462/how-to-iterate-over-unicode-grapheme-clusters-in-rust/58770681#58770681 https://stackoverflow.com/questions/58770462/how-to-iterate-... According to this they added it to the standard library at one point but then broke it back out because the lookup tables were too large
- pjgalbraith 5y agoHere is a working rust version https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=bdbea9d96e02af9ada180c6b4d1b4d8e https://play.rust-lang.org/?version=stable&mode=debug&editio...
- qw3rty01 5y agothis isn't quite set up the way the article was mentioning. Here's with it set up correctly: https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=d654f40995af3f657e30e9a7575885bd https://play.rust-lang.org/?version=stable&mode=debug&editio... It looks like it still misses some of the cases (specifically with decomposed vs precomposed). As a side note, it looks like the playground editor also doesn't correctly render some of the lengths (the cursor is 1 character off on the decomposed case)
- brundolf 5y agoHuh, I'd never heard of decomposed/precomposed before and I kind of skimmed over them in the original article. Looking at your example, I still can't figure out what the difference is between the two (I guess the Playground editor itself just can't represent the difference?). I'll have to go read up on that concept. Edit: Having read about them, I guess I assumed that if my keyboard inputs an ë it's actually inputting those two underlying characters. Or to put it differently, I assumed there weren't two separate representations for the same character; that the "decomposed" and "precomposed" versions are one in the same, at the byte level. Learned something h̶o̶r̶r̶i̶f̶y̶i̶n̶g̶ new!
- qw3rty01 5y agoAfter a little bit of research, it looks like this is how you would get the expected outputs: https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=217409b67e78635b4794f6bf87478e09 https://play.rust-lang.org/?version=stable&mode=debug&editio...