6 ms·
This idea that some developers don't need to deal with unicode, really needs to die. If you're creating a web app and don't support unicode input, you're doing
by gnud 10y ago
This idea that some developers don't need to deal with unicode, really needs to die.
If you're creating a web app and don't support unicode input, you're doing it wrong. You probably can't accurately store the names of your customers, for starters.
- reuven 10y agoOP here. I teach Python to Fortune 100 companies, more or less every day. You're totally right that people who deal with outside-world data need Unicode. Anything that deals with users MUST deal with Unicode. But a huge proportion of the people I work with -- in Israel, China, and Europe -- couldn't care less about Unicode. That's because they're processing logfiles, dealing with test automation and system administration. For them, Unicode is important a very small proportion of the time. So I stand by my assertion that a small proportion of developers need to worry about Unicode. But I also agree with you that Unicode is absolutely crucial in the modern world, and that a language lacking good Unicode support isn't going to get very far.
- orf 10y ago> That's because they're processing logfiles, dealing with test automation and system administration ... So I stand by my assertion that a small proportion of developers need to worry about Unicode. You come to this conclusion based on your dealings with a small subset of developers within a small subset of companies? I think many other developers in the fortune 500 companies outside of the subset you deal with would laugh at that conclusion, then go back to developing apps that deal with people/locales all over the world.
- reuven 10y agoFair enough. My sample is from lots of companies, but a small subset of tasks there.
- falcolas 10y agoThis also describes my entire team. And the team before that. There are at least as many developers out there who don't write user facing code as those who do. When I do run across Unicode, it's just a matter of passing it through, which is usually pretty easy to do.
- selectnull 10y agoJust because your users are dealing with logfiles, their users should still be able to use č ƒ ø or whatever unicode character in the filename or url - which will consequently end up in a log file.
- viraptor 10y agoMost logs are actually bytestrings. If they include user-controlled strings, they will likely have to be parsed as binary blobs to accommodate that.
- 1ris 10y agoMany logs are localised - and contain unicode.
- viraptor 10y agoThere's a difference though. Does the log format itself in some unicode encoding, or does it just embed some utf-X encoded strings? If you just embed some user input and assume it will decode properly, you're likely to run into issues at some point when someone provides explicitly-broken encoding.
- jcranmer 10y agoTrue story: I once broke build automation by including non-ASCII text in the commit message. Just because something is internal doesn't mean that it won't have Unicode.
- burnbabyburn 10y agoyes, in webapps you almost always import from future unicode literals in py2 apps.
- rwallace 10y agoMany programmers don't need to deal with Unicode simply because they aren't dealing with natural language text in the first place. Programs that need to support Unicode input are precisely the ones that have natural language text input.
- herge 10y agoPro tip: Use non-ascii characters in your test data. Instead of John Smith, try Helmut Müller. This will make sure your code is unicode safe, and also make sure that external dependencies (hey, our spreadsheet library silently fails when a ü is in the file name!) also work correctly with unicode.
- dom0 10y agoGerman Umlauts are not a good test vector, because various SBCS have them as well (latin-1 and others), and can slip through systems not correctly processing Unicode. Use CJK instead, eg. 나윤선, which will just explode into an obvious gooey mess if the processing pipeline isn't handling Unicode correctly somewhere.
- herge 10y agoI use a site like http://lingojam.com/FancyTextGenerator http://lingojam.com/FancyTextGenerator to generate 𝖚𝖘𝖊𝖋𝖚𝖑 𝖙𝖊𝖘𝖙 𝖉𝖆𝖙𝖆. The only downside is that it has taught me that my favourite debugger is not unicode safe :(
- WorldMaker 10y agoAlso, you can litter your test data with all the fancy emoji you can type in a modern OS these days, those are almost always instantly recognizable when correct versus incorrect and test a fun number of things given: 1) Most (but not all) emoji exist way out in the Astral Plane (a good stress test for UTF-8/UTF-16 handling and little/big endian issues) 2) Emoji can contain "exotic" sequences like Zero Width Joiners (this can be very important to avoid codepoint counting issues because a ZWJ sequence of codepoints is a very different emoji if the codepoints are incorrectly reversed or spliced) 3) Emoji can test your font display stack for inefficiencies in system font fallback (and support for modern typography in general)
- Tempest1981 10y agoI like to use "Tokyo" in kanji, since I can remember what the characters look like: 東京 Is that a good test?
- GFK_of_xmaspast 10y agoI do primarily numerical work with a little bit of embedded mixed in, it's been closing in on three years since I encountered anything involving non-ascii characters.