5 ms·
Like anything new, people had a hard time with it at the beginning. I remember that I got a home assignment in an interview for a PHP job. The person evaluatin
by BiteCode_dev 5y ago
Like anything new, people had a hard time with it at the beginning.
I remember that I got a home assignment in an interview for a PHP job. The person evaluating my code said I should not have used UTF8, which causes "compatibility problems". At the time, I didn't know better, and I answered that no, it was explicitly created to solve compatibility problems, and that they just didn't understand how to deal with encoding properly.
Needing-less to say, I didn't get the job :)
Same with Python 2 code. So many people, when migrating to Python 3, suddenly though python 3 encoding management was broken, since it was raising so many UnicodeDecodingError.
Only much later people realize the huge number of programs that couldn't deal with non ASCII characters in file paths, html attributes or user names, because they just implicitly assume ASCII. "My code used to work fine", they said. But it worked fine on their machine, set to an english locale, tested only using ascii plain text files on their ascii named directories with their ascii last name.
- andrepd 5y agoThat's in general a problem with dynamic languages with weak type systems. How "Your code runs without crashing" is really really != "your code works". How do people even manage production python! A bug could be lurking anywhere, undetected until it's actually run. Whereas in a compiled language with a strong type system, "your code compiles" is much closer to "your code is correct".
- deleted 5y ago[deleted]
- digisign 5y agoThere are a number of mitigations, so those kind of bugs are quite rare. In our large code base, about 98% of bugs we find are of the "we need to handle another case" variety. Pyflakes quickly finds typos which eliminates most of the rest.
- wk_end 5y agoThis is the difference between people who embrace static typing and everyone else. A static type lover hears that 98% of your bugs are of the "we need to handle another case" variety and says, "well, that means you could have gotten rid of 98% of your bugs with better typing".
- digisign 5y agoNo, what I mean is that an additional key comes in (with the json or similar hash) and we now need to do some thing with it, or something different than we thought we were supposed to with it. Typing is not going to fix it because the full cases were unknown at development time.
- BiteCode_dev 5y agoI don't think a type system can help you with decoding a file with the wrong charset.
- pvorb 5y agoThe compiler can raise a warning about bad characters. It can't detect all problems, but it certainly can help with some. It's not the type system, but many dynamic languages are interpreted instead of compiled.
- dotancohen 5y agoThose bad characters are fed into the program long long after the compiler has done its job.
- pvorb 5y agoNot necessarily? I'm talking about constant string literals in source code which can be validated at compile time.
- morelisp 5y agoA type system can refuse to turn a `bytes` into a `utf8str` until it's been appropriately parsed. (It doesn't even need to be a very good or strongly-enforced type system - Go makes it dangerously easy to convert between `[]byte` and `string` by other-type-system standards, and yet everything works pretty well. It's enough to hitch your thinking and make you realize you need another step.)
- BiteCode_dev 5y agoHow does the type system knows the bytes are utf8 and not CP850?
- morelisp 5y agoThe same way you can parse a string into an integer and it knows you got an either[integer, err].
- mekster 5y ago> "your code compiles" is much closer to "your code is correct". That is so far from the truth.
- ben-schaaf 5y agoHow is it anything but the truth? The express purpose of static analysis, like a type system, is to catch bugs before running your code. That pretty clearly means that code that successfully compiles is closer to being correct than code that doesn't.
- mekster 5y agoI mean, correct as in logically correct and that's just a different matter than being grammatically correct.
- morelisp 5y agoThe parser assures your code is grammatically correct; the type system assures your code is semantically consistent, which is usually a much stronger guarantee, and by most practical measures will be closer - often much closer, and for total functions on total types, sometimes all the way - to "logically correct".
- cafard 5y agoBut this is not a matter of UTF-8 in the code, rather a matter of UTF-8 in the input or output. How does compiling a program ensure that it is robust on a range of inputs?
- morelisp 5y ago> How does compiling a program ensure that it is robust on a range of inputs? This is quite literally the job of a type system: to impose a semantic interpretation on sequences of "raw bits" and let you specify legal (and only legal) operations in terms of the semantic interpretation rather than the bits.
- morelisp 5y agoPython 3 encoding management was broken, because it tried to impose Unicode semantics on things that were actually byte streams. For anyone actually correctly handling encodings in Python 2 it was awful because suddenly the language runtime was hiding half the data you needed.
- BiteCode_dev 5y agoNowadays, passing bytes to any os function returns bytes objects, not unicode. You'll get string if you pass string objects though, and they will be using utf8 surrogate escaping.
- morelisp 5y agoAnd nowadays, lots of people left the Python ecosystem completely because the 3 upgrade was broken and raising so many UnicodeDecodingErrors for so long. I'm glad it's fixed but it cost too much. (And it had nothing to do with UTF-8. Actually, it's at least partially caused by the CPython developers avoiding UTF-8 for poor reasons.)
- mark-r 5y agoI'd argue that the people correctly handling encodings in Python 2 were vastly outnumbered by the people that weren't, but were getting away with it because the code didn't outright crash. Now in Python 3 it crashes, which is a pain in the short term but better in the long run.
- morelisp 5y agoI don't really think it's better - certainly it was not better until many years after release when they started admitting their big mistakes. But even today, codepoint indexing is too easy and doesn't crash so lots of code is still subtly wrong. Memory usage of a string grows 3x if you add a single emoji. Most libraries are still agnostic about whether they take bytes or str so you still get exceptions thrown with no easy solution. (The growing popularity of type hints is fixing this, but that's not really related to Python 3.)
- SAI_Peregrinus 5y agoMy Slack name at work is "Τĥιs ñåmè įß ą váĺîδ POSIX paτĥ". My hope is that it serves as an amusing reminder to consider things like spaces and non-ASCII characters.
- tialaramex 5y agoOne of my friends is these days a colleague, with an utterly ordinary English name but his identity management data is full of spurious accents to check APIs do the Right Thing™. I was delighted recently to stumble on a history of modern women course HIST1158 named Liberté Egalité Beyoncé and I immediately thought two things: 1. Why are our Computer Science courses given unimaginative names? and 2. What a useful test input, I bet some of our systems don't work correctly for this input even though an acute accent is hardly a bleeding edge feature. I haven't been able to interest any Computer Science professors in fun names for their courses, but I was able in my test environment to name a COMP series course "Untitled Course Name" with a description explaining that "It is a lovely day in the village and there are only two hard problems in Computer Science".
- xorcist 5y ago