4 ms·

And that raises an obvious question: why didn’t every other programming language immediately break backwards compatibility when UTF-8 became a de facto standard

And that raises an obvious question: why didn’t every other programming language immediately break backwards compatibility when UTF-8 became a de facto standard? > And, well, if you’re already making a backward-incompatible version, here’s this checklist of other breaking changes you might as well bring along for the ride. Sorry, that doesn’t track. Treating quoted strings as UTF-8 by default instead of ASCII-or-arbitrary-bytes would have been a small migration that would not have taken over a decade to complete.

b2gills 7y ago
The way Perl dealt with this was to have you declare when you are using UTF8. # declare that the code itself is written in utf8 use utf8; my $ā = 'ā'; If you need unicode strings to work, you turn on the unicode strings feature. use feature 'unicode_strings'; Another way to turn it on: use v5.12; (Declaring which version of the language you need is something you should do anyway.) Really mostly what you have to do is declare the encodings of the file handles. # change it for all files use open ':encoding(UTF-8)'; To change it per file handle, you would use `binmode`. (Which was originally added to allow binary code to work on Windows.) open my $fh, '<', 'example.txt'; binmode $fh, ':utf8'; (Declaring the encoding of an opened file is something you should do anyway.) --- Basically Perl just defaults to the old original ways. If you need a new feature which would break old code, you just declare that you need it. Because of that, most code that was written for an earlier form of Perl still works on the latest version.
takeda 7y ago
Because many of these languages were created when Unicode already existed. Someone listed Java and Javascript, both of them started from the point that python 3 tries to bring. When python was written in 1989 Unicode didn't exist yet. As for your second argument, many people bring out Go, that had such amazing idea of using everything as UTF-8 and it works great. They don't realize that Go is pretty much doing the same thing that Python does (ignoring how the string is represented internally, since that shouldn't really be programmer's concern). Go clearly distinguishes between string (string type) and bytes ([]byte type) to use string as bytes you have to cast it to []byte and to convert bytes to string you need to cast them to string. That's the equivalent of doing variable.encode() to get bytes and you do variable.decode() to get a string. What python 3 inroduced is two types str and bytes, and blocked any implicit casting between them. That's exactly same thing Go does. The only difference is implementation detail, Go stores strings as utf-8 and casting doesn't require any work, they are just for having compiler catch errors it also ignores environment variables and always uses utf-8. Python has an internal[1] representation and does do conversion. It respects LANG and other variables and uses that for stdin/out/err. Initially when those variables were undefined it assumed us-ascii which created some issues, but I believe now that was fixed and utf-8 is the default. [1] Python 3 actually tries to be smart and uses UCS1 (Latin 1), UCS2 or UCS4 depending what characters are contained. If an UTTF-8 conversion was requested it will also cache that representation (as a C-string) so it won't do the conversion next time.
- philwelch 7y ago
  > Because many of these languages were created when Unicode already existed. Someone listed Java and Javascript, both of them started from the point that python 3 tries to bring. That was me in a parallel thread. Java and JavaScript internally use UTF-16 encoding. I also mentioned C, which treats strings as byte arrays, and C++, which supports C strings as well as introducing a string class that is still just byte arrays. > As for your second argument, many people bring out Go, that had such amazing idea of using everything as UTF-8 and it works great. Has Go ever broken backwards compatibility? Let me clarify my second argument: if you are going to break backwards compatibility, you should do so in a minimal way that eases the pain of migration. The Python maintainers decided that breaking backwards compatibility meant throwing in the kitchen sink, succumbing to second system effect, and essentially forking the language for over a decade. The migration from Ruby 1.8 to 1.9 was less painful, though in fairness I suppose the migration from Perl 5 to Perl 6 was even more painful.
  - takeda 7y ago
    > That was me in a parallel thread. Java and JavaScript internally use UTF-16 encoding. I also mentioned C, which treats strings as byte arrays, and C++, which supports C strings as well as introducing a string class that is still just byte arrays. C and C++ doesn't really have Unicode support, and most C and C++ applications don't support unicode. There are libraries that you need to use to get this kind of support. > Has Go ever broken backwards compatibility? Let me clarify my second argument: if you are going to break backwards compatibility, you should do so in a minimal way that eases the pain of migration. The Python maintainers decided that breaking backwards compatibility meant throwing in the kitchen sink, succumbing to second system effect, and essentially forking the language for over a decade. The migration from Ruby 1.8 to 1.9 was less painful, though in fairness I suppose the migration from Perl 5 to Perl 6 was even more painful. Go is only 10 years old Python is 31. And in fact it had some breaking changes for example in 1.4, 1.12. Those are easy to fix since they would show up during compilation. Python is a dynamic language and unless you use something like mypy you don't have that luxury. Going back to python, what was broken in Python 2 is that str type could represent both text and bytes, and the difficulty was that most Python 2 applications are broken (yes they worked fine with ascii text but broke in interesting ways whenever unicode was used. You might say, so what, why should I care if I don't use Unicode. The problem was that mixing these two types and implicit casting that python 2 did made it extremely hard to write correct code even when you know what you're doing. With python 3 is no effort. There is a good write up by one of Python developers why python 3 was necessary[1]. [1] https://snarky.ca/why-python-3-exists/ https://snarky.ca/why-python-3-exists/

b2gills 7y ago

The way Perl dealt with this was to have you declare when you are using UTF8. # declare that the code itself is written in utf8 use utf8; my $ā = 'ā'; If you need unicode strings to work, you turn on the unicode strings feature. use feature 'unicode_strings'; Another way to turn it on: use v5.12; (Declaring which version of the language you need is something you should do anyway.) Really mostly what you have to do is declare the encodings of the file handles. # change it for all files use open ':encoding(UTF-8)'; To change it per file handle, you would use `binmode`. (Which was originally added to allow binary code to work on Windows.) open my $fh, '<', 'example.txt'; binmode $fh, ':utf8'; (Declaring the encoding of an opened file is something you should do anyway.) --- Basically Perl just defaults to the old original ways. If you need a new feature which would break old code, you just declare that you need it. Because of that, most code that was written for an earlier form of Perl still works on the latest version.

takeda 7y ago

Because many of these languages were created when Unicode already existed. Someone listed Java and Javascript, both of them started from the point that python 3 tries to bring. When python was written in 1989 Unicode didn't exist yet. As for your second argument, many people bring out Go, that had such amazing idea of using everything as UTF-8 and it works great. They don't realize that Go is pretty much doing the same thing that Python does (ignoring how the string is represented internally, since that shouldn't really be programmer's concern). Go clearly distinguishes between string (string type) and bytes ([]byte type) to use string as bytes you have to cast it to []byte and to convert bytes to string you need to cast them to string. That's the equivalent of doing variable.encode() to get bytes and you do variable.decode() to get a string. What python 3 inroduced is two types str and bytes, and blocked any implicit casting between them. That's exactly same thing Go does. The only difference is implementation detail, Go stores strings as utf-8 and casting doesn't require any work, they are just for having compiler catch errors it also ignores environment variables and always uses utf-8. Python has an internal[1] representation and does do conversion. It respects LANG and other variables and uses that for stdin/out/err. Initially when those variables were undefined it assumed us-ascii which created some issues, but I believe now that was fixed and utf-8 is the default. [1] Python 3 actually tries to be smart and uses UCS1 (Latin 1), UCS2 or UCS4 depending what characters are contained. If an UTTF-8 conversion was requested it will also cache that representation (as a C-string) so it won't do the conversion next time.

philwelch 7y ago
> Because many of these languages were created when Unicode already existed. Someone listed Java and Javascript, both of them started from the point that python 3 tries to bring. That was me in a parallel thread. Java and JavaScript internally use UTF-16 encoding. I also mentioned C, which treats strings as byte arrays, and C++, which supports C strings as well as introducing a string class that is still just byte arrays. > As for your second argument, many people bring out Go, that had such amazing idea of using everything as UTF-8 and it works great. Has Go ever broken backwards compatibility? Let me clarify my second argument: if you are going to break backwards compatibility, you should do so in a minimal way that eases the pain of migration. The Python maintainers decided that breaking backwards compatibility meant throwing in the kitchen sink, succumbing to second system effect, and essentially forking the language for over a decade. The migration from Ruby 1.8 to 1.9 was less painful, though in fairness I suppose the migration from Perl 5 to Perl 6 was even more painful.
- takeda 7y ago
  > That was me in a parallel thread. Java and JavaScript internally use UTF-16 encoding. I also mentioned C, which treats strings as byte arrays, and C++, which supports C strings as well as introducing a string class that is still just byte arrays. C and C++ doesn't really have Unicode support, and most C and C++ applications don't support unicode. There are libraries that you need to use to get this kind of support. > Has Go ever broken backwards compatibility? Let me clarify my second argument: if you are going to break backwards compatibility, you should do so in a minimal way that eases the pain of migration. The Python maintainers decided that breaking backwards compatibility meant throwing in the kitchen sink, succumbing to second system effect, and essentially forking the language for over a decade. The migration from Ruby 1.8 to 1.9 was less painful, though in fairness I suppose the migration from Perl 5 to Perl 6 was even more painful. Go is only 10 years old Python is 31. And in fact it had some breaking changes for example in 1.4, 1.12. Those are easy to fix since they would show up during compilation. Python is a dynamic language and unless you use something like mypy you don't have that luxury. Going back to python, what was broken in Python 2 is that str type could represent both text and bytes, and the difficulty was that most Python 2 applications are broken (yes they worked fine with ascii text but broke in interesting ways whenever unicode was used. You might say, so what, why should I care if I don't use Unicode. The problem was that mixing these two types and implicit casting that python 2 did made it extremely hard to write correct code even when you know what you're doing. With python 3 is no effort. There is a good write up by one of Python developers why python 3 was necessary[1]. [1] https://snarky.ca/why-python-3-exists/ https://snarky.ca/why-python-3-exists/