9 ms·

I agree that the bytes/string problem is a tough one. But the dev team created it and dropped it into the laps of it's users. The Windows API had the same prob

by 21 10y ago

I agree that the bytes/string problem is a tough one. But the dev team created it and dropped it into the laps of it's users. The Windows API had the same problem, it was ASCII and then they added Unicode. But instead of just forcing your app to exclusively use a single one of them (sort of like use Py2 OR Py3), they allowed you to mix and match your code, and call either the old or the new version of the API. And gradually, people stopped using the older version and started using the Unicode one. Today, new Windows APIs are strictly Unicode. Of course, the solution was ugly from their side, having 2 duplicate APIs for the same thing, but it didn't brought pain to their users and allowed them to proceed at leisure.

kstrauser 10y ago
So, it's like Python. You can still mix and match; you just have to explicitly convert between them at the appropriate points. The main breaking change is that you can't read a stream of bytes from a byte-oriented resource (say, a network connection that by definition can only send a stream of octets) and magically treat it like text elsewhere. While that's a tempting thing to want to do, it's been an unending fountain of bugs over the years. For example, in Python 2.7: >>> a = 'this is a test ಠ_ಠ' >>> b = bytes(a) >>> b 'this is a test \xe0\xb2\xa0_\xe0\xb2\xa0' >>> type(b) <type 'str'> In Python 3.5: >>> a = 'this is a test ಠ_ಠ' >>> print(a) this is a test ಠ_ಠ >>> b = bytes(a) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: string argument without an encoding >>> b = bytes(a.encode('utf-8')) >>> b b'this is a test \xe0\xb2\xa0_\xe0\xb2\xa0' >>> type(b) <class 'bytes'> In 2.7, you encode a series of Unicode values back to... a string? This is the source of all kinds of fun as functions expecting a string as input don't have a way to tell whether it's already been encoded or if they need to do so. In 3.5, there's no ambiguity. Functions that write to files or network sockets can either receive strings and know that they have to encode the strings locally, or bytes and know that they don't have to. Right up near the top of Zen we get: >>> import this The Zen of Python, by Tim Peters Beautiful is better than ugly. Explicit is better than implicit. The Python 2-style implicit conversions were awfully convenient 15 years ago when most things were ASCII. They're absolutely painful now that we've moved past the expectation that strings are arrays of single bytes.
- BuckRogers 10y ago
  Except Python2 is the POSIX standard on how to handle this and I prefer retro OSs without a bytes/string distinction. Python3 is the Windows way of handling strings so the change was pure churn. Both are valid, but Linux is "pretty popular" right? So why break everyone's code just to churn the pot. If it was good for Kernigan and Richie, good for Joe Ossanna, good for Robert Pike well it's good enough for me.
  - klodolph 10y ago
    There isn't a "Windows" way of handling strings. You can handle strings however you want, just make sure that you pass the right data to syscalls. Windows APIs take UTF-16 or some other code page. Linux takes bytes. OS X takes UTF-8. There is no greatest common denominator. But if you use Unicode, then at least there's a well-defined way to write the syscall wrappers on all platforms. And it's not like Python 3 is forcing you to give up bytes. You can still os.listdir(b'/some/dir') and you'll get a list of bytes objects back. It's just that that's not default, since using Unicode by default simply makes more sense on a cross-platform system.
  - jowiar 10y ago
    The problem is it might be good enough for Kernighan and Ritchie, but it's not good enough for Chávez and Çelik (or, well, most of the world). The A in ASCII stands for "American" -- and it gives a good indication of where you can look to find a whole lot of folks who ASCII isn't good enough for.
    - true_religion 10y ago
      Actually I'd have naively thought it meant America as in "The Americans" including South America. It's no wonder the IANA want's to change it into US-ASCII (or USA-SCII).
    - BuckRogers 10y ago
      Python2 supports unicode. Chávez and Çelik are covered. I'll take Python2's POSIX text model over Microsoft's default unicode strings every time because I prefer Linux. At this point in time see no reason to cater to a shrinking server platform. Not to mention they didn't even get unicode right with Python3. Google did with Go (assume UTF8).
merb 10y ago
actually the win api is broken when it comes to unicode. there is no real unicode in windows and thats a huge problem. Handling UTF-8 like on Linux is definitly akward in the Win32 API. And actually that only was caused by the fact of backwards compability. Actually it would be huge if they would force users to unicode more. But I doubt that will ever happen. We maybe see Windows Active Directory on Linux before this will happen.
- mark-r 10y ago
  It's not the API that's broken, that uses UTF-16 consistently which qualifies as Unicode by any definition. The problem really only lies in file and console encoding. If they'd simply give you the option between legacy code pages and UTF-8, and made UTF-8 the default, life would be grand.
edwinyzh 10y ago
Yep! In my earlier days as a programmer, I wondered why all win32 API functions have both the Ansi and Unicode versions? Isn't it a messy? At the end I realized it's a honorable effort to keep old Windows programs compatible. On the other hand, few years ago, when Delphi the RAD development tool, introduced their new Unicode version, completely breaking backward-compatibility, by changing the literal string type from "Ansi string" to "Unicode String". I know many third party libraries were abandoned and many projects were prevented to upgrade. It's also a disaster to me...

by 21 10y ago

kstrauser 10y ago

So, it's like Python. You can still mix and match; you just have to explicitly convert between them at the appropriate points. The main breaking change is that you can't read a stream of bytes from a byte-oriented resource (say, a network connection that by definition can only send a stream of octets) and magically treat it like text elsewhere. While that's a tempting thing to want to do, it's been an unending fountain of bugs over the years. For example, in Python 2.7: >>> a = 'this is a test ಠ_ಠ' >>> b = bytes(a) >>> b 'this is a test \xe0\xb2\xa0_\xe0\xb2\xa0' >>> type(b) <type 'str'> In Python 3.5: >>> a = 'this is a test ಠ_ಠ' >>> print(a) this is a test ಠ_ಠ >>> b = bytes(a) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: string argument without an encoding >>> b = bytes(a.encode('utf-8')) >>> b b'this is a test \xe0\xb2\xa0_\xe0\xb2\xa0' >>> type(b) <class 'bytes'> In 2.7, you encode a series of Unicode values back to... a string? This is the source of all kinds of fun as functions expecting a string as input don't have a way to tell whether it's already been encoded or if they need to do so. In 3.5, there's no ambiguity. Functions that write to files or network sockets can either receive strings and know that they have to encode the strings locally, or bytes and know that they don't have to. Right up near the top of Zen we get: >>> import this The Zen of Python, by Tim Peters Beautiful is better than ugly. Explicit is better than implicit. The Python 2-style implicit conversions were awfully convenient 15 years ago when most things were ASCII. They're absolutely painful now that we've moved past the expectation that strings are arrays of single bytes.

BuckRogers 10y ago
Except Python2 is the POSIX standard on how to handle this and I prefer retro OSs without a bytes/string distinction. Python3 is the Windows way of handling strings so the change was pure churn. Both are valid, but Linux is "pretty popular" right? So why break everyone's code just to churn the pot. If it was good for Kernigan and Richie, good for Joe Ossanna, good for Robert Pike well it's good enough for me.
- klodolph 10y ago
  There isn't a "Windows" way of handling strings. You can handle strings however you want, just make sure that you pass the right data to syscalls. Windows APIs take UTF-16 or some other code page. Linux takes bytes. OS X takes UTF-8. There is no greatest common denominator. But if you use Unicode, then at least there's a well-defined way to write the syscall wrappers on all platforms. And it's not like Python 3 is forcing you to give up bytes. You can still os.listdir(b'/some/dir') and you'll get a list of bytes objects back. It's just that that's not default, since using Unicode by default simply makes more sense on a cross-platform system.
- jowiar 10y ago
  The problem is it might be good enough for Kernighan and Ritchie, but it's not good enough for Chávez and Çelik (or, well, most of the world). The A in ASCII stands for "American" -- and it gives a good indication of where you can look to find a whole lot of folks who ASCII isn't good enough for.
  - true_religion 10y ago
    Actually I'd have naively thought it meant America as in "The Americans" including South America. It's no wonder the IANA want's to change it into US-ASCII (or USA-SCII).
  - BuckRogers 10y ago
    Python2 supports unicode. Chávez and Çelik are covered. I'll take Python2's POSIX text model over Microsoft's default unicode strings every time because I prefer Linux. At this point in time see no reason to cater to a shrinking server platform. Not to mention they didn't even get unicode right with Python3. Google did with Go (assume UTF8).

merb 10y ago

actually the win api is broken when it comes to unicode. there is no real unicode in windows and thats a huge problem. Handling UTF-8 like on Linux is definitly akward in the Win32 API. And actually that only was caused by the fact of backwards compability. Actually it would be huge if they would force users to unicode more. But I doubt that will ever happen. We maybe see Windows Active Directory on Linux before this will happen.

mark-r 10y ago
It's not the API that's broken, that uses UTF-16 consistently which qualifies as Unicode by any definition. The problem really only lies in file and console encoding. If they'd simply give you the option between legacy code pages and UTF-8, and made UTF-8 the default, life would be grand.

edwinyzh 10y ago

Yep! In my earlier days as a programmer, I wondered why all win32 API functions have both the Ansi and Unicode versions? Isn't it a messy? At the end I realized it's a honorable effort to keep old Windows programs compatible. On the other hand, few years ago, when Delphi the RAD development tool, introduced their new Unicode version, completely breaking backward-compatibility, by changing the literal string type from "Ansi string" to "Unicode String". I know many third party libraries were abandoned and many projects were prevented to upgrade. It's also a disaster to me...