4 ms·

Oh I grok it just fine, it just sucks horribly. I may be missing something "pythonic" but casting between byte strings and ascii strings - a null operation - i

Oh I grok it just fine, it just sucks horribly. I may be missing something "pythonic" but casting between byte strings and ascii strings - a null operation - is by far my greatest cause of runtime bugs. To make it worse, the obvious cast is broken beyond all belief. What do we get from str(b'hello world')? Literally "b'hello world'" - a __repr__ of the object. Of course, adding an encoding changes the actual functionality of the cast ... str(b'hello world', 'ascii') gives 'hello world'. So, so broken.

Walkman 10y ago
You don't grok it. Bytes are not ASCII strings. They are NUMBERS between 0 and 255. Strings are strings. DECODING bytes with ASCII encoding into TEXT is not "casting", it's DECODING.
- Spidler 10y ago
  A thing I've found to help people about that is to override __repr__ on bytestrings in python3 and bytes in python3. If you make it not print strings as "ascii" but as hex, people grasp the difference easier.
- jstimpfle 10y ago
  I wouldn't say they are not ASCII strings, because they are if you squint. Better say "bytes are not Unicode strings".
  - Walkman 10y ago
    No, they are NOT strings, they are not any kind of strings. They are bytes: >>> b'asd'[0] 97 >>> b'asd'[1] 115 >>> b'asd'[2] 100 They happen to contain an ASCII DECODABLE string in this case, but to get the text back, you NEED TO decode it first: >>> text = b'asd'.decode() >>> text[0] 'a' >>> text[1] 's' >>> text[2] 'd'
    - jstimpfle 10y ago
      Technically true. I'd inferred a different context on this one: By "strings" I meant the abstract concept in this case, not the python str which is the one meant.
jstimpfle 10y ago
You are doing it wrong. Don't mix bytes and str. "Cast" is C lingo -- you don't "cast" completely distinct types like str and bytes. (Note: Personally I would love to have str always represented internally in UTF-8. If it were like this, I would say "cast" was acceptable vocabulary. But it's not like this). > str(b'hello world', 'ascii') gives 'hello world'. So, so broken. https://docs.python.org/3/library/stdtypes.html#str https://docs.python.org/3/library/stdtypes.html#str It's all documented and actually quite simple. By the way, I never needed this. str() is just a convenience function, it's not meant for reliability. As you surely know there is complex interplay with __str__ and __repr__. I use str() for debugging, error messages etc. But not for anything that I expect to be able to process further. How do you decode a bytes object to str object? Simple and intuitive, use .decode('utf-8')
- throwawayish 10y ago
  > (Note: Personally I would love to have str always represented internally in UTF-8. If it were like this, I would say "cast" was acceptable vocabulary. But it's not like this). UTF-8 is a good choice for transmission and storage, but since it's somewhat difficult to decode and isn't easily seekable it's not necessarily the best in-memory representation of a string. Fortunately, that's, from a Python PoV, an implementation detail (even at the C API level). > By the way, I never needed this. str() is just a convenience function, it's not meant for reliability. As you surely know there is complex interplay with __str__ and __repr__. I use str() for debugging, error message etc. But not for anything that I expect to be able to process further. Or log messages. Broadly speaking str() means "tell the object to describe itself", while repr() means "tell the object to describe itself, possibly in a way that I can eval() it"
  - jstimpfle 10y ago
    To add, I think UTF-8 is actually very easy to decode. It's true that indexing is expensive, so ideally different or additional data structures are needed for different use cases.
  - prodigal_erik 10y ago
    UTF-32 isn't easily seekable either because Unicode text has combining marks that shouldn't be separated from their base character. Almost every language has a "string" type that doesn't handle this correctly.
ageofwant 10y ago
Mate... the article was written for you. You don't 'grok' the difference between byte arrays and strings. Take a piece of paper and figure it out.
- RantyDave 10y ago
  No, really, I know it. It's a pile of bytes until it has an encoding - and I agree with this whole heartedly. I think that getting involved with 'str' is the (my) problem here.

Walkman 10y ago

You don't grok it. Bytes are not ASCII strings. They are NUMBERS between 0 and 255. Strings are strings. DECODING bytes with ASCII encoding into TEXT is not "casting", it's DECODING.

Spidler 10y ago
A thing I've found to help people about that is to override __repr__ on bytestrings in python3 and bytes in python3. If you make it not print strings as "ascii" but as hex, people grasp the difference easier.
jstimpfle 10y ago
I wouldn't say they are not ASCII strings, because they are if you squint. Better say "bytes are not Unicode strings".
- Walkman 10y ago
  No, they are NOT strings, they are not any kind of strings. They are bytes: >>> b'asd'[0] 97 >>> b'asd'[1] 115 >>> b'asd'[2] 100 They happen to contain an ASCII DECODABLE string in this case, but to get the text back, you NEED TO decode it first: >>> text = b'asd'.decode() >>> text[0] 'a' >>> text[1] 's' >>> text[2] 'd'
  - jstimpfle 10y ago
    Technically true. I'd inferred a different context on this one: By "strings" I meant the abstract concept in this case, not the python str which is the one meant.

jstimpfle 10y ago

You are doing it wrong. Don't mix bytes and str. "Cast" is C lingo -- you don't "cast" completely distinct types like str and bytes. (Note: Personally I would love to have str always represented internally in UTF-8. If it were like this, I would say "cast" was acceptable vocabulary. But it's not like this). > str(b'hello world', 'ascii') gives 'hello world'. So, so broken. https://docs.python.org/3/library/stdtypes.html#str https://docs.python.org/3/library/stdtypes.html#str It's all documented and actually quite simple. By the way, I never needed this. str() is just a convenience function, it's not meant for reliability. As you surely know there is complex interplay with __str__ and __repr__. I use str() for debugging, error messages etc. But not for anything that I expect to be able to process further. How do you decode a bytes object to str object? Simple and intuitive, use .decode('utf-8')

throwawayish 10y ago
> (Note: Personally I would love to have str always represented internally in UTF-8. If it were like this, I would say "cast" was acceptable vocabulary. But it's not like this). UTF-8 is a good choice for transmission and storage, but since it's somewhat difficult to decode and isn't easily seekable it's not necessarily the best in-memory representation of a string. Fortunately, that's, from a Python PoV, an implementation detail (even at the C API level). > By the way, I never needed this. str() is just a convenience function, it's not meant for reliability. As you surely know there is complex interplay with __str__ and __repr__. I use str() for debugging, error message etc. But not for anything that I expect to be able to process further. Or log messages. Broadly speaking str() means "tell the object to describe itself", while repr() means "tell the object to describe itself, possibly in a way that I can eval() it"
- jstimpfle 10y ago
  To add, I think UTF-8 is actually very easy to decode. It's true that indexing is expensive, so ideally different or additional data structures are needed for different use cases.
- prodigal_erik 10y ago
  UTF-32 isn't easily seekable either because Unicode text has combining marks that shouldn't be separated from their base character. Almost every language has a "string" type that doesn't handle this correctly.

ageofwant 10y ago

Mate... the article was written for you. You don't 'grok' the difference between byte arrays and strings. Take a piece of paper and figure it out.

RantyDave 10y ago
No, really, I know it. It's a pile of bytes until it has an encoding - and I agree with this whole heartedly. I think that getting involved with 'str' is the (my) problem here.