5 ms·

Are bytes... lesser known, or an especially advanced topic? I didn't think so, but then interviewing Juniors (who all invariably have "expert python" on their

Are bytes... lesser known, or an especially advanced topic? I didn't think so, but then interviewing Juniors (who all invariably have "expert python" on their CVs), nobody ever seems to know them, or anything about string handling/unicode. Are the basics about how stuff is moved around inside computers just completely opaque to people learning to program nowadays? I learned that stuff at secondary (high) school, so internally my brain just counts it as "foundational knowledge".

dymk 2y ago
Certainly not an advanced topic, but lesser known (compared to string or list) is probably an accurate description. This article seems like it's aimed at Python learners, rather than experienced programmers.
- abracadaniel 2y ago
  It's still a good reminder anyway. It's not something you often have to worry about. I just ran into this distinction dealing with handling raw http data. Was trying to check for string value in a byte string and took a bit to realize I was effectively doing: 'test' == b'test' which is, of course, False.
- sevensor 2y ago
  I've noticed I have to drastically recalibrate when I'm looking at Python learning materials. I'm trying to find something good for onboarding people, but even the "advanced content" tends to be stuff like this. I'd consider advanced to be more along the lines of "your function returns a generator, here's how to write the type annotation," or "worked examples showing how to express computations using numpy so that your program runs 30x faster," or "here's how to write a context manager."
  - physicsguy 2y ago
    At the same time, NumPy is so ubiquitous in science, I wouldn't even consider that intermediate, it falls under basics. Really it's just a case of Python being able to handle a ton of different use cases that sometimes overlap, you can be really deep into it in one area but never touch another.
DiggyJohnson 2y ago
Driveby answer to your question, sorry I can't expand: yes, for the most part. Even when these topics are covered directly in a university course, this knowledge doesn't stick around to become "foundational". Especially for those that don't have a hardware, embedded, or otherwise resource-constrained background or area of interest.
ok123456 2y ago
Raw byte string literals (b'') and their type in the language are details you can be unaware of if you didn't live through the Python 2->3 transition or don't deal with binary data buffers.
- ziml77 2y ago
  I was surprised that bytes were a lesser known type in Python, but now that I think about what you're saying I guess it's true. The bytes type doesn't get surfaced if you're only working with higher level libraries that do the bytes to str conversion for you.
  - dfox 2y ago
    I used to think that only various external libraries (eg. Flask) do various hidden conversion to shield the user from the str-bytes distinction, but even the builtin MIME implementation does that kind of stuff, which means that it will break 8bit S/MIME messages (and when you get bitten by that hard, and just go off implementing your own MIME parser the fact that bytes has all the string methods like .split() comes very handy).
    - nerdponx 2y ago
      Just about every I/O interface in the Python standard library does it: env vars, reading data in files, reading stdin, even filenames themselves.
      - dfox 2y ago
        There is a difference: packet on a wire is inherently bytes, while interaction with the OS is a can of worms. On UNIX all that are just an opaque byte blobs, but with expectation that these things mean something to humans, so it either are strings in the locale encoding or UTF-8 encoded strings, then there is Windows where this kind of interfaces use something like UTF-16. In general, the name of the thing that you use to interact with the OS probably is a meaningful string of Unicode code points, but it does not have to be so and you should have some kind of plan what to do when it is not. Doing that in a way that abstracts away the conventions of the underlying OS is almost certainly impossible.
janalsncm 2y ago
No, not really. Most of the article discusses utf8 encoding which isn’t an “advanced” topic either but I never took the time to learn it. > who all invariably have “expert python” on their CVs This is to get around HR filters, where they ask your experience with various tools on a 1-10 scale. If you’re only seeing the “experts” it’s possible the people who used more reasonable assessments of their abilities were simply rejected.
Animats 2y ago
They're an artifact of the way Python 3 went to Unicode. Early Python was ASCII-oriented. Type "str" was used both for text strings and arrays of bytes. "str" was treated much like arrays of "char" in C - it was the basic type for binary I/O. Then came Unicode and Python 3. Characters and bytes were no longer the same thing. Strings and arrays of bytes had to be split somehow. As with most languages that had to retrofit Unicode, this didn't go well. Rust, which didn't have a retrofit problem, did a clean separation. There are arrays of u8, and there is "str", which must be valid UTF-8 sequences. There is no implicit conversion. This is straightforward. Python didn't do it that way. Type "bytes" in Python 3 prints as an ASCII string, not an array of numbers. It's close to the 'str" from Python 2. The usual string-like operations, such as "split" are defined for "bytes". This was kind of weird but was supposed to simplify conversion of old code from Python 2 to Python 3. It didn't really help all that much. There's also "bytearray", which is a mutable version of "bytes". It's one of those messes left over from the ASCII to Unicode transition era, along with UTF-16, byte order marks, "wchar_t", HTML character sets, HTTP headers, and all the character representations in SQL.
- otabdeveloper4 2y ago
  Python2 has the type "str" (exactly equivalent to python3's "bytes"), and the type "unicode" (exactly equivalent to python3's "str"). The whole decade-long mess was just to fix quirks in Windows' path and file encoding handling bullshit. And they didn't even manage to fix the problem in the end. What a mess.
pgwhalen 2y ago
There is a wide gap between "known to juniors/mid level/low senior programmers" and "interesting enough to post on hacker news," which often surprises people. People can be productive for a long time having no knowledge of concepts that HN readers deem essential.
- throwaway894345 2y ago
  In most of the Python shops I’ve worked in, people have no idea how async actually works (and why they shouldn’t make sync I/O calls in their endpoints) or even the basics of performance (how Python allocates memory, how to dodge the GIL without eating a bunch of pickling costs, etc). Never mind stuff like import path rules, metaclasses, packaging, etc. It often feels like the median Python (or JavaScript for that matter) user has a very different mindset than Rust, Go, C, etc programmers (in my experience, Python programmers also are much less likely to have used other languages extensively).
  - physicsguy 2y ago
    This has been my experience too, I came from a HPC scientific background and ended up doing Django after leaving academia (for an engineering company, so still pretty maths-y). When I was hiring I really had to reset expectations to some degree about whether people had the capacity to learn this stuff rather than know it already.
  - DontchaKnowit 2y ago
    I think a lot of the times python is used almost exclusively for dead simple batch tasks where this level of detail is just not remotely needed.
    - throwaway894345 2y ago
      Sure, but I've never worked somewhere that employed people to write "dead simple batch tasks". If you're writing async endpoints, it feels reasonable to assume that people know how an event loop works (or at least not to make sync calls in an async function).
throwaway894345 2y ago
I’ve been writing Python for 20 years, and I’m nowhere near “expert”. I don’t think people realize how incredibly complex Python is. Things like metaclasses and import rules alone are mind bogglingly complex. After 5 years of Go I might reasonably have been called an expert (I started using Go in 2012)—there wasn’t much left in the language to learn apart from the things that had changed. But with Python there’s tons and tons left for me to learn even ignoring all of the changes.
- physicsguy 2y ago
  I think Go is much smaller than Python though; the standard library is tiny in comparison. Just looking at this list of modules: https://docs.python.org/3/library/index.html https://docs.python.org/3/library/index.html I reckon I've used at least once probably 50% of the list, but that's over 12 years of using Python, and I'd only consider myself fluent with a very small number of the total.
  - throwaway894345 2y ago
    I was remarking about core language features (the runtime, import resolution, the runtime type system, packaging, C extensions, etc). I was absolutely not talking about proficiency with the standard library in either case.

dymk 2y ago

Certainly not an advanced topic, but lesser known (compared to string or list) is probably an accurate description. This article seems like it's aimed at Python learners, rather than experienced programmers.

abracadaniel 2y ago
It's still a good reminder anyway. It's not something you often have to worry about. I just ran into this distinction dealing with handling raw http data. Was trying to check for string value in a byte string and took a bit to realize I was effectively doing: 'test' == b'test' which is, of course, False.
sevensor 2y ago
I've noticed I have to drastically recalibrate when I'm looking at Python learning materials. I'm trying to find something good for onboarding people, but even the "advanced content" tends to be stuff like this. I'd consider advanced to be more along the lines of "your function returns a generator, here's how to write the type annotation," or "worked examples showing how to express computations using numpy so that your program runs 30x faster," or "here's how to write a context manager."
- physicsguy 2y ago
  At the same time, NumPy is so ubiquitous in science, I wouldn't even consider that intermediate, it falls under basics. Really it's just a case of Python being able to handle a ton of different use cases that sometimes overlap, you can be really deep into it in one area but never touch another.

DiggyJohnson 2y ago

Driveby answer to your question, sorry I can't expand: yes, for the most part. Even when these topics are covered directly in a university course, this knowledge doesn't stick around to become "foundational". Especially for those that don't have a hardware, embedded, or otherwise resource-constrained background or area of interest.

ok123456 2y ago

Raw byte string literals (b'') and their type in the language are details you can be unaware of if you didn't live through the Python 2->3 transition or don't deal with binary data buffers.

ziml77 2y ago
I was surprised that bytes were a lesser known type in Python, but now that I think about what you're saying I guess it's true. The bytes type doesn't get surfaced if you're only working with higher level libraries that do the bytes to str conversion for you.
- dfox 2y ago
  I used to think that only various external libraries (eg. Flask) do various hidden conversion to shield the user from the str-bytes distinction, but even the builtin MIME implementation does that kind of stuff, which means that it will break 8bit S/MIME messages (and when you get bitten by that hard, and just go off implementing your own MIME parser the fact that bytes has all the string methods like .split() comes very handy).
  - nerdponx 2y ago
    Just about every I/O interface in the Python standard library does it: env vars, reading data in files, reading stdin, even filenames themselves.
    - dfox 2y ago
      There is a difference: packet on a wire is inherently bytes, while interaction with the OS is a can of worms. On UNIX all that are just an opaque byte blobs, but with expectation that these things mean something to humans, so it either are strings in the locale encoding or UTF-8 encoded strings, then there is Windows where this kind of interfaces use something like UTF-16. In general, the name of the thing that you use to interact with the OS probably is a meaningful string of Unicode code points, but it does not have to be so and you should have some kind of plan what to do when it is not. Doing that in a way that abstracts away the conventions of the underlying OS is almost certainly impossible.

janalsncm 2y ago

No, not really. Most of the article discusses utf8 encoding which isn’t an “advanced” topic either but I never took the time to learn it. > who all invariably have “expert python” on their CVs This is to get around HR filters, where they ask your experience with various tools on a 1-10 scale. If you’re only seeing the “experts” it’s possible the people who used more reasonable assessments of their abilities were simply rejected.

Animats 2y ago

They're an artifact of the way Python 3 went to Unicode. Early Python was ASCII-oriented. Type "str" was used both for text strings and arrays of bytes. "str" was treated much like arrays of "char" in C - it was the basic type for binary I/O. Then came Unicode and Python 3. Characters and bytes were no longer the same thing. Strings and arrays of bytes had to be split somehow. As with most languages that had to retrofit Unicode, this didn't go well. Rust, which didn't have a retrofit problem, did a clean separation. There are arrays of u8, and there is "str", which must be valid UTF-8 sequences. There is no implicit conversion. This is straightforward. Python didn't do it that way. Type "bytes" in Python 3 prints as an ASCII string, not an array of numbers. It's close to the 'str" from Python 2. The usual string-like operations, such as "split" are defined for "bytes". This was kind of weird but was supposed to simplify conversion of old code from Python 2 to Python 3. It didn't really help all that much. There's also "bytearray", which is a mutable version of "bytes". It's one of those messes left over from the ASCII to Unicode transition era, along with UTF-16, byte order marks, "wchar_t", HTML character sets, HTTP headers, and all the character representations in SQL.

otabdeveloper4 2y ago
Python2 has the type "str" (exactly equivalent to python3's "bytes"), and the type "unicode" (exactly equivalent to python3's "str"). The whole decade-long mess was just to fix quirks in Windows' path and file encoding handling bullshit. And they didn't even manage to fix the problem in the end. What a mess.

pgwhalen 2y ago

There is a wide gap between "known to juniors/mid level/low senior programmers" and "interesting enough to post on hacker news," which often surprises people. People can be productive for a long time having no knowledge of concepts that HN readers deem essential.

throwaway894345 2y ago
In most of the Python shops I’ve worked in, people have no idea how async actually works (and why they shouldn’t make sync I/O calls in their endpoints) or even the basics of performance (how Python allocates memory, how to dodge the GIL without eating a bunch of pickling costs, etc). Never mind stuff like import path rules, metaclasses, packaging, etc. It often feels like the median Python (or JavaScript for that matter) user has a very different mindset than Rust, Go, C, etc programmers (in my experience, Python programmers also are much less likely to have used other languages extensively).
- physicsguy 2y ago
  This has been my experience too, I came from a HPC scientific background and ended up doing Django after leaving academia (for an engineering company, so still pretty maths-y). When I was hiring I really had to reset expectations to some degree about whether people had the capacity to learn this stuff rather than know it already.
- DontchaKnowit 2y ago
  I think a lot of the times python is used almost exclusively for dead simple batch tasks where this level of detail is just not remotely needed.
  - throwaway894345 2y ago
    Sure, but I've never worked somewhere that employed people to write "dead simple batch tasks". If you're writing async endpoints, it feels reasonable to assume that people know how an event loop works (or at least not to make sync calls in an async function).

throwaway894345 2y ago

I’ve been writing Python for 20 years, and I’m nowhere near “expert”. I don’t think people realize how incredibly complex Python is. Things like metaclasses and import rules alone are mind bogglingly complex. After 5 years of Go I might reasonably have been called an expert (I started using Go in 2012)—there wasn’t much left in the language to learn apart from the things that had changed. But with Python there’s tons and tons left for me to learn even ignoring all of the changes.

physicsguy 2y ago
I think Go is much smaller than Python though; the standard library is tiny in comparison. Just looking at this list of modules: https://docs.python.org/3/library/index.html https://docs.python.org/3/library/index.html I reckon I've used at least once probably 50% of the list, but that's over 12 years of using Python, and I'd only consider myself fluent with a very small number of the total.
- throwaway894345 2y ago
  I was remarking about core language features (the runtime, import resolution, the runtime type system, packaging, C extensions, etc). I was absolutely not talking about proficiency with the standard library in either case.