5 ms·
> a chunk of data that can be operated on as a single unit by the hardware I didn't realize that this concept was referred to by the term "byte". But this clar
by pdonis 10y ago
> a chunk of data that can be operated on as a single unit by the hardware
I didn't realize that this concept was referred to by the term "byte". But this clarifies how you are using that term.
Given that usage, I'm not sure why 8-bit bytes are an issue at all for a language like Python that is supposed to be hardware-agnostic; the whole point is that programs should not care about how the data is physically represented in the hardware. Python 3 got rid of the distinction between int and long for the same reason. The only reason the "bytes" object exists at all is to represent binary data, for example data that gets read from/written to files, network sockets, and other I/O, and AFAIK bytes in such data are always defined to be 8-bit chunks of data. (If you really insist on having Python representations of hardware data chunks, there's the ctypes module.)
- lisper 10y ago> I didn't realize that this concept was referred to by the term "byte". It is common to see "byte" used to mean 8-bit-byte, and also to see the term used for wider units of binary data. If you want to be unambiguous, the word "octet" is commonly employed to mean 8-bit byte. > a language like Python that is supposed to be hardware-agnostic That is exactly why you want wide bytes. If you want to represent an integer >255 as fixed-width binary data and you don't have wide bytes, then you have to choose an endianness convention. If you have wide bytes then you can be endianness-agnostic. > AFAIK bytes in such data are always defined to be 8-bit chunks of data This is usually true, but not always. For example, UTF-16 and UTF-32 data is specified as 16-bit and 32-bit bytes. This is why you need a BOM. > there's the ctypes module Yes, this is why I said it's not a serious problem.
- pdonis 10y ago> If you want to represent an integer >255 as fixed-width binary data But Python 3 doesn't represent integers that way, because Python 3 doesn't put any size limit on integers. There's no way to represent integers of potentially unbounded size as fixed-width binary data. This is one of the things I meant by "hardware agnostic"--I didn't mean "able to represent the different register sizes of different hardware"; I meant "having a representation that doesn't have any concept of register size at all". (Python 2 does have a separate int type, which IIRC is either 32 or 64 bits wide depending on the platform. AFAIK this is just a wrapper around the underlying C int, so it isn't affected by any 8-bit byte restriction.) > UTF-16 and UTF-32 data is specified as 16-bit and 32-bit bytes. This is why you need a BOM. Um, no. UTF-16 and UTF-32 data is specified as 16-bit and 32-bit code points. You need a Byte Order Mark (BOM) to specify in which order the 8-bit bytes that make up a 16-bit or 32-bit code point appear. From the Unicode FAQ[0]: "Q: What is a UTF? "A: A Unicode transformation format (UTF) is an algorithmic mapping from every Unicode code point (except surrogate code points) to a unique byte sequence. The ISO/IEC 10646 standard uses the term “UCS transformation format” for UTF; the two terms are merely synonyms for the same concept." Further down, under the table listing the UTFs and their properties: "In the table <BOM> indicates that the byte order is determined by a byte order mark" And in the answer to the next question after that: "UTF-16 and UTF-32 use code units that are two and four bytes long respectively." [0]: http://unicode.org/faq/utf_bom.html http://unicode.org/faq/utf_bom.html
- lisper 10y ago> Python 3 doesn't put any size limit on integers Yes, obviously. Variable-length bytes is a feature that Python does not have. (AFAIK the only language that has variable-length bytes is Common Lisp.) > There's no way to represent integers of potentially unbounded size as fixed-width binary data. Yes, obviously. To be excruciatingly precise I should have said: if you want to represent a binary data type with N instances where N>255 (e.g. an integer in a range from 0 to N where N>255, or from -N to N where N>127) then you either need bytes wider than 8 bits, or you need to worry about endianness. > UTF-16 and UTF-32 data is specified as 16-bit and 32-bit code points. No. Unicode characters are specified as code points, which are just numbers with no specific representation. UTF-16 and UTF-32 are encodings of unicode which use 16-bit-wide and 32-bit-wide bytes respectively to encode those code points. It is only when you serialize those encodings to octets that you need a BOM. When UTF-16 and UTF-32 are used internally to one machine you don't need a BOM. There are also, as I pointed out before, many algorithms (particularly in cryptography) that are specifically designed to operate on fixed-width binary data wider than 8 bits.
- pdonis 10y ago> Variable-length bytes is a feature that Python does not have. That wasn't my point. My point was that, as far as integer objects are concerned, there is no concept of "byte" at all, not even a fixed length "byte" of 8 bits. There is no concept of "chunk of data operated on as a single unit by the hardware", and there is no concept of "unit of data of the underlying storage for this object". There are just objects representing integers of any value you like. The only kind of object in Python for which a concept of "byte" in your sense exists at all (if we leave out modules like the ctypes module) is the byte string (str in Python 2, bytes in Python 3). For those objects, you are correct that "byte" is always 8 bits in Python, and there is no way to vary it. But that has nothing to do with the underlying hardware; it is just the data model that Python has chosen for binary data. > if you want to represent a binary data type with N instances where N>255 (e.g. an integer in a range from 0 to N where N>255, or from -N to N where N>127) then you either need bytes wider than 8 bits, or you need to worry about endianness. Only if these things are exposed to you at all. In Python, you don't have to worry about this for integer objects because you don't see the underlying storage at all; things like what endianness is used to store integers > 255 are implementation details of the interpreter. Frankly, I find that to be a good thing: if I wanted to worry about stuff like that, I'd be programming in C, not Python. But of course different people will have different needs and preferences. > UTF-16 and UTF-32 are encodings of unicode which use 16-bit-wide and 32-bit-wide bytes respectively to encode those code points. I understand that you prefer to use this terminology; but my point in that particular comment was that it is not the terminology that is used in the Unicode documentation, which is the closest thing we have to an "official" terminology for Unicode. As is shown by what I quoted, that documentation clearly uses the term "byte" to mean 8 bits; it refers to UTF-16 and UTF-32 as encodings that use 2 or 4 "bytes" to encode a code point. It does not use the term "byte" to refer to 16 or 32 bit chunks of data. So it seems misleading, or at least potentially misleading, to use the term "byte" to refer to anything other than 8 bits when talking about Unicode encodings, since the Unicode documentation does not give the term "byte" that meaning.