3 ms·
> If you want to represent an integer >255 as fixed-width binary data But Python 3 doesn't represent integers that way, because Python 3 doesn't put any size l
by pdonis 10y ago
> If you want to represent an integer >255 as fixed-width binary data
But Python 3 doesn't represent integers that way, because Python 3 doesn't put any size limit on integers. There's no way to represent integers of potentially unbounded size as fixed-width binary data.
This is one of the things I meant by "hardware agnostic"--I didn't mean "able to represent the different register sizes of different hardware"; I meant "having a representation that doesn't have any concept of register size at all".
(Python 2 does have a separate int type, which IIRC is either 32 or 64 bits wide depending on the platform. AFAIK this is just a wrapper around the underlying C int, so it isn't affected by any 8-bit byte restriction.)
> UTF-16 and UTF-32 data is specified as 16-bit and 32-bit bytes. This is why you need a BOM.
Um, no. UTF-16 and UTF-32 data is specified as 16-bit and 32-bit code points. You need a Byte Order Mark (BOM) to specify in which order the 8-bit bytes that make up a 16-bit or 32-bit code point appear.
From the Unicode FAQ[0]:
"Q: What is a UTF?
"A: A Unicode transformation format (UTF) is an algorithmic mapping from every Unicode code point (except surrogate code points) to a unique byte sequence. The ISO/IEC 10646 standard uses the term “UCS transformation format” for UTF; the two terms are merely synonyms for the same concept."
Further down, under the table listing the UTFs and their properties:
"In the table <BOM> indicates that the byte order is determined by a byte order mark"
And in the answer to the next question after that:
"UTF-16 and UTF-32 use code units that are two and four bytes long respectively."
[0]: http://unicode.org/faq/utf_bom.html http://unicode.org/faq/utf_bom.html
- lisper 10y ago> Python 3 doesn't put any size limit on integers Yes, obviously. Variable-length bytes is a feature that Python does not have. (AFAIK the only language that has variable-length bytes is Common Lisp.) > There's no way to represent integers of potentially unbounded size as fixed-width binary data. Yes, obviously. To be excruciatingly precise I should have said: if you want to represent a binary data type with N instances where N>255 (e.g. an integer in a range from 0 to N where N>255, or from -N to N where N>127) then you either need bytes wider than 8 bits, or you need to worry about endianness. > UTF-16 and UTF-32 data is specified as 16-bit and 32-bit code points. No. Unicode characters are specified as code points, which are just numbers with no specific representation. UTF-16 and UTF-32 are encodings of unicode which use 16-bit-wide and 32-bit-wide bytes respectively to encode those code points. It is only when you serialize those encodings to octets that you need a BOM. When UTF-16 and UTF-32 are used internally to one machine you don't need a BOM. There are also, as I pointed out before, many algorithms (particularly in cryptography) that are specifically designed to operate on fixed-width binary data wider than 8 bits.
- pdonis 10y ago> Variable-length bytes is a feature that Python does not have. That wasn't my point. My point was that, as far as integer objects are concerned, there is no concept of "byte" at all, not even a fixed length "byte" of 8 bits. There is no concept of "chunk of data operated on as a single unit by the hardware", and there is no concept of "unit of data of the underlying storage for this object". There are just objects representing integers of any value you like. The only kind of object in Python for which a concept of "byte" in your sense exists at all (if we leave out modules like the ctypes module) is the byte string (str in Python 2, bytes in Python 3). For those objects, you are correct that "byte" is always 8 bits in Python, and there is no way to vary it. But that has nothing to do with the underlying hardware; it is just the data model that Python has chosen for binary data. > if you want to represent a binary data type with N instances where N>255 (e.g. an integer in a range from 0 to N where N>255, or from -N to N where N>127) then you either need bytes wider than 8 bits, or you need to worry about endianness. Only if these things are exposed to you at all. In Python, you don't have to worry about this for integer objects because you don't see the underlying storage at all; things like what endianness is used to store integers > 255 are implementation details of the interpreter. Frankly, I find that to be a good thing: if I wanted to worry about stuff like that, I'd be programming in C, not Python. But of course different people will have different needs and preferences. > UTF-16 and UTF-32 are encodings of unicode which use 16-bit-wide and 32-bit-wide bytes respectively to encode those code points. I understand that you prefer to use this terminology; but my point in that particular comment was that it is not the terminology that is used in the Unicode documentation, which is the closest thing we have to an "official" terminology for Unicode. As is shown by what I quoted, that documentation clearly uses the term "byte" to mean 8 bits; it refers to UTF-16 and UTF-32 as encodings that use 2 or 4 "bytes" to encode a code point. It does not use the term "byte" to refer to 16 or 32 bit chunks of data. So it seems misleading, or at least potentially misleading, to use the term "byte" to refer to anything other than 8 bits when talking about Unicode encodings, since the Unicode documentation does not give the term "byte" that meaning.
- lisper 10y agoI think we're mostly in violent agreement here. Let me try to re-state the point I was trying to make: the only reason there is anything special about 8 bits as a unit of data is because hardware is built to operate on units of 8 bits at a time. Whether you call these things "bytes" or "words" doesn't matter. You can call them florbs for all I care. The fact that florbs are 8-bits wide is mainly for historical reasons. There is no particular reason to make the magic number be 8 (except that it's a power of 2), and indeed there have been historical examples of hardware with florbs of other sizes. The Intel 4004 had 4-bit florbs. The PDP-8 had 12-bit florbs. Modern hardware is built to natively handle florbs of multiple sizes, typically 8, 16, 32 and 64 bits, but sometimes other sizes as well (e.g. 80 bits for extended precision floats, or 128 for SSE5 instructions). Because of this, many algorithms are designed to operate on binary data in these sizes, and because of that it can be helpful to have binary data in these sizes exposed in your programming language. Python exposes 8-bit florbs, but not other sizes of florbs. Common Lisp has arbitrarily sized florbs. If you're doing certain things (like crypto) having access to florbs larger than 8 bits can be useful. That's all.