3 ms·
> Variable-length bytes is a feature that Python does not have. That wasn't my point. My point was that, as far as integer objects are concerned, there is no c
by pdonis 10y ago
> Variable-length bytes is a feature that Python does not have.
That wasn't my point. My point was that, as far as integer objects are concerned, there is no concept of "byte" at all, not even a fixed length "byte" of 8 bits. There is no concept of "chunk of data operated on as a single unit by the hardware", and there is no concept of "unit of data of the underlying storage for this object". There are just objects representing integers of any value you like.
The only kind of object in Python for which a concept of "byte" in your sense exists at all (if we leave out modules like the ctypes module) is the byte string (str in Python 2, bytes in Python 3). For those objects, you are correct that "byte" is always 8 bits in Python, and there is no way to vary it. But that has nothing to do with the underlying hardware; it is just the data model that Python has chosen for binary data.
> if you want to represent a binary data type with N instances where N>255 (e.g. an integer in a range from 0 to N where N>255, or from -N to N where N>127) then you either need bytes wider than 8 bits, or you need to worry about endianness.
Only if these things are exposed to you at all. In Python, you don't have to worry about this for integer objects because you don't see the underlying storage at all; things like what endianness is used to store integers > 255 are implementation details of the interpreter. Frankly, I find that to be a good thing: if I wanted to worry about stuff like that, I'd be programming in C, not Python. But of course different people will have different needs and preferences.
> UTF-16 and UTF-32 are encodings of unicode which use 16-bit-wide and 32-bit-wide bytes respectively to encode those code points.
I understand that you prefer to use this terminology; but my point in that particular comment was that it is not the terminology that is used in the Unicode documentation, which is the closest thing we have to an "official" terminology for Unicode. As is shown by what I quoted, that documentation clearly uses the term "byte" to mean 8 bits; it refers to UTF-16 and UTF-32 as encodings that use 2 or 4 "bytes" to encode a code point. It does not use the term "byte" to refer to 16 or 32 bit chunks of data. So it seems misleading, or at least potentially misleading, to use the term "byte" to refer to anything other than 8 bits when talking about Unicode encodings, since the Unicode documentation does not give the term "byte" that meaning.
- lisper 10y agoI think we're mostly in violent agreement here. Let me try to re-state the point I was trying to make: the only reason there is anything special about 8 bits as a unit of data is because hardware is built to operate on units of 8 bits at a time. Whether you call these things "bytes" or "words" doesn't matter. You can call them florbs for all I care. The fact that florbs are 8-bits wide is mainly for historical reasons. There is no particular reason to make the magic number be 8 (except that it's a power of 2), and indeed there have been historical examples of hardware with florbs of other sizes. The Intel 4004 had 4-bit florbs. The PDP-8 had 12-bit florbs. Modern hardware is built to natively handle florbs of multiple sizes, typically 8, 16, 32 and 64 bits, but sometimes other sizes as well (e.g. 80 bits for extended precision floats, or 128 for SSE5 instructions). Because of this, many algorithms are designed to operate on binary data in these sizes, and because of that it can be helpful to have binary data in these sizes exposed in your programming language. Python exposes 8-bit florbs, but not other sizes of florbs. Common Lisp has arbitrarily sized florbs. If you're doing certain things (like crypto) having access to florbs larger than 8 bits can be useful. That's all.
- pdonis 10y ago> the only reason there is anything special about 8 bits as a unit of data is because hardware is built to operate on units of 8 bits at a time. More precisely, hardware at the time that "the natural unit of data" was becoming a standardized convention was built to operate on units of 8 bits at a time. As you say, the 8-bit convention is mainly for historical reasons, and hardware today can handle larger chunk sizes. > Python exposes 8-bit florbs, but not other sizes of florbs To the extent it exposes florbs at all, yes. But that does not mean you are stuck with 8-bit operations in Python in all cases. For example, operations on Python floats use the IEEE standard data chunks, which, as you note, are not 8-bit. Operations on Python integers are not restricted to using 8-bit CPU instructions; they will use whatever register sizes the hardware allows (more precisely, the hardware the Python interpreter was compiled for--if you run a 32-bit Python interpreter on a 64-bit machine, you won't be using 64-bit register operations). It is true that the programmer has no control over the data chunk sizes that Python uses; they are hard-coded into the interpreter. > Common Lisp has arbitrarily sized florbs. Can you give an example of how this capability in CL is used?
- lispm 10y ago> Can you give an example of how this capability in CL is used? Common Lisp for example can extract/set a byte given its width and its position: CL-USER 145 > (write (ldb (byte 11 2) #b1100101011010110) :base 2) 1010110101 (byte 11 2) is a specifier for this: 11 = width, 2 = position. Historically this comes form a DEC PDP-10 instruction 'load byte'.
- lisper 10y agoThe best example I can think of is implementing certain cryptographic algorithms, particularly symmetric ciphers and cryptographically secure hash functions, which are often defined in terms of 32 and 64-bit chunks and mod 2^32 and mod 2^64 arithmetic. Why not just use C? Because the C language spec has "features" that allow a C compiler optimize away parts of your code that you really don't want optimized away in crypto code. For example, if you do this: int secret_key[SIZE]; ... for (int i=0; i<SIZE; i++) secret_key[i]=0; // Clear secret key so attacker can't read it A C compiler is allowed to get rid of that code if it can prove that it doesn't contribute to the final result (the "as-if" rule). Forcing a C compiler to not do such things is generally very tricky.