9 ms·
Fortunately this is a bit less relevant today as Windows loses market share in database and server applications, but: UUIDs have historically massively screwed
by amluto 4y ago
Fortunately this is a bit less relevant today as Windows loses market share in database and server applications, but:
UUIDs have historically massively screwed up endian handling. While this new draft discusses sorting UUIDs as strings of octets (bytes) and the text of RFC4122 is fairly explicit about most significant bytes coming first, the C UUID structure in RFC 4122 appendix A is entirely misguided:
typedef struct {
unsigned32 time_low;
unsigned16 time_mid;
unsigned16 time_hi_and_version;
unsigned8 clock_seq_hi_and_reserved;
unsigned8 clock_seq_low;
byte node[6];
} uuid_t;
Those aren’t bytes — they’re integers of various sizes. (Hint: do not use integer types in C code for portable data structures. ntohl, etc are a mess. Just use arrays of bytes.)
I don’t know the whole history, but MS somehow took this structure at face value and caused problems like this:
https://github.com/uuid-rs/uuid/issues/277 https://github.com/uuid-rs/uuid/issues/277
So, if you want to do anything (e.g. sorting) that depends on the representation of a UUID (or even depends on converting between string and binary representations), be aware that UUIDs coming from Windows may be little-endian. In my book, this is a Windows bug, but opinions may differ here.
- blueflow 4y agoThis mishap lives forth in the UUID stored in a machines DMI data, as well in GPT partition tables, which are required when EFI is used. It would be really cool if we had some replacement for EFI that would not harbour these kind of painful legacies.
- dtech 4y agoOn the other hand, this is an easy to implement conversion, while changing such a fundamental thing from EFI sounds pretty hard, making it not worth it.
- blueflow 4y agoYou cannot fix this with an conversion because you do not know if your UUID is correct or needs conversion. DMI data for example has inconsistent endian-ness depending on the vendor. So if you have a UUID sticker on a new server, you still have two options which UUID the machine will send during PXE, either the printed UUID in big-endian encoding or in the microsoft mixed-endian encoding. Use BIOS boot instead of EFI, it has less legacy to implement: PE executables, FAT file system, Win64 ABI
- smallnamespace 4y agoIsn’t endianness marked by the variant field? 1 is IETF, 2 is Microsoft, and this can be inferred from the sticker.
- iforgotpassword 4y agoAt my last workplace we twice got new workstations where they all had the same bogus UUID. Once it was Dell, the other I don't remember. The fix was to either manually set a new one in the BIOS setup, or install a BIOS Update. So I have my doubts that the variant field can be trusted here.
- blueflow 4y agoAfter failed attempts with the UUID due to above problems, we now do machine identification for PXE via MAC address for virtual machines (you see/set it in the hypervisor) and the DMI product_serial for physical machines (written on chassis).
- Someone 4y ago> Hint: do not use integer types in C code for portable data structures. ntohl, etc are a mess. Just use arrays of bytes. I don’t see how that helps much. If a developer forgets to call ntohl on multi-byte integer fields, I don’t trust them to correctly convert said integers to arrays of bytes, either.
- kevincox 4y agoIf you write it the naive way it works. uint8_t bytes[4]; uint32_t = bytes[0] << 24 + bytes[1] << 16 + bytes[2] << 8 + bytes[3]; The endianness is whatever you write in the indexing and will be the same across architectures.
- e4m2 4y agoUnfortunately, the naive way also turns out to be wrong in C. uint8_t gets promoted to a signed int when shifting, which in turn causes undefined behavior for specific input. One way of fixing this is casting to the desired type before the shift, thus avoiding surprising conversions. On a side note, compiler warnings and sanitizers help with this kind of stuff greatly, use them if you have the option: https://godbolt.org/z/8oq9GTcze https://godbolt.org/z/8oq9GTcze
- zimmerfrei 4y ago>> uint8_t gets promoted to a signed int when shifting, which in turn causes undefined behavior for specific input. >> One way of fixing this is casting to the desired type before the shift, thus avoiding surprising conversions. Agree. For clarity, the "specific input" in the example would be a bytes[0] value larger than 127. This behavior is also explained here: https://wiki.sei.cmu.edu/confluence/display/c/INT34-C.+Do+not+shift+an+expression+by+a+negative+number+of+bits+or+by+greater+than+or+equal+to+the+number+of+bits+that+exist+in+the+operand https://wiki.sei.cmu.edu/confluence/display/c/INT34-C.+Do+no...
- firebird84 4y agoIt's actually worse than that. The first 3 groupings (textually) of the uuid might be little endian while the other 2 are big endian. Learning this cost me more time than I care to admit. https://en.wikipedia.org/wiki/Universally_unique_identifier#Variants https://en.wikipedia.org/wiki/Universally_unique_identifier#...
- cpach 4y agoWhat the…? This fact will haunt me in my dreams :-p
- throwaway9870 4y agoI had to write some EFI/GPT code earlier this year and was dumbfounded to learn this. This is right up there with the mork file format.
- formerly_proven 4y agoMaking mixed-endian even more haunted is quite the achievement. I congratulate whoever did this at Microsoft for their lasting contribution.
- amluto 4y agoThis is consistent with the misguided structure in the RFC. The first three fields (the time fields) are multibyte integers. The remainder is just bytes. The dashes in the textual representation are just there to confuse you.
- qwerty456127 4y agoWhy would this matter (except when you build an index for a big database)? Note: > 6.8. Opacity: UUIDs SHOULD be treated as opaque values and implementations SHOULD NOT examine the bits in a UUID to whatever extent is possible.
- legulere 4y agoWhen you translate between UUIDs in binary and in text form and communicate with other code the binary UUIDs. The other code might expect the uuids in a different encoding.
- quotemstr 4y agoLittle endian won. I don't see the point of maintaining theoretical compatibility with big endian systems that are at best esoteric right now and soon to be extinct. Likewise, every reasonable platform aligns data fields on natural alignment these days. It's just a waste of effort to make software portable to evolutionary dead ends.
- wolverine876 4y agoYou're saying it won for this application (UUIDs)? Universally? What is the most common remaining use of big endian?
- tech2 4y agoTCP/IP might be a reasonable example. There's a reason "network byte order" and "big endian" are the same thing.
- wolverine876 4y ago> TCP/IP might be a reasonable example. That's a pretty significant example of widespread usage that isn't going away soon. Perhaps the GGP was only referring to UUID applications.
- mort96 4y agoThis isn't about compatibility with big-endian machines. This is about compatibility between different uuid libraries, potentially on different operating systems, all on little endian CPU architectures.
- ntauthority 4y agoEvery existing library follows the standard which specifies host byte order, which usually means little endian. The Rust library cited a few levels up this chain ignored that, somehow assuming big endian, and then had to correct for this mistake.
- deleted 4y ago
- secondcoming 4y agoI thought Windows uses GUIDs, not UUIDs
- takeda 4y agoIt looks like GUID is synonymous with UUID, but the name GUID also implies that it could contain that bug/feature mentioned[1] [1] https://en.wikipedia.org/wiki/Universally_unique_identifier#Variants https://en.wikipedia.org/wiki/Universally_unique_identifier#...
- JonathonW 4y agoUUIDs (as GUIDs) in Windows predate RFC 4122, so I don't think it's that unreasonable that they're not compliant with the spec (since the exact contents of a UUID aren't typically that important, but consistency in how you produce them is). The GUID implementation in Windows derives from the DCE RPC specification [1]. That's where the multibyte integers in the RFC 4122 specification come from (they're stated the same way in the DCE RPC spec), and it doesn't explicitly specify their endianness. It does call them "NDR integers", but NDR integers can be little endian or big endian depending on implementation. DCE specifies a mechanism by which you'd indicate which you're using in an RPC call, but that data's not included in the UUID format-- you get whatever byte order the system's decided to use, which, for Windows, is little-endian. [1] https://pubs.opengroup.org/onlinepubs/9629399/apdxa.htm#tagcjh_20 https://pubs.opengroup.org/onlinepubs/9629399/apdxa.htm#tagc...
- WorldMaker 4y agoRaymond Chen's post on UUID sort orders has been indispensable to me at various points: https://devblogs.microsoft.com/oldnewthing/20190426-00/?p=102450 https://devblogs.microsoft.com/oldnewthing/20190426-00/?p=10... It's fun to note that the worst possible UUID sort order in existence isn't Microsoft's fault but Sun's. They missed the "unsigned" catch to those integers in the struct and Java sorts UUIDs as signed integers. (Which is why sometimes you'll notice in for instance Android apps Guids get sorted such that that 8000... < FFFF... < 0000... < 7FFF...)
- throw868788 4y agoHe's linked it here: https://devblogs.microsoft.com/oldnewthing/20190913-00/?p=102859 https://devblogs.microsoft.com/oldnewthing/20190913-00/?p=10.... Its a good read. The LE format, especially with UUID 7 like UUID (seq UUID's), seems to be easier/faster to sort just using binary sorting on LE architectures which is most computers nowdays. Interesting to see all these tradeoffs.