4 ms·
I never said anything about bit or key length at all? Let alone how much was random or not? Perhaps you’re confused?
by lazide 3y ago
I never said anything about bit or key length at all? Let alone how much was random or not? Perhaps you’re confused?
- Dylan16807 3y agoLet's start over. People were talking about the value of time+random UUIDs versus all-random UUIDs, and how those behave. You said that sometimes the random behavior is preferable. In response to that, I was saying that even if you want to sort randomly at some particular step, you should use the time+random format, because other steps might not want to sort randomly. You should directly choose to use the random part, instead of indirectly forcing it by making the entire UUID random. Then you said "Only when writing all at once and when you know what the shard boundaries are and the number of shards (and boundaries) are stable." I can't figure out how that relates to my post. I thought you were worried about insufficient random bits to use for sharding, but apparently that wasn't your concern. So I have no idea what your concern is. If you have a use case for randomness, use the random half of the UUID.
- lazide 3y agoUUIDv7 has a specific format that doesn’t support that. For the case I’m describing, you can’t use it. For situations you want write coalescing it’s fine though. Not sure it we’re agreeing here?
- Dylan16807 3y agoWhat do you mean it doesn't support that? There's some flexibility in how you fill in a UUIDv7, but let's go ahead and say that the ones we're worried about have the first 32 bits filled with timestamp and the last 32 bits filled with random. If you want pure sort-by-time, then use it the normal way. If you want pure sort-by-random, then it's slightly awkward but you can prioritize the random part. But the additional power is that you can shard by the last 32 bits, then sort by the first 32 bits within a shard. And you don't need weird workarounds like hashing the UUID. You said "it’s easier to shard when writes occur randomly over the overall space. it’s easier to coalesce when writes all happen in a given place (head or tail)". But you can have both at the same time. You can have easy sharding and easy coalescing.
- lazide 3y agoExcept you literally can't do random distribution AND be compliant with UUIDv7 if you use any sort of normal lexical sorting/indexing, as they use the start of the key as the most significant bits. UUIDv7 is literally designed to have stable lexical sorting orders, have the time as the most significant bits, and have the most significant bits of the time as the most significant bits of the key! It's their primary design criteria! You can't 'prioritize' random parts of a key for sorting without writing a bunch of custom sorting (and key parsing) logic, which is generally undesirable for a number of reasons - and frankly completely unnecessary in these cases. You just wouldn't use UUIDv7 (or probably a UUID in general), and the benefits would pay for themselves very quickly anyway. To quote the UUIDv7 RFC: "This document presents new time-based UUID formats which are suited for use as a database key." (as the first line of the abstract) "Due to the shortcomings of UUIDv1 and UUIDv4 details so far, many widely distributed database applications and large application vendors have sought to solve the problem of creating a better time-based, sortable unique identifier for use as a database key." "- Timestamps MUST be k-sortable. That is, values within or close to the same timestamp are ordered properly by sorting algorithms. - Timestamps SHOULD be big-endian with the most-significant bits of the time embedded as-is without reordering. - Timestamps SHOULD utilize millisecond precision and Unix Epoch as timestamp source. Although, there is some variation to this among implementations depending on the application requirements. - The ID format SHOULD be Lexicographically sortable while in the textual representation. - IDs MUST ensure proper embedded sequencing to facilitate sorting when multiple UUIDs are created during a given timestamp. - IDs MUST NOT require unique network identifiers as part of achieving uniqueness. - Distributed nodes MUST be able to create collision resistant Unique IDs without a consulting a centralized resource." [https://www.ietf.org/archive/id/draft-peabody-dispatch-new-uuid-format-01.html https://www.ietf.org/archive/id/draft-peabody-dispatch-new-u...] I'm pointing out that for some systems, that makes UUIDv7 unsuitable because you WANT the keys to be randomly distributed to avoid hotspots. Using UUIDv7 in these situations will result in a single node receiving all writes (and all reads for a given time range), which in the dataset sizes I'm referring to is usually impossible to handle. No single node can handle that kind of load, regardless of how efficient it may be. For other types of systems (such as single machine databases or 'tight' clusters of databases without extreme write loads), UUIDv7 and similar is great, as it allows easy/cheap write combining when that is actually possible for a machine to handle the load.
- eru 3y agoJust explicitly use (time, random uuid) as a key in your sorting, instead of sullying your uuid with time information?
- 15155 3y agoULID schemes aren't just about big endian sorting advantages, they often better enable time-based sorting mechanisms.