5 ms·

Fast cryptographically safe GUID generator for Go

20 points by sdrapkin 1y ago

sdrapkin 1y ago
Much faster (~10x) than standard github.com/google/uuid package I'm interested in feedback from the HN community.
- throwaway894345 1y ago
  Why is it so much faster than `uuid`?
  - sdrapkin 1y ago
    It generates entropy 4kb-at-a-time (instead of on each call), and uses a cache-pool instead of single cache behind a lock (which is what standard uuid does in "RandPool=ON" mode).
    - cyberax 1y ago
      So this automatically makes it unsafe in case of VM snapshots. The Linux kernel now has an optimization that makes it safe: https://lwn.net/Articles/983186/ https://lwn.net/Articles/983186/ Go should automatically benefit from this, if they use the vDSO getrandom().
    - maxmcd 1y ago
      Ah cool, the note here is also interesting: https://pkg.go.dev/github.com/google/uuid#EnableRandPool https://pkg.go.dev/github.com/google/uuid#EnableRandPool
- evil-olive 1y ago
  what real-world problem, if any, does 10x faster UUID generation solve? from your readme, `guid.New()` is 6~10 ns, so presumably the standard UUID package takes 60-100 ns? say I generate a UUID, and then use that UUID when inserting a row into my database, let's say committing that transaction takes 1 msec (1 million ns) if I get a speedup of 90 ns from using a faster UUID package, will that even be noticeable in my benchmarks? it seems likely to be lost in the noise. honestly, this seems like going on a 7-day road trip, and sprinting from your front door to your car because it'll get you there faster.
  - sdrapkin 1y ago
    Amazon AWS S3 web servers process millions of requests per second, and each response generates a random Request-Id. It’s not exactly 16 bytes, but this is a very realistic scenario where guids are used in hot path. If you are writing a cute-kitten blog, might as well use Python instead..
amluto 1y ago
On an extremely quick review: - This uses global state under the hood. Surprise! Is it thread safe? I’m not a Go expert, but it looks non-thread-safe. - The copying code reminds me of old-school awful C buffer handling code. Maybe it’s right. Maybe it’s wrong. But it’s not obviously right. - The actual meat is a cryptographic randomness cache. This is a subtle thing, and all the best practices are missing. Where’s the backtracking protection? What if the program forks? vDSO getrandom() knows how to do this correctly — something high-level should use it, not reimplement it incorrectly.
- sdrapkin 1y ago
  Thanks for your feedback. If you are skilled in Golang, I suggest you review the code more thoroughly for a more accurate understanding (especially compared to what standard uuid does).
- Groxx 1y ago
  Global use looks fine - it's a very-simply-used sync pool to do larger blocks of rand reads, which makes plenty of sense for performance. Unsafe use also looks fine, values either don't escape the function (a type string->byte type cast for function signature reasons) or they do but they're new temporary data (the byte->string cast, which is fine because there's no risk of reusing or modifying the original bytes). I'm going to intentionally not make any claims to "cryptographic security" or "is this a GUID" as I'm not super clear on the details there. The code looks pretty normal to me though, with the possible exception of the base64 encoding (why not base64.URLEncoding? https://pkg.go.dev/encoding/base64#pkg-variables https://pkg.go.dev/encoding/base64#pkg-variables).
- sa46 1y ago
  > This uses global state under the hood. Looks safe to me. It uses `crypto/rand.Read` which is declared as safe for concurrent use. The cache is accessed via sync.Pool which is thread safe. As a check, I ran the tests with `-race` and it passed.
CafeRacer 1y ago
Would have been nice if that included timestamp information, to make them orderable. Similar to what uuid v7 does.
- sdrapkin 1y ago
  It's on the roadmap (already implemented in a similar .NET library - https://github.com/sdrapkin/SecurityDriven.FastGuid https://github.com/sdrapkin/SecurityDriven.FastGuid).
deathanatos 1y ago
These aren't GUIDs[1]. If it isn't meant to follow the RFC, … just find a new word. (There are plenty of alternate schemes out there, too.) [1]: https://www.rfc-editor.org/rfc/rfc9562.html https://www.rfc-editor.org/rfc/rfc9562.html
- sdrapkin 1y ago
  IMHO "Guid" is just as well known (Wikipedia agrees: https://en.wikipedia.org/wiki/Universally_unique_identifier https://en.wikipedia.org/wiki/Universally_unique_identifier), and "UUID" was already taken by Google.
  - majewsky 1y ago
    > "UUID" was already taken by Google Your link also says that the term UUID predates the founding of Google by over a decade.
  - danbruc 1y ago
    I think the point is that this just generates 16 random bytes whereas UUIDs/GUIDs have structure, they at least have a variant fields indicating what kind of UUID/GUID it is. The closest thing to all random bytes would be variant 10xx, version 4 or 8.
    - sdrapkin 1y ago
      You are correct - Guid very specifically and intentionally generates a structure of 16 random bytes. In decades of programming I've never needed a random 16-byte structure to have a "internal versioned structure". In very rare cases this is truly needed, bit-twiddling post-generation can cheaply fix it (but not the other way around). Which is why all these "versions" and "variants" in standard universally applicable libraries are a complete waste of entropy and cycles.
      - danbruc 1y ago
        I do not think I have seen or noticed code that inspects UUID variants either but I could certainly imagine that such code is out there, for example to protect against accidental information leakage from UUID variants that are not purely random. With that in mind it seems a good idea to adhere to the standards if one uses an established name. Neither the few lost bits nor the effort to correctly indicate the variant sound like real issues to me.
sdrapkin 1y ago
In case you missed it, "guid.Read()" is a much faster alternative to "crypto/rand". https://pkg.go.dev/github.com/sdrapkin/guid#Read https://pkg.go.dev/github.com/sdrapkin/guid#Read
imiric 1y ago
My understanding was that speed is not something you want in a UUID generator, since it makes it more susceptible to brute force attacks. Is this not the case? I've been using Cuid2[1] in most of my personal projects (this Go implementation[2], actually), which is fast enough, but not "too fast". It's also secure, collision resistant, and has everything I would need from a UUID. [1]: https://github.com/paralleldrive/cuid2 https://github.com/paralleldrive/cuid2 [2]: https://github.com/nrednav/cuid2 https://github.com/nrednav/cuid2
- sdrapkin 1y ago
  cuid2 generates variable-length strings. If you want fast cryptographically strong string generation, I recommend https://github.com/sdrapkin/randstring https://github.com/sdrapkin/randstring. It will likely be faster than cuid2.
  - imiric 1y ago
    That doesn't address what I said. Nor explains why your package is better.
    - sdrapkin 1y ago
      Guid package generates guids/uuids. Your linked package generates variable length strings. These are different usecases (oh, and your benchmarks are inferior to https://github.com/sdrapkin/randstring https://github.com/sdrapkin/randstring). Nothing to argue about.
      - shakna 1y ago
        But this doesn't generate guid/uuids? It generates random bytes.
        sdrapkin 1y ago
        Guid/uuid is defined as a 16-byte structure. Are you questioning the “byte” part, or the “random” part?
saclark11 1y ago
Advertising any UUID/GUID generator as cryptographically secure, or relying on it to be so, is a mistake, in my opinion. You use a UUID when you need a universally unique ID whose guessability properties are not a critical security requirement. While the V4 UUID spec (which this package does not implement, but most users might assume it does) states that a UUID implementation SHOULD be cryptographically secure [1], it also states that they MUST NOT be used as security capabilities [2]. This is b/c they are not intended as secure tokens, but many users mistakenly assume them to be suitable as such. Not to mention, V4 UUIDs only have 122 bits of entropy, not 128, since 6 bits are reserved for version and variant information, which many users don't realize. So you can generate a UUID that is suitable as a secure token, but at that point don't call it a UUID. Just call it a secure token. And if you need a secure token, use something like Go's `Text()` function from `crypto/rand` [3]. The situation reminds me of how the Go team updated the `math/rand` and `math/rand/v2` packages to use a CSPRNG as a defensive measure [4], while still urging users to use `crypto/rand` in secure contexts. [1]: https://www.rfc-editor.org/rfc/rfc9562.html#unguessability https://www.rfc-editor.org/rfc/rfc9562.html#unguessability [2]: https://www.rfc-editor.org/rfc/rfc9562.html#Security https://www.rfc-editor.org/rfc/rfc9562.html#Security [3]: https://pkg.go.dev/crypto/rand@go1.24.5#Text https://pkg.go.dev/crypto/rand@go1.24.5#Text [4]: https://go.dev/blog/chacha8rand https://go.dev/blog/chacha8rand
- sdrapkin 1y ago
  The vast majority of Golang developers would benefit from using Guid library instead of UUID library. It’s substantially faster in all cases, more secure (by 2^6) and has more functionality. For random token-as-string generation Golang developers should be using https://github.com/sdrapkin/randstring https://github.com/sdrapkin/randstring instead of crypto/rand.Text (faster and more flexible).
  - stouset 1y ago
    The vast majority of Golang developers are neither hobbled by the lack of gigabyte throughput for random identifier generation nor are they on the verge of becoming victims to attacks on identifiers with "only" 2^122 random bits.
gg-plz 1y ago
[dead]

20 points by sdrapkin 1y ago

sdrapkin 1y ago

Much faster (~10x) than standard github.com/google/uuid package I'm interested in feedback from the HN community.

throwaway894345 1y ago
Why is it so much faster than `uuid`?
- sdrapkin 1y ago
  It generates entropy 4kb-at-a-time (instead of on each call), and uses a cache-pool instead of single cache behind a lock (which is what standard uuid does in "RandPool=ON" mode).
  - cyberax 1y ago
    So this automatically makes it unsafe in case of VM snapshots. The Linux kernel now has an optimization that makes it safe: https://lwn.net/Articles/983186/ https://lwn.net/Articles/983186/ Go should automatically benefit from this, if they use the vDSO getrandom().
  - maxmcd 1y ago
    Ah cool, the note here is also interesting: https://pkg.go.dev/github.com/google/uuid#EnableRandPool https://pkg.go.dev/github.com/google/uuid#EnableRandPool
evil-olive 1y ago
what real-world problem, if any, does 10x faster UUID generation solve? from your readme, `guid.New()` is 6~10 ns, so presumably the standard UUID package takes 60-100 ns? say I generate a UUID, and then use that UUID when inserting a row into my database, let's say committing that transaction takes 1 msec (1 million ns) if I get a speedup of 90 ns from using a faster UUID package, will that even be noticeable in my benchmarks? it seems likely to be lost in the noise. honestly, this seems like going on a 7-day road trip, and sprinting from your front door to your car because it'll get you there faster.
- sdrapkin 1y ago
  Amazon AWS S3 web servers process millions of requests per second, and each response generates a random Request-Id. It’s not exactly 16 bytes, but this is a very realistic scenario where guids are used in hot path. If you are writing a cute-kitten blog, might as well use Python instead..

amluto 1y ago

On an extremely quick review: - This uses global state under the hood. Surprise! Is it thread safe? I’m not a Go expert, but it looks non-thread-safe. - The copying code reminds me of old-school awful C buffer handling code. Maybe it’s right. Maybe it’s wrong. But it’s not obviously right. - The actual meat is a cryptographic randomness cache. This is a subtle thing, and all the best practices are missing. Where’s the backtracking protection? What if the program forks? vDSO getrandom() knows how to do this correctly — something high-level should use it, not reimplement it incorrectly.

sdrapkin 1y ago
Thanks for your feedback. If you are skilled in Golang, I suggest you review the code more thoroughly for a more accurate understanding (especially compared to what standard uuid does).
Groxx 1y ago
Global use looks fine - it's a very-simply-used sync pool to do larger blocks of rand reads, which makes plenty of sense for performance. Unsafe use also looks fine, values either don't escape the function (a type string->byte type cast for function signature reasons) or they do but they're new temporary data (the byte->string cast, which is fine because there's no risk of reusing or modifying the original bytes). I'm going to intentionally not make any claims to "cryptographic security" or "is this a GUID" as I'm not super clear on the details there. The code looks pretty normal to me though, with the possible exception of the base64 encoding (why not base64.URLEncoding? https://pkg.go.dev/encoding/base64#pkg-variables https://pkg.go.dev/encoding/base64#pkg-variables).
sa46 1y ago
> This uses global state under the hood. Looks safe to me. It uses `crypto/rand.Read` which is declared as safe for concurrent use. The cache is accessed via sync.Pool which is thread safe. As a check, I ran the tests with `-race` and it passed.

CafeRacer 1y ago

Would have been nice if that included timestamp information, to make them orderable. Similar to what uuid v7 does.

sdrapkin 1y ago
It's on the roadmap (already implemented in a similar .NET library - https://github.com/sdrapkin/SecurityDriven.FastGuid https://github.com/sdrapkin/SecurityDriven.FastGuid).

deathanatos 1y ago

These aren't GUIDs[1]. If it isn't meant to follow the RFC, … just find a new word. (There are plenty of alternate schemes out there, too.) [1]: https://www.rfc-editor.org/rfc/rfc9562.html https://www.rfc-editor.org/rfc/rfc9562.html

sdrapkin 1y ago
IMHO "Guid" is just as well known (Wikipedia agrees: https://en.wikipedia.org/wiki/Universally_unique_identifier https://en.wikipedia.org/wiki/Universally_unique_identifier), and "UUID" was already taken by Google.
- majewsky 1y ago
  > "UUID" was already taken by Google Your link also says that the term UUID predates the founding of Google by over a decade.
- danbruc 1y ago
  I think the point is that this just generates 16 random bytes whereas UUIDs/GUIDs have structure, they at least have a variant fields indicating what kind of UUID/GUID it is. The closest thing to all random bytes would be variant 10xx, version 4 or 8.
  - sdrapkin 1y ago
    You are correct - Guid very specifically and intentionally generates a structure of 16 random bytes. In decades of programming I've never needed a random 16-byte structure to have a "internal versioned structure". In very rare cases this is truly needed, bit-twiddling post-generation can cheaply fix it (but not the other way around). Which is why all these "versions" and "variants" in standard universally applicable libraries are a complete waste of entropy and cycles.
    - danbruc 1y ago
      I do not think I have seen or noticed code that inspects UUID variants either but I could certainly imagine that such code is out there, for example to protect against accidental information leakage from UUID variants that are not purely random. With that in mind it seems a good idea to adhere to the standards if one uses an established name. Neither the few lost bits nor the effort to correctly indicate the variant sound like real issues to me.

sdrapkin 1y ago

In case you missed it, "guid.Read()" is a much faster alternative to "crypto/rand". https://pkg.go.dev/github.com/sdrapkin/guid#Read https://pkg.go.dev/github.com/sdrapkin/guid#Read

imiric 1y ago

My understanding was that speed is not something you want in a UUID generator, since it makes it more susceptible to brute force attacks. Is this not the case? I've been using Cuid2[1] in most of my personal projects (this Go implementation[2], actually), which is fast enough, but not "too fast". It's also secure, collision resistant, and has everything I would need from a UUID. [1]: https://github.com/paralleldrive/cuid2 https://github.com/paralleldrive/cuid2 [2]: https://github.com/nrednav/cuid2 https://github.com/nrednav/cuid2

sdrapkin 1y ago
cuid2 generates variable-length strings. If you want fast cryptographically strong string generation, I recommend https://github.com/sdrapkin/randstring https://github.com/sdrapkin/randstring. It will likely be faster than cuid2.
- imiric 1y ago
  That doesn't address what I said. Nor explains why your package is better.
  - sdrapkin 1y ago
    Guid package generates guids/uuids. Your linked package generates variable length strings. These are different usecases (oh, and your benchmarks are inferior to https://github.com/sdrapkin/randstring https://github.com/sdrapkin/randstring). Nothing to argue about.
    - shakna 1y ago
      But this doesn't generate guid/uuids? It generates random bytes.
      - sdrapkin 1y ago
        Guid/uuid is defined as a 16-byte structure. Are you questioning the “byte” part, or the “random” part?

saclark11 1y ago

Advertising any UUID/GUID generator as cryptographically secure, or relying on it to be so, is a mistake, in my opinion. You use a UUID when you need a universally unique ID whose guessability properties are not a critical security requirement. While the V4 UUID spec (which this package does not implement, but most users might assume it does) states that a UUID implementation SHOULD be cryptographically secure [1], it also states that they MUST NOT be used as security capabilities [2]. This is b/c they are not intended as secure tokens, but many users mistakenly assume them to be suitable as such. Not to mention, V4 UUIDs only have 122 bits of entropy, not 128, since 6 bits are reserved for version and variant information, which many users don't realize. So you can generate a UUID that is suitable as a secure token, but at that point don't call it a UUID. Just call it a secure token. And if you need a secure token, use something like Go's `Text()` function from `crypto/rand` [3]. The situation reminds me of how the Go team updated the `math/rand` and `math/rand/v2` packages to use a CSPRNG as a defensive measure [4], while still urging users to use `crypto/rand` in secure contexts. [1]: https://www.rfc-editor.org/rfc/rfc9562.html#unguessability https://www.rfc-editor.org/rfc/rfc9562.html#unguessability [2]: https://www.rfc-editor.org/rfc/rfc9562.html#Security https://www.rfc-editor.org/rfc/rfc9562.html#Security [3]: https://pkg.go.dev/crypto/rand@go1.24.5#Text https://pkg.go.dev/crypto/rand@go1.24.5#Text [4]: https://go.dev/blog/chacha8rand https://go.dev/blog/chacha8rand

sdrapkin 1y ago
The vast majority of Golang developers would benefit from using Guid library instead of UUID library. It’s substantially faster in all cases, more secure (by 2^6) and has more functionality. For random token-as-string generation Golang developers should be using https://github.com/sdrapkin/randstring https://github.com/sdrapkin/randstring instead of crypto/rand.Text (faster and more flexible).
- stouset 1y ago
  The vast majority of Golang developers are neither hobbled by the lack of gigabyte throughput for random identifier generation nor are they on the verge of becoming victims to attacks on identifiers with "only" 2^122 random bits.

gg-plz 1y ago

[dead]