3 ms·
Hey, this is pretty insightful and not insulting at all - I always love hearing about the experiences of others. The RNG being used is not the random.random()
by makmanalp 13y ago
Hey, this is pretty insightful and not insulting at all - I always love hearing about the experiences of others.
The RNG being used is not the random.random() Mersenne Twister one, it's the OS-provided cryptographically secure one:
https://github.com/SawdustSoftware/simpleflake/blob/master/simpleflake.py#L48 https://github.com/SawdustSoftware/simpleflake/blob/master/s...
But in any case, the reason you need NTPd sync is not to seed the RNG, but because the IDs themselves are prefixed with millisecond timestamps. This is to ensure that you're not using the same sub-keyspace for that millisecond over and over again from multiple processes.
If you're seeing that many collisions, obviously, it's time to change methods like you mention. Data being dropped out of existence sounds like an application code issue, and not an inherent problem with your IDing scheme. That sounds like a place where ops should get a big honking error message ("tried to insert twice, collided both times"). In any case, the contingency plan for that for us is to just switch to snowflake, since with this specific scheme it should take a while before we hit that barrier, as calculated in the article.
Adding more metadata is definitely an option, but I'm not a fan of it since data schemas (and data itself) have a nasty habit of changing over time, meaning you have to change your IDing scheme. Completely surrogate IDs don't have this issue.