3 ms·

Though this is a great idea ( similar to one I had created in the past) there is a major problem I think users should be aware of, concurrency. Usually using a

by Canausa 13y ago

Though this is a great idea ( similar to one I had created in the past) there is a major problem I think users should be aware of, concurrency. Usually using a ID generating system is used when data needs to be shared or asking for an ID from a database is just not an option. It is not typically used in small environments. I think this solution has an upper bar of effectiveness that is just to low for the problem this is trying to solve. The use of a pseudo random number generator ( python's random module) is too vulnerable to not generating "random" numbers and so many collisions can occur at high concurrency. Most pseudo random number generators usually are seeded with the time on the host system. As stated in the docs, the best way to use simpleflake is to setup NTP so all the servers times are in sync, the random module on all the servers/processes could be seeded with the same time ( like after a code push). If all the servers get seeded with the same time and the objects being operated on take a uniform time the probability of collisions go up and slow down all processes trying to interact with the objects, as they generate new IDs. As new Ids are being generated you lose the time sort that you originally wanted by including the time in the ID. I do not mean this to be insulting. I am speaking from experience from having debugged a problem caused the the ID generator at the company I formerly worked for. We used a similar generation technique to identify messages passing through the system, but we included more meta data in our IDs such as sender and recipient of the message and even the server name. With all of that when we ran 150 processes on a 4 core server, there was a .01% of collisions, but that jump significantly once the servers were upgraded to 16 core machines and at that point in the process there was no way to regenerate a new ID, the data was simply dropped out of existence.

makmanalp 13y ago
Hey, this is pretty insightful and not insulting at all - I always love hearing about the experiences of others. The RNG being used is not the random.random() Mersenne Twister one, it's the OS-provided cryptographically secure one: https://github.com/SawdustSoftware/simpleflake/blob/master/simpleflake.py#L48 https://github.com/SawdustSoftware/simpleflake/blob/master/s... But in any case, the reason you need NTPd sync is not to seed the RNG, but because the IDs themselves are prefixed with millisecond timestamps. This is to ensure that you're not using the same sub-keyspace for that millisecond over and over again from multiple processes. If you're seeing that many collisions, obviously, it's time to change methods like you mention. Data being dropped out of existence sounds like an application code issue, and not an inherent problem with your IDing scheme. That sounds like a place where ops should get a big honking error message ("tried to insert twice, collided both times"). In any case, the contingency plan for that for us is to just switch to snowflake, since with this specific scheme it should take a while before we hit that barrier, as calculated in the article. Adding more metadata is definitely an option, but I'm not a fan of it since data schemas (and data itself) have a nasty habit of changing over time, meaning you have to change your IDing scheme. Completely surrogate IDs don't have this issue.

by Canausa 13y ago

makmanalp 13y ago

Hey, this is pretty insightful and not insulting at all - I always love hearing about the experiences of others. The RNG being used is not the random.random() Mersenne Twister one, it's the OS-provided cryptographically secure one: https://github.com/SawdustSoftware/simpleflake/blob/master/simpleflake.py#L48 https://github.com/SawdustSoftware/simpleflake/blob/master/s... But in any case, the reason you need NTPd sync is not to seed the RNG, but because the IDs themselves are prefixed with millisecond timestamps. This is to ensure that you're not using the same sub-keyspace for that millisecond over and over again from multiple processes. If you're seeing that many collisions, obviously, it's time to change methods like you mention. Data being dropped out of existence sounds like an application code issue, and not an inherent problem with your IDing scheme. That sounds like a place where ops should get a big honking error message ("tried to insert twice, collided both times"). In any case, the contingency plan for that for us is to just switch to snowflake, since with this specific scheme it should take a while before we hit that barrier, as calculated in the article. Adding more metadata is definitely an option, but I'm not a fan of it since data schemas (and data itself) have a nasty habit of changing over time, meaning you have to change your IDing scheme. Completely surrogate IDs don't have this issue.