4 ms·

Unfortunately, 8 bits for sequence for each 1/100th of a second is way too low. I gather the person that designed this has never seen high traffic service. But

Unfortunately, 8 bits for sequence for each 1/100th of a second is way too low. I gather the person that designed this has never seen high traffic service. But for some having cap of 25.6k unique ids per second on a node is a deal breaker.

Jweb_Guru 6y ago
There are very, very few services which actually need to handle more than 25.6k unique IDs per node per second. I'm not saying they don't exist, but it is not a problem most companies have.
- lmilcin 6y ago
  Ah, yes... and 640kB of RAM was supposed to be enough, too. Or 4B IP addresses. If anything, my experience is whenever somebody sets close hard limits like that somebody else is going to pay for it. The service I work on creates up to couple millions of unique objects per second, on a single node in a batch process. It tries to do a large batch of work as fast as is possible. It fetches I think something like 2GB of data per second from MongoDB node, processes it and creates up to 5 million objects in span of 2-3 seconds. We also have realtime services that are about 20-30k TPS per node. We are currently using MongoDB and it uses plenty bits for that: https://docs.mongodb.com/manual/reference/method/ObjectId/ https://docs.mongodb.com/manual/reference/method/ObjectId/
  - secondcoming 6y ago
    Why the hell has this been downvoted? Not everyone on HN is a web developer. Some of the audience works on high-throughput systems. Instead of downvoting, perhaps people would prefer to learn? They may save their company lots of money!
    - lmilcin 6y ago
      Hey, it is a Rust library. So probably not many web devs are going to use it. Which is even more reason to point out limitations like that because if you choose Rust for your project it means very likely you have some important performance constraints. And that is either linked with a small system with low resources (in which case it is ok) or a large system with huge traffic (in which case it is not).
    - staticassertion 6y ago
      Yeah I get pretty bored constantly reading "You don't need X". Some of us do. You don't have to be Google to deal with a lot of data, and to want that to be fast. Lots of domains deal with high volumes of requests and deal with massive amounts of volume.
    - jfrisby 6y ago
      Well, for starters, it's simply untrue that every company (likely) will (barring bankruptcy of course) eventually need extreme scale, and the implicit assumption that the up-front cost of implementing such scalability is necessarily worthwhile. That may be true in Staruplandia, but the industry is a _lot_ bigger than the world of Silicon Valley style startups. For example: My current company is a small lifestyle biz. Sure, it's a "tech company", has an event-processing pipeline and the like, but if we were 100x more successful than our wildest ambitions, we still wouldn't need anywhere near 25.6k IDs/sec/server. And, given the objectives of the company, it would make more sense to turn customers away than to grow the team to accommodate that sort of demand. The simple fact is that in a great many situations extreme scale isn't needed. And, either way, the cost of implementing extreme scalability can inflict its own harm. Highly scalable systems usually come with more operational complexity, and steeper learning curves. When you have a small team, this can impair -- or even cripple -- product development. If your company hasn't found product/market fit yet, this attitude of sacrificing the present for an imagined future can materially reduce the likelihood of that future coming to pass. Of course, the opposite is sometimes true as well. Companies have, in fact, failed because they found product market fit but took so many shortcuts they couldn't adapt and grow into their success. But the point is that determining how much to invest in future-proofing is a complex and nuanced problem not amenable to sweeping generalizations. Now, this clearly isn't such an extreme case, but frankly Sonyflake (the original, not the Rust implementation) seems to be operationally simpler than Snowflake while offering perfectly reasonable tradeoffs. The Rust implementation might prove entirely useful to any number of organizations _based on their needs_. If they have a Rust codebase with a single process per machine this could easily be a simpler and more robust option than Sonyflake. The kind of arrogant dismissiveness based on one's own personal (and highly specialized) experience that's shown in the downvoted comment tends to leave a bad taste in peoples' mouths. Thus the downvoting.
      - secondcoming 6y ago
        > And, either way, the cost of implementing extreme scalability can inflict its own harm. Highly scalable systems usually come with more operational complexity, and steeper learning curves I think you're overestimating the cost factor of future-proofing an architecture for extreme scale (and 26K/sec/server isn't actually that 'extreme'). And instead of downvoting people who've walked that walk, perhaps people may realise this by engaging instead. Also, I didn't read any 'arrogant dismissiveness' in the post. Each to their own! Happy New Year!
  - Jweb_Guru 6y ago
    I can almost guarantee you that your service could be rewritten to not require millions of unique objects per second per node, and/or split up so that the nodes were on different cores. I'm quite confident you are not anywhere close to optimization boundaries around throughput because MongoDB is far from the best-optimized database for batch processing.
dljsjr 6y ago
It was designed by Twitter…
- lmilcin 6y ago
  Which is a proof of what?
  - dljsjr 6y ago
    I mean if Twitter isn't "high traffic" then I'm not sure what is.
    - lmilcin 6y ago
      I can imagine Twitter has a lot of services that don't see a lot of traffic and a lot of people who don't have experience with high traffic services. Take your calculator and do calculations along with me: 10ms = 1/100th of a second. 8 bits = 256 possible values. For a given node you have at most 256 possible IDs every 1/100th of a second or 25600 every second. This is absolute limitation which means, you can't just straight use the library without some workarounds to get more traffic through a node. You could of course do things like assign more hardware addresses to each node (maybe even messages on one and odd on another hw address) but then you soon find out the workaround is more complex than just writing it from scratch.
      - dljsjr 6y ago
        Well sure but that's not what this UID scheme was designed to solve. It's an engineering trade-off. It doesn't mean it's bad it just mean it doesn't work for incredibly high throughput use-cases where things scale vertically instead of horizontally. That's fine.
    - jfrisby 6y ago
      I think you're crossing the streams a bit here. Twitter designed _Snow_flake. _Sony_flake is a reimplementation that changes the allocation of bits, and the author acknowledges that the maximum ID generation throughput ceiling is lower than that of Snowflake. Snowflake uses a millisecond-precision timestamp, and had a 12-bit sequence number. So 4,096,000 IDs per node-second. At that point, the bottleneck won't be the format of the IDs but the performance of the code, and IPC mechanism. Which, in this case, is non-trivial since Snowflake uses a socket-based approach to communication. Lower-overhead, native IPC mechanisms are certainly possible via JNI but would probably take some doing to implement. For Sonyflake, I don't imagine the socket overhead is all that much of an issue given the low throughput it's capable of with its bit allocations. Were I to design something like this again[1], I might start with something like Sonyflake (the self-assignment of host ID w/out needing to coordinate via Zookeeper is nice), shave a couple bits from the top of the timestamp, maybe a couple from the top of the host ID, and pack the remainder at the top, leaving a few zero bits at the bottom. That would essentially mean the value returned was the start of a _range_, and anything needing to generate IDs in large quantities can keep its own in-process counter. Only one API call per N IDs generated by a given thread/process. And, of course, unix-domain socket or other lower-overhead approach to communication for when the API calls are needed. [1] - A decade prior to Snowflake's release, I wound up taking a very similar approach at one of my first startups, albeit much more crude/inelegant and without nice properties like k-ordering.
deleted 6y ago
[deleted]

Jweb_Guru 6y ago

There are very, very few services which actually need to handle more than 25.6k unique IDs per node per second. I'm not saying they don't exist, but it is not a problem most companies have.

lmilcin 6y ago
Ah, yes... and 640kB of RAM was supposed to be enough, too. Or 4B IP addresses. If anything, my experience is whenever somebody sets close hard limits like that somebody else is going to pay for it. The service I work on creates up to couple millions of unique objects per second, on a single node in a batch process. It tries to do a large batch of work as fast as is possible. It fetches I think something like 2GB of data per second from MongoDB node, processes it and creates up to 5 million objects in span of 2-3 seconds. We also have realtime services that are about 20-30k TPS per node. We are currently using MongoDB and it uses plenty bits for that: https://docs.mongodb.com/manual/reference/method/ObjectId/ https://docs.mongodb.com/manual/reference/method/ObjectId/
- secondcoming 6y ago
  Why the hell has this been downvoted? Not everyone on HN is a web developer. Some of the audience works on high-throughput systems. Instead of downvoting, perhaps people would prefer to learn? They may save their company lots of money!
  - lmilcin 6y ago
    Hey, it is a Rust library. So probably not many web devs are going to use it. Which is even more reason to point out limitations like that because if you choose Rust for your project it means very likely you have some important performance constraints. And that is either linked with a small system with low resources (in which case it is ok) or a large system with huge traffic (in which case it is not).
  - staticassertion 6y ago
    Yeah I get pretty bored constantly reading "You don't need X". Some of us do. You don't have to be Google to deal with a lot of data, and to want that to be fast. Lots of domains deal with high volumes of requests and deal with massive amounts of volume.
  - jfrisby 6y ago
    Well, for starters, it's simply untrue that every company (likely) will (barring bankruptcy of course) eventually need extreme scale, and the implicit assumption that the up-front cost of implementing such scalability is necessarily worthwhile. That may be true in Staruplandia, but the industry is a _lot_ bigger than the world of Silicon Valley style startups. For example: My current company is a small lifestyle biz. Sure, it's a "tech company", has an event-processing pipeline and the like, but if we were 100x more successful than our wildest ambitions, we still wouldn't need anywhere near 25.6k IDs/sec/server. And, given the objectives of the company, it would make more sense to turn customers away than to grow the team to accommodate that sort of demand. The simple fact is that in a great many situations extreme scale isn't needed. And, either way, the cost of implementing extreme scalability can inflict its own harm. Highly scalable systems usually come with more operational complexity, and steeper learning curves. When you have a small team, this can impair -- or even cripple -- product development. If your company hasn't found product/market fit yet, this attitude of sacrificing the present for an imagined future can materially reduce the likelihood of that future coming to pass. Of course, the opposite is sometimes true as well. Companies have, in fact, failed because they found product market fit but took so many shortcuts they couldn't adapt and grow into their success. But the point is that determining how much to invest in future-proofing is a complex and nuanced problem not amenable to sweeping generalizations. Now, this clearly isn't such an extreme case, but frankly Sonyflake (the original, not the Rust implementation) seems to be operationally simpler than Snowflake while offering perfectly reasonable tradeoffs. The Rust implementation might prove entirely useful to any number of organizations _based on their needs_. If they have a Rust codebase with a single process per machine this could easily be a simpler and more robust option than Sonyflake. The kind of arrogant dismissiveness based on one's own personal (and highly specialized) experience that's shown in the downvoted comment tends to leave a bad taste in peoples' mouths. Thus the downvoting.
    - secondcoming 6y ago
      > And, either way, the cost of implementing extreme scalability can inflict its own harm. Highly scalable systems usually come with more operational complexity, and steeper learning curves I think you're overestimating the cost factor of future-proofing an architecture for extreme scale (and 26K/sec/server isn't actually that 'extreme'). And instead of downvoting people who've walked that walk, perhaps people may realise this by engaging instead. Also, I didn't read any 'arrogant dismissiveness' in the post. Each to their own! Happy New Year!
- Jweb_Guru 6y ago
  I can almost guarantee you that your service could be rewritten to not require millions of unique objects per second per node, and/or split up so that the nodes were on different cores. I'm quite confident you are not anywhere close to optimization boundaries around throughput because MongoDB is far from the best-optimized database for batch processing.

dljsjr 6y ago

It was designed by Twitter…

lmilcin 6y ago
Which is a proof of what?
- dljsjr 6y ago
  I mean if Twitter isn't "high traffic" then I'm not sure what is.
  - lmilcin 6y ago
    I can imagine Twitter has a lot of services that don't see a lot of traffic and a lot of people who don't have experience with high traffic services. Take your calculator and do calculations along with me: 10ms = 1/100th of a second. 8 bits = 256 possible values. For a given node you have at most 256 possible IDs every 1/100th of a second or 25600 every second. This is absolute limitation which means, you can't just straight use the library without some workarounds to get more traffic through a node. You could of course do things like assign more hardware addresses to each node (maybe even messages on one and odd on another hw address) but then you soon find out the workaround is more complex than just writing it from scratch.
    - dljsjr 6y ago
      Well sure but that's not what this UID scheme was designed to solve. It's an engineering trade-off. It doesn't mean it's bad it just mean it doesn't work for incredibly high throughput use-cases where things scale vertically instead of horizontally. That's fine.
  - jfrisby 6y ago
    I think you're crossing the streams a bit here. Twitter designed _Snow_flake. _Sony_flake is a reimplementation that changes the allocation of bits, and the author acknowledges that the maximum ID generation throughput ceiling is lower than that of Snowflake. Snowflake uses a millisecond-precision timestamp, and had a 12-bit sequence number. So 4,096,000 IDs per node-second. At that point, the bottleneck won't be the format of the IDs but the performance of the code, and IPC mechanism. Which, in this case, is non-trivial since Snowflake uses a socket-based approach to communication. Lower-overhead, native IPC mechanisms are certainly possible via JNI but would probably take some doing to implement. For Sonyflake, I don't imagine the socket overhead is all that much of an issue given the low throughput it's capable of with its bit allocations. Were I to design something like this again[1], I might start with something like Sonyflake (the self-assignment of host ID w/out needing to coordinate via Zookeeper is nice), shave a couple bits from the top of the timestamp, maybe a couple from the top of the host ID, and pack the remainder at the top, leaving a few zero bits at the bottom. That would essentially mean the value returned was the start of a _range_, and anything needing to generate IDs in large quantities can keep its own in-process counter. Only one API call per N IDs generated by a given thread/process. And, of course, unix-domain socket or other lower-overhead approach to communication for when the API calls are needed. [1] - A decade prior to Snowflake's release, I wound up taking a very similar approach at one of my first startups, albeit much more crude/inelegant and without nice properties like k-ordering.

deleted 6y ago

[deleted]