4 ms·

There are very, very few services which actually need to handle more than 25.6k unique IDs per node per second. I'm not saying they don't exist, but it is not

There are very, very few services which actually need to handle more than 25.6k unique IDs per node per second. I'm not saying they don't exist, but it is not a problem most companies have.

lmilcin 6y ago
Ah, yes... and 640kB of RAM was supposed to be enough, too. Or 4B IP addresses. If anything, my experience is whenever somebody sets close hard limits like that somebody else is going to pay for it. The service I work on creates up to couple millions of unique objects per second, on a single node in a batch process. It tries to do a large batch of work as fast as is possible. It fetches I think something like 2GB of data per second from MongoDB node, processes it and creates up to 5 million objects in span of 2-3 seconds. We also have realtime services that are about 20-30k TPS per node. We are currently using MongoDB and it uses plenty bits for that: https://docs.mongodb.com/manual/reference/method/ObjectId/ https://docs.mongodb.com/manual/reference/method/ObjectId/
- secondcoming 6y ago
  Why the hell has this been downvoted? Not everyone on HN is a web developer. Some of the audience works on high-throughput systems. Instead of downvoting, perhaps people would prefer to learn? They may save their company lots of money!
  - lmilcin 6y ago
    Hey, it is a Rust library. So probably not many web devs are going to use it. Which is even more reason to point out limitations like that because if you choose Rust for your project it means very likely you have some important performance constraints. And that is either linked with a small system with low resources (in which case it is ok) or a large system with huge traffic (in which case it is not).
  - staticassertion 6y ago
    Yeah I get pretty bored constantly reading "You don't need X". Some of us do. You don't have to be Google to deal with a lot of data, and to want that to be fast. Lots of domains deal with high volumes of requests and deal with massive amounts of volume.
  - jfrisby 6y ago
    Well, for starters, it's simply untrue that every company (likely) will (barring bankruptcy of course) eventually need extreme scale, and the implicit assumption that the up-front cost of implementing such scalability is necessarily worthwhile. That may be true in Staruplandia, but the industry is a _lot_ bigger than the world of Silicon Valley style startups. For example: My current company is a small lifestyle biz. Sure, it's a "tech company", has an event-processing pipeline and the like, but if we were 100x more successful than our wildest ambitions, we still wouldn't need anywhere near 25.6k IDs/sec/server. And, given the objectives of the company, it would make more sense to turn customers away than to grow the team to accommodate that sort of demand. The simple fact is that in a great many situations extreme scale isn't needed. And, either way, the cost of implementing extreme scalability can inflict its own harm. Highly scalable systems usually come with more operational complexity, and steeper learning curves. When you have a small team, this can impair -- or even cripple -- product development. If your company hasn't found product/market fit yet, this attitude of sacrificing the present for an imagined future can materially reduce the likelihood of that future coming to pass. Of course, the opposite is sometimes true as well. Companies have, in fact, failed because they found product market fit but took so many shortcuts they couldn't adapt and grow into their success. But the point is that determining how much to invest in future-proofing is a complex and nuanced problem not amenable to sweeping generalizations. Now, this clearly isn't such an extreme case, but frankly Sonyflake (the original, not the Rust implementation) seems to be operationally simpler than Snowflake while offering perfectly reasonable tradeoffs. The Rust implementation might prove entirely useful to any number of organizations _based on their needs_. If they have a Rust codebase with a single process per machine this could easily be a simpler and more robust option than Sonyflake. The kind of arrogant dismissiveness based on one's own personal (and highly specialized) experience that's shown in the downvoted comment tends to leave a bad taste in peoples' mouths. Thus the downvoting.
    - secondcoming 6y ago
      > And, either way, the cost of implementing extreme scalability can inflict its own harm. Highly scalable systems usually come with more operational complexity, and steeper learning curves I think you're overestimating the cost factor of future-proofing an architecture for extreme scale (and 26K/sec/server isn't actually that 'extreme'). And instead of downvoting people who've walked that walk, perhaps people may realise this by engaging instead. Also, I didn't read any 'arrogant dismissiveness' in the post. Each to their own! Happy New Year!
- Jweb_Guru 6y ago
  I can almost guarantee you that your service could be rewritten to not require millions of unique objects per second per node, and/or split up so that the nodes were on different cores. I'm quite confident you are not anywhere close to optimization boundaries around throughput because MongoDB is far from the best-optimized database for batch processing.

There are very, very few services which actually need to handle more than 25.6k unique IDs per node per second. I'm not saying they don't exist, but it is not a problem most companies have.

lmilcin 6y ago

Ah, yes... and 640kB of RAM was supposed to be enough, too. Or 4B IP addresses. If anything, my experience is whenever somebody sets close hard limits like that somebody else is going to pay for it. The service I work on creates up to couple millions of unique objects per second, on a single node in a batch process. It tries to do a large batch of work as fast as is possible. It fetches I think something like 2GB of data per second from MongoDB node, processes it and creates up to 5 million objects in span of 2-3 seconds. We also have realtime services that are about 20-30k TPS per node. We are currently using MongoDB and it uses plenty bits for that: https://docs.mongodb.com/manual/reference/method/ObjectId/ https://docs.mongodb.com/manual/reference/method/ObjectId/

secondcoming 6y ago
Why the hell has this been downvoted? Not everyone on HN is a web developer. Some of the audience works on high-throughput systems. Instead of downvoting, perhaps people would prefer to learn? They may save their company lots of money!
- lmilcin 6y ago
  Hey, it is a Rust library. So probably not many web devs are going to use it. Which is even more reason to point out limitations like that because if you choose Rust for your project it means very likely you have some important performance constraints. And that is either linked with a small system with low resources (in which case it is ok) or a large system with huge traffic (in which case it is not).
- staticassertion 6y ago
  Yeah I get pretty bored constantly reading "You don't need X". Some of us do. You don't have to be Google to deal with a lot of data, and to want that to be fast. Lots of domains deal with high volumes of requests and deal with massive amounts of volume.
- jfrisby 6y ago
  Well, for starters, it's simply untrue that every company (likely) will (barring bankruptcy of course) eventually need extreme scale, and the implicit assumption that the up-front cost of implementing such scalability is necessarily worthwhile. That may be true in Staruplandia, but the industry is a _lot_ bigger than the world of Silicon Valley style startups. For example: My current company is a small lifestyle biz. Sure, it's a "tech company", has an event-processing pipeline and the like, but if we were 100x more successful than our wildest ambitions, we still wouldn't need anywhere near 25.6k IDs/sec/server. And, given the objectives of the company, it would make more sense to turn customers away than to grow the team to accommodate that sort of demand. The simple fact is that in a great many situations extreme scale isn't needed. And, either way, the cost of implementing extreme scalability can inflict its own harm. Highly scalable systems usually come with more operational complexity, and steeper learning curves. When you have a small team, this can impair -- or even cripple -- product development. If your company hasn't found product/market fit yet, this attitude of sacrificing the present for an imagined future can materially reduce the likelihood of that future coming to pass. Of course, the opposite is sometimes true as well. Companies have, in fact, failed because they found product market fit but took so many shortcuts they couldn't adapt and grow into their success. But the point is that determining how much to invest in future-proofing is a complex and nuanced problem not amenable to sweeping generalizations. Now, this clearly isn't such an extreme case, but frankly Sonyflake (the original, not the Rust implementation) seems to be operationally simpler than Snowflake while offering perfectly reasonable tradeoffs. The Rust implementation might prove entirely useful to any number of organizations _based on their needs_. If they have a Rust codebase with a single process per machine this could easily be a simpler and more robust option than Sonyflake. The kind of arrogant dismissiveness based on one's own personal (and highly specialized) experience that's shown in the downvoted comment tends to leave a bad taste in peoples' mouths. Thus the downvoting.
  - secondcoming 6y ago
    > And, either way, the cost of implementing extreme scalability can inflict its own harm. Highly scalable systems usually come with more operational complexity, and steeper learning curves I think you're overestimating the cost factor of future-proofing an architecture for extreme scale (and 26K/sec/server isn't actually that 'extreme'). And instead of downvoting people who've walked that walk, perhaps people may realise this by engaging instead. Also, I didn't read any 'arrogant dismissiveness' in the post. Each to their own! Happy New Year!
Jweb_Guru 6y ago
I can almost guarantee you that your service could be rewritten to not require millions of unique objects per second per node, and/or split up so that the nodes were on different cores. I'm quite confident you are not anywhere close to optimization boundaries around throughput because MongoDB is far from the best-optimized database for batch processing.