5 ms·

What these guys are doing with Bloom/Bud are searching for dramatically better abstractions for building distributed systems. Getting the performance right shou

What these guys are doing with Bloom/Bud are searching for dramatically better abstractions for building distributed systems. Getting the performance right should and will come later. I'm a big believer in the mantra "First make it possible. Then make it beautiful. Then make it fast." In the distributed systems community, there's a lot of experience with "making it possible": Hadoop, Dynamo, etc. Bloom/Bud is attempting to figure out the "make it beautiful" part by leveraging what we already understand about the problem domain of building distributed systems. Worrying about the kinds of constant-time performance things you mentioned at this stage would be premature optimization. I commend them for building this system in a language that allows them to iterate fast and experiment. I'm sure in the future they'll look at using technology like JRuby to improve the performance of the project. I think what they're doing is very interesting and potentially groundbreaking -- I can't wait to see where this project goes.

jerf 16y ago
I get where you are coming from, and it's a good plan, as long as the plan is to eventually fully detach from Ruby. Being even two or three times as fast as Ruby, which seems to be an optimistic interpretation of JRuby's performance, is still starting from a terrible position in so many ways. I don't get the idea that some people seem to have that performance doesn't matter for distributed systems, when the truth is the exact opposite. Desktops and even cell phones, we see a great deal of sloppiness around performance, because it doesn't really matter that much. Small servers or small clusters, we still say throw more hardware at it and just hack some stuff together for clustering. But when you're serious about distributed systems is also when you are counting every one of something; maybe disk hits, maybe CPU cycles, maybe bytes of RAM, but there is something you are obsessing over. And maybe you're obsessing over more than one of these at once, all with an intensity that would credit an Atari 2600 programmer. (Facebook apparently published the specs for their machines today. Tell me they aren't too concerned about performance.) I'm not sure leaving performance for later is a good idea, they may well iterate their way into a cool abstraction that will never perform. Designing a distributed system abstraction without worrying about performance strikes me as about as sensible as designing a new 3D framework without worrying about performance... not necessarily a fatal flaw but I sure hope you have a good plan.
- nathanmarz 16y ago
  Perhaps I was unclear. Performance is critical. They just shouldn't worry about the kinds of performance issues that can be optimized later. These include things like language choice and other micro-optimizations like zero-copy, etc. Getting the abstraction right is making something that can perform well once it's optimized. That is, you worry about the high level performance properties -- things like how it will perform asymptotically -- when figuring out the abstraction. For this project especially, the really difficult part is figuring out what those abstractions look like. Even having to do a complete rewrite later on in a faster language isn't that big of a deal cost-wise.
- jacques_chester 16y ago
  > I don't get the idea that some people seem to have that performance doesn't matter for distributed systems, when the truth is the exact opposite. I think that investment in performance follows a curve. A bowl, actually. And that this interest is based on the cost of optimisation vs the payoff. - - -- -- --- --- ---- ---- ------ ------ --------- --------- --------------- ---------------- ------------------------------------------------ <-- embedded ... SME web/desktop ... data-centre --> Assume that an optimisation costs $X of programmer time and pays back $Y dollars. When your cost of production is very large, $Y > $X. That's what you see for embedded systems with millions of units shipped and for data-centre computing with tens of thousands of units installed. The cost of one programmer optimising is well worth it. But for the sunny plain of mediocrity in the middle, the cost of extra hardware ($Y) will be less than the cost of the programmer time $X. Here endeth the extemporising.

jerf 16y ago

I get where you are coming from, and it's a good plan, as long as the plan is to eventually fully detach from Ruby. Being even two or three times as fast as Ruby, which seems to be an optimistic interpretation of JRuby's performance, is still starting from a terrible position in so many ways. I don't get the idea that some people seem to have that performance doesn't matter for distributed systems, when the truth is the exact opposite. Desktops and even cell phones, we see a great deal of sloppiness around performance, because it doesn't really matter that much. Small servers or small clusters, we still say throw more hardware at it and just hack some stuff together for clustering. But when you're serious about distributed systems is also when you are counting every one of something; maybe disk hits, maybe CPU cycles, maybe bytes of RAM, but there is something you are obsessing over. And maybe you're obsessing over more than one of these at once, all with an intensity that would credit an Atari 2600 programmer. (Facebook apparently published the specs for their machines today. Tell me they aren't too concerned about performance.) I'm not sure leaving performance for later is a good idea, they may well iterate their way into a cool abstraction that will never perform. Designing a distributed system abstraction without worrying about performance strikes me as about as sensible as designing a new 3D framework without worrying about performance... not necessarily a fatal flaw but I sure hope you have a good plan.

nathanmarz 16y ago
Perhaps I was unclear. Performance is critical. They just shouldn't worry about the kinds of performance issues that can be optimized later. These include things like language choice and other micro-optimizations like zero-copy, etc. Getting the abstraction right is making something that can perform well once it's optimized. That is, you worry about the high level performance properties -- things like how it will perform asymptotically -- when figuring out the abstraction. For this project especially, the really difficult part is figuring out what those abstractions look like. Even having to do a complete rewrite later on in a faster language isn't that big of a deal cost-wise.
jacques_chester 16y ago
> I don't get the idea that some people seem to have that performance doesn't matter for distributed systems, when the truth is the exact opposite. I think that investment in performance follows a curve. A bowl, actually. And that this interest is based on the cost of optimisation vs the payoff. - - -- -- --- --- ---- ---- ------ ------ --------- --------- --------------- ---------------- ------------------------------------------------ <-- embedded ... SME web/desktop ... data-centre --> Assume that an optimisation costs $X of programmer time and pays back $Y dollars. When your cost of production is very large, $Y > $X. That's what you see for embedded systems with millions of units shipped and for data-centre computing with tens of thousands of units installed. The cost of one programmer optimising is well worth it. But for the sunny plain of mediocrity in the middle, the cost of extra hardware ($Y) will be less than the cost of the programmer time $X. Here endeth the extemporising.