4 ms·

There's a great blog post on this: https://blog.twitter.com/2015/building-distributedlog-twitter-s-high-performance-replicated-log-service https://blog.twitter.

by murph 10y ago

There's a great blog post on this: https://blog.twitter.com/2015/building-distributedlog-twitter-s-high-performance-replicated-log-service https://blog.twitter.com/2015/building-distributedlog-twitte...

DannyBee 10y ago
Which says " At design time we had concerns about Kafka’s I/O model and its lack of strong durability guarantees‐a non-starter for an application like a distributed transaction log[3]" Seems reasonable, right? Except "[3] Kafka addressed these durability concerns in version 0.8" So they built a whole thing, because they didn't bother to ask or say "hey, if we help fix the durability, would that we welcome?" or even "do you guys have a plan and timeline to fix it?" That's .... not great. Now, maybe there are other reasons, but they aren't elucidated in this blog post :P. Even then, my general view would be "did you approach the community and discuss your concerns or just dismiss them out of hand as infeasible", mainly because my experience is that if you do this, you often find they have exactly the same set of concerns/goals, and just need more resources to make it happen. The desire to build shiny objects is very large. Outside of the paper plans of engineering teams, these things rarely end up up more shiny than what already exists or will be built by the time you are done.
- sametmax 10y ago
  Or even "we are ready to pay 3 people for a year to build that, if we don't and give you the money instead what can you do ?". The software industry has a real problem contributing to open source, the stuff, you know, allowing them to make money in the first place.
  - placeybordeaux 10y ago
    They did open source their solution. It sounds like you are arguing against parallel development. That happens all the time in open source.
    - sametmax 10y ago
      Contributing to other open source projects is very different than to open source your project. NIH is one of the worst problem of open source : companies are spending a lot of money to dev new software instead of improving existing one.
- d0vs 10y ago
  Seriously this seems like a common pattern in open source: [big company] could just improve [X] but instead builds something from the ground up.
  - mgodave 10y ago
    Sometimes building something from the ground up allows you to make fundamental design decisions not available in the alternatives.
  - vosper 10y ago
    Sometimes you need to let talented engineers build things from the ground up, because it's good for them, makes them happy, and stops them from going to work somewhere else. Keeping people with the skills to solve these types of problems around and happy is also great for recruiting and for helping your less capable engineers learn and grow.
    - jbooth 10y ago
      Sure, but duplicating Kafka? How many man-months did they put into building this, proving it out, and dealing with fallout from any bugs or production issues? What's the point of retaining an engineer who's doing nothing for the business but re-inventing existing successful software?
    - DannyBee 10y ago
      "Sometimes you need to let talented engineers build things from the ground up, because it's good for them, makes them happy, and stops them from going to work somewhere else. " actually, trying to let people do work you don't need done, in order to keep them happy, is a pretty rookie manager mistake. If they aren't passionate about it, and you can't persuade them to do the things you need doing, they aren't the right person for the job. That is always true, even if they were the right person in the past. Your goal in that case should be to try find stuff the company needs done that they want to do, and push them to work on that. But if you find nothing, ...
  - btym 10y ago
    Or forks it and maintains an incompatible, non-contributable internal version based on a release from five years ago.
- Smudge 10y ago
  Arguably there are benefits to developing in-house expertise, and no better way to develop expertise than to architect and build a solution end-to-end. Twitter now has several domain experts on staff who can continue maintaining DistributedLog and/or weigh its benefits against Kafka's and make more informed decisions going forward. Not saying that they couldn't have worked more closely with Kafka's team in the first place, but, hey, now we have two Kafkaesque log services instead of just one. Seems like a win to me.
  - Xylakant 10y ago
    > Arguably there are benefits to developing in-house expertise, and no better way to develop expertise than to architect and build a solution end-to-end There are also drawbacks to consider: Those experts might just decide to leave and then you have an in-house solution that you have basically no chance to find experts on. At least with an open source solution you may have an easier time.
  - cwilkes 10y ago
    > hey, now we have two Kafkaesque log services instead of just one. Seems like a win to me. CantTellIfSerious.jpg

by murph 10y ago

DannyBee 10y ago

Which says " At design time we had concerns about Kafka’s I/O model and its lack of strong durability guarantees‐a non-starter for an application like a distributed transaction log[3]" Seems reasonable, right? Except "[3] Kafka addressed these durability concerns in version 0.8" So they built a whole thing, because they didn't bother to ask or say "hey, if we help fix the durability, would that we welcome?" or even "do you guys have a plan and timeline to fix it?" That's .... not great. Now, maybe there are other reasons, but they aren't elucidated in this blog post :P. Even then, my general view would be "did you approach the community and discuss your concerns or just dismiss them out of hand as infeasible", mainly because my experience is that if you do this, you often find they have exactly the same set of concerns/goals, and just need more resources to make it happen. The desire to build shiny objects is very large. Outside of the paper plans of engineering teams, these things rarely end up up more shiny than what already exists or will be built by the time you are done.

sametmax 10y ago
Or even "we are ready to pay 3 people for a year to build that, if we don't and give you the money instead what can you do ?". The software industry has a real problem contributing to open source, the stuff, you know, allowing them to make money in the first place.
- placeybordeaux 10y ago
  They did open source their solution. It sounds like you are arguing against parallel development. That happens all the time in open source.
  - sametmax 10y ago
    Contributing to other open source projects is very different than to open source your project. NIH is one of the worst problem of open source : companies are spending a lot of money to dev new software instead of improving existing one.
d0vs 10y ago
Seriously this seems like a common pattern in open source: [big company] could just improve [X] but instead builds something from the ground up.
- mgodave 10y ago
  Sometimes building something from the ground up allows you to make fundamental design decisions not available in the alternatives.
- vosper 10y ago
  Sometimes you need to let talented engineers build things from the ground up, because it's good for them, makes them happy, and stops them from going to work somewhere else. Keeping people with the skills to solve these types of problems around and happy is also great for recruiting and for helping your less capable engineers learn and grow.
  - jbooth 10y ago
    Sure, but duplicating Kafka? How many man-months did they put into building this, proving it out, and dealing with fallout from any bugs or production issues? What's the point of retaining an engineer who's doing nothing for the business but re-inventing existing successful software?
  - DannyBee 10y ago
    "Sometimes you need to let talented engineers build things from the ground up, because it's good for them, makes them happy, and stops them from going to work somewhere else. " actually, trying to let people do work you don't need done, in order to keep them happy, is a pretty rookie manager mistake. If they aren't passionate about it, and you can't persuade them to do the things you need doing, they aren't the right person for the job. That is always true, even if they were the right person in the past. Your goal in that case should be to try find stuff the company needs done that they want to do, and push them to work on that. But if you find nothing, ...
- btym 10y ago
  Or forks it and maintains an incompatible, non-contributable internal version based on a release from five years ago.
Smudge 10y ago
Arguably there are benefits to developing in-house expertise, and no better way to develop expertise than to architect and build a solution end-to-end. Twitter now has several domain experts on staff who can continue maintaining DistributedLog and/or weigh its benefits against Kafka's and make more informed decisions going forward. Not saying that they couldn't have worked more closely with Kafka's team in the first place, but, hey, now we have two Kafkaesque log services instead of just one. Seems like a win to me.
- Xylakant 10y ago
  > Arguably there are benefits to developing in-house expertise, and no better way to develop expertise than to architect and build a solution end-to-end There are also drawbacks to consider: Those experts might just decide to leave and then you have an in-house solution that you have basically no chance to find experts on. At least with an open source solution you may have an easier time.
- cwilkes 10y ago
  > hey, now we have two Kafkaesque log services instead of just one. Seems like a win to me. CantTellIfSerious.jpg