5 ms·
It's hard to drive non-incremental innovation without a benevolent dictator model. For example, the Hadoop ecosystem moves very slowly, and most of the innovat
by jfim 8y ago
It's hard to drive non-incremental innovation without a benevolent dictator model.
For example, the Hadoop ecosystem moves very slowly, and most of the innovation happens in new projects that are added, not in core infrastructure. An example of moving slowly is the fact that even to this day, the HDFS high availability story for NameNodes is pretty wonky.
In a community model, if someone wants to do a large scale refactoring or rework parts of the architecture, these projects die of a thousand cuts with objections or change requests from many stakeholders. The decision-making then becomes "design by committee," instead of pushing towards a long term vision.
With a single benevolent dictator, since decision making is more centralized, it's significantly easier to do large reworks of the architecture to improve the product in non-incremental ways.
- jacobian 8y ago> It's hard to drive non-incremental innovation without a benevolent dictator model. Hmm, dunno that this is a universal truth. Most of the biggest improvements to Django came after Adrian and I stepped down and we moved to a more democratic leadership structure.
- zapita 8y agoThe key word is “improvement”, which implies that there was something already there to improve. So, a democratic structure can improve Django once it’s already well-defined and widely used. But could it have created Django and made it successful in the first place? I think the answer is no, and that’s what the gp meant by “non-incremental innovation”.
- softwaredoug 8y agoMost of these projects reached that stage before or early in the life of these companies. So by that logic the companies were not necessary to support the projects.
- zapita 8y agoI think “bdfl” and “exclusive corporate sponsor” are 2 distinct issues. I was addressing “bdfl vs. democratic leadership”. I personally don’t think it matters where the bdfl is employed in the early days of a project, there are examples of wildly successful projects with and without an exclusive corporate sponsor.
- jfim 8y agoGood point, they are probably indeed distinct. In some cases the BDFL is employed by the corporate sponsor (eg. Confluent) which makes it hard to separate the influence of each.
- softwaredoug 8y agoI don’t disagree. But many projects have a benevolent dictator but are still run by a foundation not a company (like Python, Ruby, etc).
- jfim 8y agoGood point, hadn't thought about foundations. I wonder if programming languages are different from infrastructure components, though. Programming languages tend to be harder to sell, while infrastructure components can be resold by providing them as a service. Think of Python as a service vs Jupyter as a service.
- softwaredoug 8y agoI was wondering this too. Though it used to be you could monetize programming languages more readily. I wonder as databases approach a commoditization point if the DB market becomes more like the programming language market. Right now turnkey hosting is the main way of making money. But if open source push button distributed database in an arbitrary cloud happened (via easier containerized orchestration than currently exists) we wouldn’t really have much of a hosted database market. Then database open source might start to look more like programming languages and less like money making entities unto themselves. Honestly this is probably the bigger threat to the Mongos, Confluents etc than anything else. And why AWS pushed proprietary tech like lambda.
- jfim 8y agoDistributed databases will always require some kind of specialist knowledge to maintain. The physical deployment of code is actually the easiest part, the big problem is how to keep things running and fix issues once things invariably start breaking. The value proposition for hosted services is "we'll take care of everything, you don't have to worry about operations" while the value proposition for the development companies is "we'll help you run things and we'll reinvest the proceeds in maintaining the platform." The incentive for AWS is to extract maximum profit from open source projects (with no real regards for sustainability), while the incentive for the development companies are more aligned with having a sustainable long term community and product.
- jamesblonde 8y agoYou should check out hopsfs. It solves the namenode problem using distributed metadata and stateless namenodes. Kind of validates your point. It's a fork.