3 ms·

A lot of the complexity is just intrinsic to working on distributed systems, and if you're doing that kind of work, then these products really can help reduce o

A lot of the complexity is just intrinsic to working on distributed systems, and if you're doing that kind of work, then these products really can help reduce overall complexity by giving you an admittedly complex solution that is at least robust and pretty well tested; that tends to be better than constructing your own. I think where a lot of the pain comes from, and the article hints at this, is there are a LOT of problems that do not require a distributed solution, and while you can totally solve those problems with a distributed system, it's not really useful to do so in a way that dramatically increases complexity. This is a known issue, and there are simpler models for these simpler cases. I think the problem is all the hype around k8s has driven a lot of square pegs into round holes.

StillBored 6y ago
Yes distributed systems are complex, but IMHO at least a large part of that complexity is caused by people who think they understand the problem, and go off and create some "great" new technology. Which then gets a lot of press, and other people building systems on it, only to discover that the edge case being ignored as inconsequential starts to create lots of failure modes. Sometimes these critical problems are "human factor" ones like with kubernetes configuration, or they are just a lack of proper engineering hygiene. Then you get another layer of crap on top. Repeat that a few times and what you actually have is a giant mess that frequently only works through the sheer effort of a large operations team. To put this another way, if you read the Jepsen reports overwhelmingly what comes across is a lot of hubris. Its overwhelmingly old stodgy technology these new systems are meant to replace (posgres for example) that are satisfying the promises being proclaimed on high by $NEW_TECHNOLOGY.
- cbsmith 6y ago
  I don't look at the Jepsen reports as being a representation of hubris. Most of what Jepsen finds may have been preventable problems, but they are also fixable problems, which is why I don't see hubris. There's just a reality that we don't really build these systems on top of a distributed operating system. They're starting with an platform abstraction that is already terrible imperfect for the job. There is a ton of pressure to get something out quickly and make it accessible to a broad set of developers rather than to do it correctly (MongoDB is perhaps the most classic example of this). I was building big data systems in the early days, and the whole mentality was that you were building tools that were riddled with flaws, limitations, and outright bugs, but in the right context would make capabilities accessible to a broad set of developers that were otherwise completely off the table. The truth is, if developers were already experts at distributed systems, they wouldn't need or want most of these tools; a LOT of the value is in accessibility and having something that is "good enough" rather than correct. The products reflect that more than they reflect hubris. What I do see is a combination of marketing and customer ignorance that does believe in silver bullets and belief that these tools don't have to be understood to be effective. It's an unfortunate byproduct of focusing on that accessibility over correctness.

StillBored 6y ago

Yes distributed systems are complex, but IMHO at least a large part of that complexity is caused by people who think they understand the problem, and go off and create some "great" new technology. Which then gets a lot of press, and other people building systems on it, only to discover that the edge case being ignored as inconsequential starts to create lots of failure modes. Sometimes these critical problems are "human factor" ones like with kubernetes configuration, or they are just a lack of proper engineering hygiene. Then you get another layer of crap on top. Repeat that a few times and what you actually have is a giant mess that frequently only works through the sheer effort of a large operations team. To put this another way, if you read the Jepsen reports overwhelmingly what comes across is a lot of hubris. Its overwhelmingly old stodgy technology these new systems are meant to replace (posgres for example) that are satisfying the promises being proclaimed on high by $NEW_TECHNOLOGY.

cbsmith 6y ago
I don't look at the Jepsen reports as being a representation of hubris. Most of what Jepsen finds may have been preventable problems, but they are also fixable problems, which is why I don't see hubris. There's just a reality that we don't really build these systems on top of a distributed operating system. They're starting with an platform abstraction that is already terrible imperfect for the job. There is a ton of pressure to get something out quickly and make it accessible to a broad set of developers rather than to do it correctly (MongoDB is perhaps the most classic example of this). I was building big data systems in the early days, and the whole mentality was that you were building tools that were riddled with flaws, limitations, and outright bugs, but in the right context would make capabilities accessible to a broad set of developers that were otherwise completely off the table. The truth is, if developers were already experts at distributed systems, they wouldn't need or want most of these tools; a LOT of the value is in accessibility and having something that is "good enough" rather than correct. The products reflect that more than they reflect hubris. What I do see is a combination of marketing and customer ignorance that does believe in silver bullets and belief that these tools don't have to be understood to be effective. It's an unfortunate byproduct of focusing on that accessibility over correctness.