3 ms·

> Once Docker, Swarm, Mesos and Kubernetes become more established, then we are likely to see the software industry behave more like an economy of scale. > Eve

> Once Docker, Swarm, Mesos and Kubernetes become more established, then we are likely to see the software industry behave more like an economy of scale. > Even people with basic programming knowledge will be able to create powerful, highly scalable enterprise-quality apps which scale to millions of users out of the box. I must disagree. The real problem with scalability is that any system that scales enough must become distributed, and distributed systems are obnoxiously difficult to reason about, and as such remain difficult to program and to verify. Talk to me about Docker and Swarm and the like hosting technology platforms and frameworks that make it trivially straightforward to program distributed systems reliably, and really hard to program them wrong, and we might have the utopia you speak of.

greenshackle 10y ago
Powerful abstractions make the promise that you'll only have to learn the abstraction to write powerful software, so "even people with only basic knowledge will be able to do X using abstraction Y". The promise is almost always false. All abstractions are leaky, and if you do serious development with them inevitably bugs will bubble up from below and you'll have to dive into the messy internals. For example, ZeroMQ makes distributed messaging relatively painless. Someone with very little knowlegde of the network stack can write simple programs with it easily. But for any serious enterprise application with high reliability requirements you'll eventually run into problems that require deep knowledge of the network stack. >Talk to me about Docker and Swarm and the like hosting technology platforms and frameworks that make it trivially straightforward to program distributed systems reliably, and really hard to program them wrong, and we might have the utopia you speak of. Yes, that.
- jondubois 10y ago
  Your argument definitely applies to Backend-as-a-Service kinds of software and I agree 100%, but the nice thing about the Docker/Kubernetes combo is that it gives you an abstraction but it does so in a way that doesn't prevent you from tinkering with the internals. The only downside I can think of is that tinkering with those internals can become trickier in some cases (because now you have to understand how the container and orchestration layer works). But if you pick the right abstraction as the base for your project, then you may never have to think about the container and orchestration layer.
  - greenshackle 10y ago
    Maybe I was unclear, what I meant is that usually you need to tinker with the internals at some point. Which is fine, but it does mean you need more than basic knowledge to use the tool productively. (And if the software is proprietary and poorly documented, you're SOL). The lie is that this tool is so easy, you just have to read this 30 minute tutorial and you'll be able to write powerflu software and you don't even need to learn the internal mechanics of it. I havn't used Kubernetes, it's possible it's so good that you don't need the learn the messy details, I'm just sceptical of that claim in general.
jondubois 10y ago
Your last line pretty much describes exactly what I think the next phase will be in the container/orchestration movement. The problem with 'Docker and friends' at the moment is that they are disjointed general-purpose pieces (highly decoupled from any sort of business logic) - To make anything meaningful with them, you have to do a lot of assembling (and configuration). I was a Docker skeptic before I stumbled across Rancher http://rancher.com/ http://rancher.com/. In Rancher, you have the concept of a 'Catalog' and in this catalog, you have some services like Redis which you can deploy at scale through a simple UI with only a few clicks. I think that this concept can be taken further; that we can deploy entire stacks/boilerplates at scale using a few clicks (or running a few commands). The hard part is designing/customizing those stacks/boilerplates to run and scale automatically on a specific orchestration infrastructure. It's 100% possible, I'm in the process of making some boilerplates for my project http://socketcluster.io/ http://socketcluster.io/ as in my case, but you do have to have deep understanding of both the specific software stack/frameworks that you're dealing with and the orchestration software that you're setting it up for (and that's quite time-consuming). But once the boilerplate is setup and you expose the right APIs/hooks to an outside developer, it should be foolproof for them - All the complexity of scalability is in the base boilerplate.

greenshackle 10y ago

Powerful abstractions make the promise that you'll only have to learn the abstraction to write powerful software, so "even people with only basic knowledge will be able to do X using abstraction Y". The promise is almost always false. All abstractions are leaky, and if you do serious development with them inevitably bugs will bubble up from below and you'll have to dive into the messy internals. For example, ZeroMQ makes distributed messaging relatively painless. Someone with very little knowlegde of the network stack can write simple programs with it easily. But for any serious enterprise application with high reliability requirements you'll eventually run into problems that require deep knowledge of the network stack. >Talk to me about Docker and Swarm and the like hosting technology platforms and frameworks that make it trivially straightforward to program distributed systems reliably, and really hard to program them wrong, and we might have the utopia you speak of. Yes, that.

jondubois 10y ago
Your argument definitely applies to Backend-as-a-Service kinds of software and I agree 100%, but the nice thing about the Docker/Kubernetes combo is that it gives you an abstraction but it does so in a way that doesn't prevent you from tinkering with the internals. The only downside I can think of is that tinkering with those internals can become trickier in some cases (because now you have to understand how the container and orchestration layer works). But if you pick the right abstraction as the base for your project, then you may never have to think about the container and orchestration layer.
- greenshackle 10y ago
  Maybe I was unclear, what I meant is that usually you need to tinker with the internals at some point. Which is fine, but it does mean you need more than basic knowledge to use the tool productively. (And if the software is proprietary and poorly documented, you're SOL). The lie is that this tool is so easy, you just have to read this 30 minute tutorial and you'll be able to write powerflu software and you don't even need to learn the internal mechanics of it. I havn't used Kubernetes, it's possible it's so good that you don't need the learn the messy details, I'm just sceptical of that claim in general.

jondubois 10y ago

Your last line pretty much describes exactly what I think the next phase will be in the container/orchestration movement. The problem with 'Docker and friends' at the moment is that they are disjointed general-purpose pieces (highly decoupled from any sort of business logic) - To make anything meaningful with them, you have to do a lot of assembling (and configuration). I was a Docker skeptic before I stumbled across Rancher http://rancher.com/ http://rancher.com/. In Rancher, you have the concept of a 'Catalog' and in this catalog, you have some services like Redis which you can deploy at scale through a simple UI with only a few clicks. I think that this concept can be taken further; that we can deploy entire stacks/boilerplates at scale using a few clicks (or running a few commands). The hard part is designing/customizing those stacks/boilerplates to run and scale automatically on a specific orchestration infrastructure. It's 100% possible, I'm in the process of making some boilerplates for my project http://socketcluster.io/ http://socketcluster.io/ as in my case, but you do have to have deep understanding of both the specific software stack/frameworks that you're dealing with and the orchestration software that you're setting it up for (and that's quite time-consuming). But once the boilerplate is setup and you expose the right APIs/hooks to an outside developer, it should be foolproof for them - All the complexity of scalability is in the base boilerplate.