3 ms·

CockroachDB engineer here. > However key management in clusters is an absolute nightmare. They need to look at Docker Swarm for how to build a usable "join" me

CockroachDB engineer here. > However key management in clusters is an absolute nightmare. They need to look at Docker Swarm for how to build a usable "join" mechanism. I'll contend that key management is one of the great unsolved problems in software deployment. I don't want to diminish the value of removing friction from the user experience, but fundamentally there is not that much of a difference between providing a token directly as a command line argument vs. providing a certificate that lives on disk. I'm curious if your frustration is mostly from the overhead of creating node certificates or if there's a deeper problem I'm not seeing. I should also mention: this is an area we're actively looking to improve in! If you have time to write up a slightly more detailed proposal as a GitHub issue, we'd be happy to entertain it.

dsl 8y ago
> I'll contend that key management is one of the great unsolved problems in software deployment. Docker Swarm and Salt minions are really good examples of how it should work from a user interaction standpoint. Let me spin up a dozen or a hundred instances from the same configuration management template without having to seed each one with a secret, or build my own tooling to distribute certs to them. From one central point I approve new joins. Manage the CA and certificates within the cluster, don't expect your customers to do it. Make a big poster for your office that says "Our real customers don't know how to do devops." MySQL took off because dumb people like me can get it to work. Thanks for listening.
smnscu 8y ago
Any chance of integration with k8s/istio cert management? A quick Google search doesn't bring up anything relevant.
trhway 8y ago
>fundamentally there is not that much of a difference between providing a token directly as a command line argument vs. providing a certificate that lives on disk. certificates with related infrastructure is more like a well-managed injected security "aspect" whereis key on the command line is an architecture style of a kitchen sink ad-hoc mix-in.
jillesvangurp 8y ago
I encounter way too much server software that seems written by people that are insensitive to how modern deployments work. Any instructions that involve "just login here, just run X, just fiddle with that file over there", "just chant these incantations and perform our ritual startup dance",etc are kind of broken. The more convoluted those instructions get, the harder and more error prone it gets to automate the deployment in a sane way. Manually messing with files on a filesystem happens on development machines and laptops, never on production servers (unless you are dealing with amateurs). And even there I would argue running docker is the preferred way these days. For me software comes in two forms these days. If it is stateful, I might not run it in docker and I'll be building an AMI using packer and ansible. If it is not, it is a docker image. In both cases I build one image or container (on a build server) for one thing that gets booted many times on many servers in the context of some deployment template. All of my configuration management happens at image build time (packer or docker). After that no modifications should be needed to their file systems and they should be read only for all practical purposes. Any data goes into separate volumes. At boot time you inject all relevant configuration via environment variables. Ideally that is the same for all nodes of a type with no variation. Ideally the software understands that environment variables are (by far) the preferred mechanism to take configuration. Surprisingly few packages seem to get this right and assume some grey beard is going to ssh into a server and be all over their vintage nineteen seventies configuration files with extra weird syntax in some completely non standard location. It's been decades since software deployment actually worked that way. These days those files get templated, written once and never modified again. The template cruft gets driven from a script and that script is driven by environment variables. The more of that cruft is needed, the harder it gets to deploy stuff. So, insisting on per node provisioning of certificates to some location in a filesystem kind of really sucks. I can totally see devops people disabling that instead and relying on firewalls, security groups and network security instead.
- ants_a 8y ago
  Huh? A modern infrastructure setup should have PKI in place to issue certificates to nodes. You are making it sound like SSL has no place in the modern world.

dsl 8y ago

> I'll contend that key management is one of the great unsolved problems in software deployment. Docker Swarm and Salt minions are really good examples of how it should work from a user interaction standpoint. Let me spin up a dozen or a hundred instances from the same configuration management template without having to seed each one with a secret, or build my own tooling to distribute certs to them. From one central point I approve new joins. Manage the CA and certificates within the cluster, don't expect your customers to do it. Make a big poster for your office that says "Our real customers don't know how to do devops." MySQL took off because dumb people like me can get it to work. Thanks for listening.

smnscu 8y ago

Any chance of integration with k8s/istio cert management? A quick Google search doesn't bring up anything relevant.

trhway 8y ago

>fundamentally there is not that much of a difference between providing a token directly as a command line argument vs. providing a certificate that lives on disk. certificates with related infrastructure is more like a well-managed injected security "aspect" whereis key on the command line is an architecture style of a kitchen sink ad-hoc mix-in.

jillesvangurp 8y ago

I encounter way too much server software that seems written by people that are insensitive to how modern deployments work. Any instructions that involve "just login here, just run X, just fiddle with that file over there", "just chant these incantations and perform our ritual startup dance",etc are kind of broken. The more convoluted those instructions get, the harder and more error prone it gets to automate the deployment in a sane way. Manually messing with files on a filesystem happens on development machines and laptops, never on production servers (unless you are dealing with amateurs). And even there I would argue running docker is the preferred way these days. For me software comes in two forms these days. If it is stateful, I might not run it in docker and I'll be building an AMI using packer and ansible. If it is not, it is a docker image. In both cases I build one image or container (on a build server) for one thing that gets booted many times on many servers in the context of some deployment template. All of my configuration management happens at image build time (packer or docker). After that no modifications should be needed to their file systems and they should be read only for all practical purposes. Any data goes into separate volumes. At boot time you inject all relevant configuration via environment variables. Ideally that is the same for all nodes of a type with no variation. Ideally the software understands that environment variables are (by far) the preferred mechanism to take configuration. Surprisingly few packages seem to get this right and assume some grey beard is going to ssh into a server and be all over their vintage nineteen seventies configuration files with extra weird syntax in some completely non standard location. It's been decades since software deployment actually worked that way. These days those files get templated, written once and never modified again. The template cruft gets driven from a script and that script is driven by environment variables. The more of that cruft is needed, the harder it gets to deploy stuff. So, insisting on per node provisioning of certificates to some location in a filesystem kind of really sucks. I can totally see devops people disabling that instead and relying on firewalls, security groups and network security instead.

ants_a 8y ago
Huh? A modern infrastructure setup should have PKI in place to issue certificates to nodes. You are making it sound like SSL has no place in the modern world.