3 ms·

It's a "big company" thing. In my opinion, the best way to add mTLS to your stack is to just adjust your application code to verify the certificate on the othe

It's a "big company" thing. In my opinion, the best way to add mTLS to your stack is to just adjust your application code to verify the certificate on the other end of the connection. But if the "dev team" has the mandate "add features X, Y, and Z", and the "devops team" has the mandate "implement mTLS by the end of Q1", you can see why "bolt on a bunch of sidecars" becomes the selected solution. The two teams don't have to talk with each other, but they both accomplish their goals. The cost is less understanding, debuggability, and the cost of the service mesh product. But, from both teams' perspective, it looks like the best option. I'm not a big fan of this approach; the two teams need to have a meeting and need to have a shared goal to implement the business's selected security requirements together. But sometimes fixing the org is too hard, so there is a Plan B.

davewritescode 4y ago
I very much disagree the sentiment that adding mTLS is just “verifying the certificate on the other end of the connection”. You ignore the process of distribution and rotation of certificates which is non-trivial to implement application side.
- alasdair_ 4y ago
  Can’t each service just have a job that calls the Let’s Encrypt api once a day to get a new cert?
- jrockway 4y ago
  I honestly thought about covering a few ideas in the post, but decided it was off topic. The service meshes do include some rudimentary key generation and distribution code, which is nice to not have to build yourself. The simplest thing, if you're deployed in k8s or similar, is cert-manager + a CA + code that reloads keys when the secret is updated (pretty easy to write). This has downsides (good luck when your CA expires!) but it is easy and does keep itself functional. Cloud providers also have a service like this, which protects the root key with their own IAM (and presumably dedicated hardware); it's definitely a route you'll want to look into. What's missing are a bunch of things you probably want to check before issuing keys; was the release approved, was all the code reviewed before release, is the code reading the foo-service key actually foo-service? That involves some input from your orchestration layer; i.e. an admission controller that checks all these things against your policies, and only then injects a key that the application can read. (Picking up rotated keys becomes more difficult, but this might be a good thing. "If you don't re-deploy your code for 90 days, it stops being able to talk to other services" doesn't seem like the worst policy I can think of in a world where Dependabot opens up 8 PRs a day against your project.) This all has the downside that it doesn't really prevent untrusted applications from ruining the security; a dump_keys endpoint that prints the secret key to a log, nefarious code checked into source control but approved (perhaps due to a compromised developer workstation), etc. Fixing those problems is well outside the scope of a service mesh, but something you have to have a plan for. CircleCI didn't! Now you read 3 blog posts a day about how they got hacked. Anyway, not sure where I was going with this, but application teams need to consider their threat model and protect against it. Security isn't a checkbox that can be checked by someone that didn't write the code. Sure, you can get all sorts of certifications this way that look nice on your marketing page, but the certifications really only cover "did they do the bare minimum to look kind of competent if it was 10 years ago". If you have sophisticated adversaries, you're going to need a sophisticated security team.

davewritescode 4y ago

I very much disagree the sentiment that adding mTLS is just “verifying the certificate on the other end of the connection”. You ignore the process of distribution and rotation of certificates which is non-trivial to implement application side.

alasdair_ 4y ago
Can’t each service just have a job that calls the Let’s Encrypt api once a day to get a new cert?
jrockway 4y ago
I honestly thought about covering a few ideas in the post, but decided it was off topic. The service meshes do include some rudimentary key generation and distribution code, which is nice to not have to build yourself. The simplest thing, if you're deployed in k8s or similar, is cert-manager + a CA + code that reloads keys when the secret is updated (pretty easy to write). This has downsides (good luck when your CA expires!) but it is easy and does keep itself functional. Cloud providers also have a service like this, which protects the root key with their own IAM (and presumably dedicated hardware); it's definitely a route you'll want to look into. What's missing are a bunch of things you probably want to check before issuing keys; was the release approved, was all the code reviewed before release, is the code reading the foo-service key actually foo-service? That involves some input from your orchestration layer; i.e. an admission controller that checks all these things against your policies, and only then injects a key that the application can read. (Picking up rotated keys becomes more difficult, but this might be a good thing. "If you don't re-deploy your code for 90 days, it stops being able to talk to other services" doesn't seem like the worst policy I can think of in a world where Dependabot opens up 8 PRs a day against your project.) This all has the downside that it doesn't really prevent untrusted applications from ruining the security; a dump_keys endpoint that prints the secret key to a log, nefarious code checked into source control but approved (perhaps due to a compromised developer workstation), etc. Fixing those problems is well outside the scope of a service mesh, but something you have to have a plan for. CircleCI didn't! Now you read 3 blog posts a day about how they got hacked. Anyway, not sure where I was going with this, but application teams need to consider their threat model and protect against it. Security isn't a checkbox that can be checked by someone that didn't write the code. Sure, you can get all sorts of certifications this way that look nice on your marketing page, but the certifications really only cover "did they do the bare minimum to look kind of competent if it was 10 years ago". If you have sophisticated adversaries, you're going to need a sophisticated security team.