5 ms·

12 factor seems to have stood the test of time really well—I was introduced via Heroku (who I think invented it?) quite a long time ago in tech years, and yet i

12 factor seems to have stood the test of time really well—I was introduced via Heroku (who I think invented it?) quite a long time ago in tech years, and yet it still seems to be probably the most popular ‘framework’ for devops. In fact, my startup EnvKey[1] was heavily inspired by the 12 factor approach. While it has always worked well for me, one bit that always felt thorny was using the environment for configuration and secrets. It’s obviously great to get this stuff out of code, but then you face new issues on how to keep it all in sync across many environments and pass potentially highly sensitive data around securely. EnvKey fills in this gap using end-to-end encryption and a seamless integration that builds on top of environment variables—it’s a drop-in replacement if you already use the environment for config. Check it out if you’re looking for something to smooth out this aspect of 12 factor! We have lots of folks using it with GCP/GKE. 1 - https://www.envkey.com https://www.envkey.com

WrtCdEvrydy 7y ago
> I think invented it I don't think so... the 12 factor app was existing before that... but came to more common usage during the rise of Rails (and Heroku)
- jbarham 7y ago
  Both https://12factor.net/ https://12factor.net/ and the Wikipedia page https://en.wikipedia.org/wiki/Twelve-Factor_App_methodology https://en.wikipedia.org/wiki/Twelve-Factor_App_methodology credit Heroku developers for formalizing the 12 factor methodology.
  - rarecoil 7y ago
    If you go to the bottom of the page, you'll notice that https://12factor.net/ https://12factor.net/ is written by Adam Wiggins, a Heroku cofounder.
- glenngillen 7y ago
  It was (former Herokai here).
paulddraper 7y ago
Indeed it has. I wanted to put secrets as env vars for my ECS containers. There wasn't an easy to do that. I wound up writing an entrypoint inside the container that would download secrets from AWS SSM and expose them to application code via env vars. Then AWS ECS released the feature to specify env vars as SSM values naively from the ECS agent. It felt good knowing someone else saw this as the "right" way too. --- I will say that Factor 9 (Disposability) seems rarely followed in many applications, SIGTERM is not handled gracefully.
- m0zg 7y ago
  Program shutdown is not supposed to be handled "gracefully", simply because there's no guarantee that your program will be able to gracefully shut down. Google itself doesn't handle program shutdown at all. Your program must be written so that it's safe to outright kill it at any moment, because at scale that's what's going to happen from time to time whether you want it or not. It is best to shed this illusion that your program will have the opportunity to shut down in an orderly fashion, because when it runs on tens of thousands of computers, you can pretty much count on graceful shutdown not happening at least every now and then. This has the added benefit of making the datacenters easier to manage. Say you have a bunch of workloads packed into racks of servers. Say one of those racks needs an electrical upgrade or hardware replacement. If all programs are preemptible by design, you can just tell cluster management software (Borg) to kill them and restart the tasks elsewhere. Or, in fact, you could even just pull the plug on the rack without telling Borg to do anything. Because workloads are spread out between fault domains, only a small fraction of tasks gets restarted elsewhere, and a properly designed system will not corrupt data or even let the external users know that anything happened.
  - joatmon-snoo 7y ago
    I'm sorry, but this is mostly wrong. Yes, you need to design your code to withstand disaster shutdown SIGKILL type situations. That doesn't mean you get to ignore SIGTERM. The vast majority of shutdowns are due to routine maintenance events. If you get SIGTERM-ed and all you did was crash, here's a short list off the top of my head of bad things that can and do result: * Tail latency goes up, because clients talking to tasks that don't shut down gracefully have to wait for at least one RPC timeout - possibly more if channel timeout is longer - before they retry elsewhere. (This will manifest in multiple ways, because things like load balancers will also be slow to respond.) * System contention goes up - if your server was holding any kind of distributed lock (e.g. DB write) when it went down, everyone else needs to wait for that lock to time out before someone else can take it. (Hopefully your locks require keep-alives to hold them!) * Corruption of data in transit - a crashing binary is basically a big ol blob of UB. With enough replication and checksumming you can mitigate this, but it doesn't mean you get to do dangerous things now! Guardrails only work if you don't take a sledgehammer to them. * I really, really hope you don't have any request affinity in your system, because if you do, now your caches are empty, your dependencies are going to see you doing expensive ops much more frequently, and so on. (And if you're a streaming system, well now you're just all kinds of screwed.)
thomascgalvin 7y ago
Twelve factor is one of the few methodologies where, when I drift away from its suggestions, I always stop and ask myself why, because there's a good chance I'm making a mistake.
glenngillen 7y ago
Love seeing this! After my time at Heroku I briefly spiked out a similar idea, but never managed to get enough traction to justify making it a full-time effort. I'm excited to see someone actually executing on it. Well done.

WrtCdEvrydy 7y ago

> I think invented it I don't think so... the 12 factor app was existing before that... but came to more common usage during the rise of Rails (and Heroku)

jbarham 7y ago
Both https://12factor.net/ https://12factor.net/ and the Wikipedia page https://en.wikipedia.org/wiki/Twelve-Factor_App_methodology https://en.wikipedia.org/wiki/Twelve-Factor_App_methodology credit Heroku developers for formalizing the 12 factor methodology.
- rarecoil 7y ago
  If you go to the bottom of the page, you'll notice that https://12factor.net/ https://12factor.net/ is written by Adam Wiggins, a Heroku cofounder.
glenngillen 7y ago
It was (former Herokai here).

paulddraper 7y ago

Indeed it has. I wanted to put secrets as env vars for my ECS containers. There wasn't an easy to do that. I wound up writing an entrypoint inside the container that would download secrets from AWS SSM and expose them to application code via env vars. Then AWS ECS released the feature to specify env vars as SSM values naively from the ECS agent. It felt good knowing someone else saw this as the "right" way too. --- I will say that Factor 9 (Disposability) seems rarely followed in many applications, SIGTERM is not handled gracefully.

m0zg 7y ago
Program shutdown is not supposed to be handled "gracefully", simply because there's no guarantee that your program will be able to gracefully shut down. Google itself doesn't handle program shutdown at all. Your program must be written so that it's safe to outright kill it at any moment, because at scale that's what's going to happen from time to time whether you want it or not. It is best to shed this illusion that your program will have the opportunity to shut down in an orderly fashion, because when it runs on tens of thousands of computers, you can pretty much count on graceful shutdown not happening at least every now and then. This has the added benefit of making the datacenters easier to manage. Say you have a bunch of workloads packed into racks of servers. Say one of those racks needs an electrical upgrade or hardware replacement. If all programs are preemptible by design, you can just tell cluster management software (Borg) to kill them and restart the tasks elsewhere. Or, in fact, you could even just pull the plug on the rack without telling Borg to do anything. Because workloads are spread out between fault domains, only a small fraction of tasks gets restarted elsewhere, and a properly designed system will not corrupt data or even let the external users know that anything happened.
- joatmon-snoo 7y ago
  I'm sorry, but this is mostly wrong. Yes, you need to design your code to withstand disaster shutdown SIGKILL type situations. That doesn't mean you get to ignore SIGTERM. The vast majority of shutdowns are due to routine maintenance events. If you get SIGTERM-ed and all you did was crash, here's a short list off the top of my head of bad things that can and do result: * Tail latency goes up, because clients talking to tasks that don't shut down gracefully have to wait for at least one RPC timeout - possibly more if channel timeout is longer - before they retry elsewhere. (This will manifest in multiple ways, because things like load balancers will also be slow to respond.) * System contention goes up - if your server was holding any kind of distributed lock (e.g. DB write) when it went down, everyone else needs to wait for that lock to time out before someone else can take it. (Hopefully your locks require keep-alives to hold them!) * Corruption of data in transit - a crashing binary is basically a big ol blob of UB. With enough replication and checksumming you can mitigate this, but it doesn't mean you get to do dangerous things now! Guardrails only work if you don't take a sledgehammer to them. * I really, really hope you don't have any request affinity in your system, because if you do, now your caches are empty, your dependencies are going to see you doing expensive ops much more frequently, and so on. (And if you're a streaming system, well now you're just all kinds of screwed.)

thomascgalvin 7y ago

Twelve factor is one of the few methodologies where, when I drift away from its suggestions, I always stop and ask myself why, because there's a good chance I'm making a mistake.

glenngillen 7y ago

Love seeing this! After my time at Heroku I briefly spiked out a similar idea, but never managed to get enough traction to justify making it a full-time effort. I'm excited to see someone actually executing on it. Well done.