15 ms·

Twelve-factor app development on Google Cloud

228 points by 9nGQluzmnq3M 7y ago

danenania 7y ago
12 factor seems to have stood the test of time really well—I was introduced via Heroku (who I think invented it?) quite a long time ago in tech years, and yet it still seems to be probably the most popular ‘framework’ for devops. In fact, my startup EnvKey[1] was heavily inspired by the 12 factor approach. While it has always worked well for me, one bit that always felt thorny was using the environment for configuration and secrets. It’s obviously great to get this stuff out of code, but then you face new issues on how to keep it all in sync across many environments and pass potentially highly sensitive data around securely. EnvKey fills in this gap using end-to-end encryption and a seamless integration that builds on top of environment variables—it’s a drop-in replacement if you already use the environment for config. Check it out if you’re looking for something to smooth out this aspect of 12 factor! We have lots of folks using it with GCP/GKE. 1 - https://www.envkey.com https://www.envkey.com
- WrtCdEvrydy 7y ago
  > I think invented it I don't think so... the 12 factor app was existing before that... but came to more common usage during the rise of Rails (and Heroku)
  - jbarham 7y ago
    Both https://12factor.net/ https://12factor.net/ and the Wikipedia page https://en.wikipedia.org/wiki/Twelve-Factor_App_methodology https://en.wikipedia.org/wiki/Twelve-Factor_App_methodology credit Heroku developers for formalizing the 12 factor methodology.
    - rarecoil 7y ago
      If you go to the bottom of the page, you'll notice that https://12factor.net/ https://12factor.net/ is written by Adam Wiggins, a Heroku cofounder.
  - glenngillen 7y ago
    It was (former Herokai here).
- paulddraper 7y ago
  Indeed it has. I wanted to put secrets as env vars for my ECS containers. There wasn't an easy to do that. I wound up writing an entrypoint inside the container that would download secrets from AWS SSM and expose them to application code via env vars. Then AWS ECS released the feature to specify env vars as SSM values naively from the ECS agent. It felt good knowing someone else saw this as the "right" way too. --- I will say that Factor 9 (Disposability) seems rarely followed in many applications, SIGTERM is not handled gracefully.
  - m0zg 7y ago
    Program shutdown is not supposed to be handled "gracefully", simply because there's no guarantee that your program will be able to gracefully shut down. Google itself doesn't handle program shutdown at all. Your program must be written so that it's safe to outright kill it at any moment, because at scale that's what's going to happen from time to time whether you want it or not. It is best to shed this illusion that your program will have the opportunity to shut down in an orderly fashion, because when it runs on tens of thousands of computers, you can pretty much count on graceful shutdown not happening at least every now and then. This has the added benefit of making the datacenters easier to manage. Say you have a bunch of workloads packed into racks of servers. Say one of those racks needs an electrical upgrade or hardware replacement. If all programs are preemptible by design, you can just tell cluster management software (Borg) to kill them and restart the tasks elsewhere. Or, in fact, you could even just pull the plug on the rack without telling Borg to do anything. Because workloads are spread out between fault domains, only a small fraction of tasks gets restarted elsewhere, and a properly designed system will not corrupt data or even let the external users know that anything happened.
    - joatmon-snoo 7y ago
      I'm sorry, but this is mostly wrong. Yes, you need to design your code to withstand disaster shutdown SIGKILL type situations. That doesn't mean you get to ignore SIGTERM. The vast majority of shutdowns are due to routine maintenance events. If you get SIGTERM-ed and all you did was crash, here's a short list off the top of my head of bad things that can and do result: * Tail latency goes up, because clients talking to tasks that don't shut down gracefully have to wait for at least one RPC timeout - possibly more if channel timeout is longer - before they retry elsewhere. (This will manifest in multiple ways, because things like load balancers will also be slow to respond.) * System contention goes up - if your server was holding any kind of distributed lock (e.g. DB write) when it went down, everyone else needs to wait for that lock to time out before someone else can take it. (Hopefully your locks require keep-alives to hold them!) * Corruption of data in transit - a crashing binary is basically a big ol blob of UB. With enough replication and checksumming you can mitigate this, but it doesn't mean you get to do dangerous things now! Guardrails only work if you don't take a sledgehammer to them. * I really, really hope you don't have any request affinity in your system, because if you do, now your caches are empty, your dependencies are going to see you doing expensive ops much more frequently, and so on. (And if you're a streaming system, well now you're just all kinds of screwed.)
- thomascgalvin 7y ago
  Twelve factor is one of the few methodologies where, when I drift away from its suggestions, I always stop and ask myself why, because there's a good chance I'm making a mistake.
- glenngillen 7y ago
  Love seeing this! After my time at Heroku I briefly spiked out a similar idea, but never managed to get enough traction to justify making it a full-time effort. I'm excited to see someone actually executing on it. Well done.
ethagnawl 7y ago
> A better approach is to store configuration in environment variables. It's about damn time, Google! I've been working on a number of GCP projects this year and have consistently bumped up against the awkwardness that was Google's (former?) preference for configuration files over environment variables.
- emmelaich 7y ago
  This is the weakest or least well-defined aspect of 12factor. How can you 'store' anything in something as ephemeral as environment variables? Where do these environment variables come from? When are they set? You should of course store everything in version control; the point of this guideline is that the scope should not be the same as the app or an instance. There is particular tension between this guideline and X. environments should be a similar as possible and I. there should be a single codebase for all deployments.
  - emmelaich 7y ago
    Also ... it's a bit circuitous. > requires strict separation of config from code. Config varies substantially across deploys, code does not. What's the difference between config and code? Config is that which changes between deployments / environments.
    - pushrax 7y ago
      Config also benefits from review and version control, and it's inconvenient to use multiple systems for these. It's been working out well for us to store secrets encrypted with public key crypto alongside source, or in a deployment repository using the same source control. The private key is inserted into the infrastructure as a K8s secret.
      - thomascgalvin 7y ago
        One of the concepts twelve factor pushes is that you should be able to release your source code right now without compromising any credentials. Basically, you'd have a git repo for code, and a separate git repo for environment scripts. I've seen organizations use a whole separate infrastructure for config, but that always seemed like overkill to me.
kccqzy 7y ago
> In non-cloud environments, web apps are often written to run in app containers such as GlassFish, Apache Tomcat, and Apache HTTP Server. In contrast, twelve-factor apps don't rely on external app containers. This comment stood out to me. Why does this have anything to do with cloud vs non-cloud? Furthermore I think there are merits to both of the approaches. But the article phrased it as if one is clearly more superior than the other. I think we should also not forget that for the longest time, even before there was Google Cloud, there was App Engine that adopted the former approach: you simply write your handlers in Python, and the server provided by Google runs your handlers. Nowadays in the new Python App Engine runtime they added gunicorn by default so it's less clear.
- thomascgalvin 7y ago
  Primarily because instances are disposable. Getting more traffic to your Foo service than you expected? Spin up another Foo, and let the load balancer figure it out. Then, an hour later, when traffic goes back to normal, kill the new instance. This is easier to do when your service is self-contained. It's certainly possible with an app server, but 12-factor apps tend to view the service as an atomic unit that can be started and stopped wholesale.
  - kccqzy 7y ago
    These are orthogonal. Have you used Google App Engine prior to Python 3? When you get more traffic, the App Engine spins up more instances, each running both the handlers you wrote and the server code you didn't write.
- YawningAngel 7y ago
  I might be able to elaborate on this a bit. At $JOB, we're coming to the end of migrating a bunch of PHP services to Kubernetes from their various homes. I have seen the following configurations: Apache/mod_php,.htaccess disabled; Apache/mod_php,.htaccess required; Nginx/PHP-FPM colocated; Nginx/PHP-FPM using remote FastCGI; We now only use one of these, and every other variant webserver configuration required a non-trivial amount of work to convert. What does this have to do with clouds? In non cloud environments, deployments are pretty sticky - you put something in place on a VM and it stays there for an expected lifespan of perhaps a decade, so the fact that the webserver configuration might be hard to port to a different environment is not especially relevant. Conversely, it's easy enough to chop and change cloud environments (and the ability to do so is in some sense their USP!) that the portability issues of external Web servers come to the fore.
SenHeng 7y ago
I wonder if anyone else does this. I have a private, env-var only repo per project that I very simply import (git clone) while building. I'm sure there are some risks like possible leakage if a team member's github account is compromised but it seems like a very nice, clean way of managing secrets. I even get the benefit of a version control.
- firepoet 7y ago
  We definitely avoid this approach. Fine grained access control per secret per environment is impossible. We use explicit secret management software.
  - dhbradshaw 7y ago
    "We use explicit secret management software" This is interesting. Can you go into more detail?
    - wikibob 7y ago
      https://www.vaultproject.io/ https://www.vaultproject.io/ Fairly easy to integrate with Kubernetes too.
adamwiggins 7y ago
I'm the author of 12factor (although really it is an aggregation of the work and insights from many people at Heroku). It continues to surprise and please me that this piece continues to be relevant eight years later—a virtual eternity in software/internet time. Fun fact: I debated whether to call it "the Heroku way" or somesuch. Glad I went with a standalone name, feel like that allowed it to take on a life beyond that product. For example I doubt Google would have wanted a page about "Heroku Way app development on GCP" in their documentation. :-)
- butisaidsudo 7y ago
  Just wanted to thank you for this. When I was a lost and confused junior, I had no one more experienced around to ask for advice. I stumbled across this and it answered so many questions that I had, and it still very much applies many years later.
- tristanperry 7y ago
  Great work :) When we switched from a monolith to microservices in a previous job, this was mentioned a lot - both internally and to external clients in presentations etc. I agree, I think the fact that it's not called "the [platform/software] way" is a big factor in its success - it's easier to quote publicly and to external clients when it's seen as a more agnostic set of principles. Out of interest, in hindsight do you think that any of the 12 factors could be tweaked/improved (since it was originally written)?
- dc2447 7y ago
  I wonder if your Heroku values, would be more widely known if it had a more generic name https://gist.github.com/adamwiggins/5687294 https://gist.github.com/adamwiggins/5687294 Again, like 12factor, it's a collection of themes from around the internet but it also has stood the test of time.
dathinab 7y ago
Hmm, non of the 12 factors is really be specific to the cloud. I mean ok, the implementation of some of the thinks might slightly differ. E.g. you should still abstract over storage, but it's not necessary a external service as this additional external dependency is just not no always necessary. It's kinda funny how cloud forces people to finally use patterns recommended since years.
rasikjain 7y ago
Its interesting to see this 12-factor popping up today. We used this 12-factor app development methodology at one of my client and they had good documentation about it. This was in 2015-2016 and I believe this is still relevant.
Dowwie 7y ago
Backing Services section is over promising. Changing databases isn't a simple configuration change. Every database doesn't speak the same SQL. It doesn't have the same optimizations nor configuration. Also, your organization probably won't be making that change-- ever. Even if you could swap, you very likely won't. This is similar to the claim that by using an ORM, you can easily change databases except that the ORM actually can achieve it because it abstracts away the differences. Yet, very rarely does this valuable feature get used.
- adrianN 7y ago
  Many organizations are in fact switching databases. It's usually a fairly complicated process, but for a surprising number of companies it makes sense to, e.g. move away from on-prem Oracle to a cloud DB.
- hunterloftis 7y ago
  The backing services factor doesn't mean that you should expect to be able to hot-swap database types. Rather, it means you should be able to "dependency inject" a compatible database into your app. Say your app is designed to speak Postgres - that Postgres db should be able to run locally, or on a managed cloud service, and the app shouldn't know or care which. You should be able to switch between such services without any code changes. In practice, this sometimes does mean that multiple database implementations satisfy the interface expected by your app, and you can in fact hot-swap underlying stores. But the spirit of the factor is that the app should have its dependencies ("backing services") provided, rather than hard-coded.
  - Dowwie 7y ago
    Ah, that makes sense. Thanks for clarifying
bogle 7y ago
Also worth reading is "Beyond the Twelve-Factor App" (O’Reilly). It goes into the 12 factors and expands them to 15 for cloud-native apps. I believe I got a free e-book via Pivotal's content library.
- dv35z 7y ago
  Beyond the 12 factor app: (1) One codebase, one application (2) API first (3) Dependency management (4) Design, build, release, and run (5) Configuration, credentials, and code (6) Logs (7) Disposability (8) Backing services (9) Environment parity (10) Administrative processes (11) Port binding (12) Stateless processes (13) Concurrency (14) Telemetry (15) Authentication and authorization More info: https://content.pivotal.io/blog/beyond-the-twelve-factor-app https://content.pivotal.io/blog/beyond-the-twelve-factor-app

228 points by 9nGQluzmnq3M 7y ago

12 factor seems to have stood the test of time really well—I was introduced via Heroku (who I think invented it?) quite a long time ago in tech years, and yet it still seems to be probably the most popular ‘framework’ for devops. In fact, my startup EnvKey[1] was heavily inspired by the 12 factor approach. While it has always worked well for me, one bit that always felt thorny was using the environment for configuration and secrets. It’s obviously great to get this stuff out of code, but then you face new issues on how to keep it all in sync across many environments and pass potentially highly sensitive data around securely. EnvKey fills in this gap using end-to-end encryption and a seamless integration that builds on top of environment variables—it’s a drop-in replacement if you already use the environment for config. Check it out if you’re looking for something to smooth out this aspect of 12 factor! We have lots of folks using it with GCP/GKE. 1 - https://www.envkey.com https://www.envkey.com

WrtCdEvrydy 7y ago
> I think invented it I don't think so... the 12 factor app was existing before that... but came to more common usage during the rise of Rails (and Heroku)
- jbarham 7y ago
  Both https://12factor.net/ https://12factor.net/ and the Wikipedia page https://en.wikipedia.org/wiki/Twelve-Factor_App_methodology https://en.wikipedia.org/wiki/Twelve-Factor_App_methodology credit Heroku developers for formalizing the 12 factor methodology.
  - rarecoil 7y ago
    If you go to the bottom of the page, you'll notice that https://12factor.net/ https://12factor.net/ is written by Adam Wiggins, a Heroku cofounder.
- glenngillen 7y ago
  It was (former Herokai here).
paulddraper 7y ago
Indeed it has. I wanted to put secrets as env vars for my ECS containers. There wasn't an easy to do that. I wound up writing an entrypoint inside the container that would download secrets from AWS SSM and expose them to application code via env vars. Then AWS ECS released the feature to specify env vars as SSM values naively from the ECS agent. It felt good knowing someone else saw this as the "right" way too. --- I will say that Factor 9 (Disposability) seems rarely followed in many applications, SIGTERM is not handled gracefully.
- m0zg 7y ago
  Program shutdown is not supposed to be handled "gracefully", simply because there's no guarantee that your program will be able to gracefully shut down. Google itself doesn't handle program shutdown at all. Your program must be written so that it's safe to outright kill it at any moment, because at scale that's what's going to happen from time to time whether you want it or not. It is best to shed this illusion that your program will have the opportunity to shut down in an orderly fashion, because when it runs on tens of thousands of computers, you can pretty much count on graceful shutdown not happening at least every now and then. This has the added benefit of making the datacenters easier to manage. Say you have a bunch of workloads packed into racks of servers. Say one of those racks needs an electrical upgrade or hardware replacement. If all programs are preemptible by design, you can just tell cluster management software (Borg) to kill them and restart the tasks elsewhere. Or, in fact, you could even just pull the plug on the rack without telling Borg to do anything. Because workloads are spread out between fault domains, only a small fraction of tasks gets restarted elsewhere, and a properly designed system will not corrupt data or even let the external users know that anything happened.
  - joatmon-snoo 7y ago
    I'm sorry, but this is mostly wrong. Yes, you need to design your code to withstand disaster shutdown SIGKILL type situations. That doesn't mean you get to ignore SIGTERM. The vast majority of shutdowns are due to routine maintenance events. If you get SIGTERM-ed and all you did was crash, here's a short list off the top of my head of bad things that can and do result: * Tail latency goes up, because clients talking to tasks that don't shut down gracefully have to wait for at least one RPC timeout - possibly more if channel timeout is longer - before they retry elsewhere. (This will manifest in multiple ways, because things like load balancers will also be slow to respond.) * System contention goes up - if your server was holding any kind of distributed lock (e.g. DB write) when it went down, everyone else needs to wait for that lock to time out before someone else can take it. (Hopefully your locks require keep-alives to hold them!) * Corruption of data in transit - a crashing binary is basically a big ol blob of UB. With enough replication and checksumming you can mitigate this, but it doesn't mean you get to do dangerous things now! Guardrails only work if you don't take a sledgehammer to them. * I really, really hope you don't have any request affinity in your system, because if you do, now your caches are empty, your dependencies are going to see you doing expensive ops much more frequently, and so on. (And if you're a streaming system, well now you're just all kinds of screwed.)
thomascgalvin 7y ago
Twelve factor is one of the few methodologies where, when I drift away from its suggestions, I always stop and ask myself why, because there's a good chance I'm making a mistake.
glenngillen 7y ago
Love seeing this! After my time at Heroku I briefly spiked out a similar idea, but never managed to get enough traction to justify making it a full-time effort. I'm excited to see someone actually executing on it. Well done.

ethagnawl 7y ago

> A better approach is to store configuration in environment variables. It's about damn time, Google! I've been working on a number of GCP projects this year and have consistently bumped up against the awkwardness that was Google's (former?) preference for configuration files over environment variables.

emmelaich 7y ago
This is the weakest or least well-defined aspect of 12factor. How can you 'store' anything in something as ephemeral as environment variables? Where do these environment variables come from? When are they set? You should of course store everything in version control; the point of this guideline is that the scope should not be the same as the app or an instance. There is particular tension between this guideline and X. environments should be a similar as possible and I. there should be a single codebase for all deployments.
- emmelaich 7y ago
  Also ... it's a bit circuitous. > requires strict separation of config from code. Config varies substantially across deploys, code does not. What's the difference between config and code? Config is that which changes between deployments / environments.
  - pushrax 7y ago
    Config also benefits from review and version control, and it's inconvenient to use multiple systems for these. It's been working out well for us to store secrets encrypted with public key crypto alongside source, or in a deployment repository using the same source control. The private key is inserted into the infrastructure as a K8s secret.
    - thomascgalvin 7y ago
      One of the concepts twelve factor pushes is that you should be able to release your source code right now without compromising any credentials. Basically, you'd have a git repo for code, and a separate git repo for environment scripts. I've seen organizations use a whole separate infrastructure for config, but that always seemed like overkill to me.

kccqzy 7y ago

> In non-cloud environments, web apps are often written to run in app containers such as GlassFish, Apache Tomcat, and Apache HTTP Server. In contrast, twelve-factor apps don't rely on external app containers. This comment stood out to me. Why does this have anything to do with cloud vs non-cloud? Furthermore I think there are merits to both of the approaches. But the article phrased it as if one is clearly more superior than the other. I think we should also not forget that for the longest time, even before there was Google Cloud, there was App Engine that adopted the former approach: you simply write your handlers in Python, and the server provided by Google runs your handlers. Nowadays in the new Python App Engine runtime they added gunicorn by default so it's less clear.

thomascgalvin 7y ago
Primarily because instances are disposable. Getting more traffic to your Foo service than you expected? Spin up another Foo, and let the load balancer figure it out. Then, an hour later, when traffic goes back to normal, kill the new instance. This is easier to do when your service is self-contained. It's certainly possible with an app server, but 12-factor apps tend to view the service as an atomic unit that can be started and stopped wholesale.
- kccqzy 7y ago
  These are orthogonal. Have you used Google App Engine prior to Python 3? When you get more traffic, the App Engine spins up more instances, each running both the handlers you wrote and the server code you didn't write.
YawningAngel 7y ago
I might be able to elaborate on this a bit. At $JOB, we're coming to the end of migrating a bunch of PHP services to Kubernetes from their various homes. I have seen the following configurations: Apache/mod_php,.htaccess disabled; Apache/mod_php,.htaccess required; Nginx/PHP-FPM colocated; Nginx/PHP-FPM using remote FastCGI; We now only use one of these, and every other variant webserver configuration required a non-trivial amount of work to convert. What does this have to do with clouds? In non cloud environments, deployments are pretty sticky - you put something in place on a VM and it stays there for an expected lifespan of perhaps a decade, so the fact that the webserver configuration might be hard to port to a different environment is not especially relevant. Conversely, it's easy enough to chop and change cloud environments (and the ability to do so is in some sense their USP!) that the portability issues of external Web servers come to the fore.

SenHeng 7y ago

I wonder if anyone else does this. I have a private, env-var only repo per project that I very simply import (git clone) while building. I'm sure there are some risks like possible leakage if a team member's github account is compromised but it seems like a very nice, clean way of managing secrets. I even get the benefit of a version control.

firepoet 7y ago
We definitely avoid this approach. Fine grained access control per secret per environment is impossible. We use explicit secret management software.
- dhbradshaw 7y ago
  "We use explicit secret management software" This is interesting. Can you go into more detail?
  - wikibob 7y ago
    https://www.vaultproject.io/ https://www.vaultproject.io/ Fairly easy to integrate with Kubernetes too.

adamwiggins 7y ago

I'm the author of 12factor (although really it is an aggregation of the work and insights from many people at Heroku). It continues to surprise and please me that this piece continues to be relevant eight years later—a virtual eternity in software/internet time. Fun fact: I debated whether to call it "the Heroku way" or somesuch. Glad I went with a standalone name, feel like that allowed it to take on a life beyond that product. For example I doubt Google would have wanted a page about "Heroku Way app development on GCP" in their documentation. :-)

butisaidsudo 7y ago
Just wanted to thank you for this. When I was a lost and confused junior, I had no one more experienced around to ask for advice. I stumbled across this and it answered so many questions that I had, and it still very much applies many years later.
tristanperry 7y ago
Great work :) When we switched from a monolith to microservices in a previous job, this was mentioned a lot - both internally and to external clients in presentations etc. I agree, I think the fact that it's not called "the [platform/software] way" is a big factor in its success - it's easier to quote publicly and to external clients when it's seen as a more agnostic set of principles. Out of interest, in hindsight do you think that any of the 12 factors could be tweaked/improved (since it was originally written)?
dc2447 7y ago
I wonder if your Heroku values, would be more widely known if it had a more generic name https://gist.github.com/adamwiggins/5687294 https://gist.github.com/adamwiggins/5687294 Again, like 12factor, it's a collection of themes from around the internet but it also has stood the test of time.

dathinab 7y ago

Hmm, non of the 12 factors is really be specific to the cloud. I mean ok, the implementation of some of the thinks might slightly differ. E.g. you should still abstract over storage, but it's not necessary a external service as this additional external dependency is just not no always necessary. It's kinda funny how cloud forces people to finally use patterns recommended since years.

rasikjain 7y ago

Its interesting to see this 12-factor popping up today. We used this 12-factor app development methodology at one of my client and they had good documentation about it. This was in 2015-2016 and I believe this is still relevant.

Dowwie 7y ago

Backing Services section is over promising. Changing databases isn't a simple configuration change. Every database doesn't speak the same SQL. It doesn't have the same optimizations nor configuration. Also, your organization probably won't be making that change-- ever. Even if you could swap, you very likely won't. This is similar to the claim that by using an ORM, you can easily change databases except that the ORM actually can achieve it because it abstracts away the differences. Yet, very rarely does this valuable feature get used.

adrianN 7y ago
Many organizations are in fact switching databases. It's usually a fairly complicated process, but for a surprising number of companies it makes sense to, e.g. move away from on-prem Oracle to a cloud DB.
hunterloftis 7y ago
The backing services factor doesn't mean that you should expect to be able to hot-swap database types. Rather, it means you should be able to "dependency inject" a compatible database into your app. Say your app is designed to speak Postgres - that Postgres db should be able to run locally, or on a managed cloud service, and the app shouldn't know or care which. You should be able to switch between such services without any code changes. In practice, this sometimes does mean that multiple database implementations satisfy the interface expected by your app, and you can in fact hot-swap underlying stores. But the spirit of the factor is that the app should have its dependencies ("backing services") provided, rather than hard-coded.
- Dowwie 7y ago
  Ah, that makes sense. Thanks for clarifying

bogle 7y ago

Also worth reading is "Beyond the Twelve-Factor App" (O’Reilly). It goes into the 12 factors and expands them to 15 for cloud-native apps. I believe I got a free e-book via Pivotal's content library.

dv35z 7y ago
Beyond the 12 factor app: (1) One codebase, one application (2) API first (3) Dependency management (4) Design, build, release, and run (5) Configuration, credentials, and code (6) Logs (7) Disposability (8) Backing services (9) Environment parity (10) Administrative processes (11) Port binding (12) Stateless processes (13) Concurrency (14) Telemetry (15) Authentication and authorization More info: https://content.pivotal.io/blog/beyond-the-twelve-factor-app https://content.pivotal.io/blog/beyond-the-twelve-factor-app