15 ms·
The Twelve-Factor App (2011)
- revskill 3y agoSorry, i put all secrets in my code. I don't need 12 factors at all. It's super easy to deploy to multiple environments, because code is single source of truth.
- Aurornis 3y agoSecrets in code is bad practice because it puts the secrets in source control and gives them to anyone with access to the code. You separate out the secrets to keep them secret.
- revskill 3y agoNext evolution of Git should automatically introspect the code base for comment to leaving out all "secrets". My point is, make the code easy to clone in a runnable shape wihtout any hacking on host system.
- gdprrrr 3y agoYou can configure GitLab to reject the push if you comitted secrets, based on regex marching with common formats for tokens.
- pjs_ 3y agoHell yeah brother. Move fast and break things philosophy
- porsager 3y agoIt's not a problem. Here you go: https://github.com/AGWA/git-crypt https://github.com/AGWA/git-crypt
- samus 3y agoThis only works if secrets are frequently rotated and contain enough entropy. Else, all secrets will eventually be decrypted from leaked repositories and distribution artifacts.
- brodouevencode 3y agoWhat company do you work for? I'd like to cancel my subscription. Seriously...bad idea. One repo leak or even one bad copy/paste in StackOverflow and you're done.
- ris58h 3y agoYou should add 2017 in the title.
- wesleytodd 3y agoyeah I got confused seeing this on the top. in my head said "what is this, 2012?". Am I wrong this existed way before 2017? edit: I am correct, it is much more old. https://github.com/heroku/12factor/graphs/contributors https://github.com/heroku/12factor/graphs/contributors
- larsnystrom 3y agoFirst commit was 2011.
- webmaven 3y agoThe 12 factors go back even further, to Joel Spolsky's "Joel Test" in 2000: https://www.joelonsoftware.com/2000/08/09/the-joel-test-12-steps-to-better-code/ https://www.joelonsoftware.com/2000/08/09/the-joel-test-12-s...
- richardwhiuk 3y agoThat's a rather different set of 12 factors.
- deleted 3y ago[deleted]
- screamingninja 3y agoPrevious discussion- https://news.ycombinator.com/item?id=31198956 https://news.ycombinator.com/item?id=31198956 (102 comments)
- didntcheck 3y agoAnother interesting thread about how these practices have held up or changed over the past decade: "Twelve-factor app anno 2022" Article: https://xenitab.github.io/blog/2022/02/23/12factor/ https://xenitab.github.io/blog/2022/02/23/12factor/ Discussion: https://news.ycombinator.com/item?id=31225921 https://news.ycombinator.com/item?id=31225921
- songshu 3y agoI love this generally but some of it has leaked out to the wrong people and there are strong cargo cults around such things as one codebase===one app.
- notoverthere 3y agoBack in the early days of Docker, I did a whole bunch of work to make WordPress behave as a Twelve-Factor App. It traditionally hasn't behaved as one – which sort of makes sense, because WordPress grew-up in a world where long-lived servers with writable and persistent local disk storage was commonplace. I'm sure things have moved on since those days. This was back in 2016. But it sure was a fun challenge!
- DanielHB 3y agoI remember when I learned about making servers stateless by storing session information in a database and just not writing to disk and stuff. I was amazed at how much simpler it made things and added the capacity to load-balance multiple nodes without having to bother with session stickness. Of course it made things harder in other ways like having a separate DB for sessions.
- frankfrank13 3y agoI like this in general, but the number of times I've had to delay a release because some non/psuedo-technical person has pulled out "12 factor" like a universal yellow card has been enough for me, overall, to ignore it almost completely. I guess the same has happened to me for "agile" honestly. I recognize the intent of guidelines like these, but they seem to be much more useful to people whose only "value" is to provide ivory tower technical leadership.
- Waterluvian 3y agoI've had issues like that before. The solution was to have a clear, documented process, and point them to that. I've learned to take issues like that in good faith. Why are they raising an issue? Is the existing process not clear? Is there a deficiency in it? Is there a lack of reliability? If reasonable concerns are found, the process can change (and the documentation is to be updated).
- ChrisMarshallNY 3y ago> The solution was to have a clear, documented process, and point them to that. This has its own drawbacks. I worked for a company, in which Process Was God, and Thou Shalt Do Only That Which God Has Proclaimed Good. It worked most of the time, but made it very, very difficult to introduce flexibility and orthogonal approaches. There's really no way to duplicate the results of experienced, cohesive teams of high-functioning engineers, with process. We can get similar results, in many ways, but there's always a price to be paid. T. A. N. S. T. A. A. F. L.
- willseth 3y agoGive an example. Like most things, it's context dependent and there are grey areas, but I think most devs use 12 factor like a north star. I wouldn't block a release due to some minor deviation, but generally lacking alignment should be treated as tech debt as a minimum. If a release regresses significantly in one of the factors, I think it's fair to block or at least force more detailed review to understand why the dev felt the tradeoff was worth it for that release.
- ireallywantthat 3y agoNowadays, I see many articles titled "Beyond 12 factor Apps" which proposes even more factors for App prominent among them being one from IBM. What's your opinion on that? https://developer.ibm.com/articles/15-factor-applications/ https://developer.ibm.com/articles/15-factor-applications/
- Moto7451 3y agoThis is a small but real trend. My experience at a company that went deep into expanding beyond/removing some of the 12 Factors is that it breaks portability of containerized software built with those assumptions. There were a few times I needed to install a vendor product or open source project that was built to the 12 factors but spent a month or more just building a shim to make it conform to our particular bend on it. I’m sure there are good reasons to deviate but I wasn’t particularly impressed with our decisions and more importantly why we made them.
- aj_g 3y agoFor sure an influential engineering codex. Feels almost strange thinking about all the easy abstractions we have now for hosting, like Render or Vercel or that this was written in 2012, when web apps were much much more of a wild west in terms of accepted/shared practices.
- RugnirViking 3y agoI was kinda expecting this from the title to be some sort of commentary on two factor authentication, like an app that needs passport photo, face scan, drivers licence, sms text, google authenticator, email link, password, fingerprint, all together to log in just once.
- patrickaljord 3y agoYour average centralized crypto exchange.
- moffkalast 3y agoSounds like roughly what the average stock or crypto exchange requires before they let you trade haha.
- rando_dfad 3y agoThank KYC/AML for that. Money is not data, it is a complex social construct. Moving money requires interacting with that social construct, i.e. government regulations. Yes, crypto was supposed to get around that. No, government didn't agree.
- moffkalast 3y agoWell I'm glad they at least aren't collecting skin samples yet.
- ACV001 3y agoBoring. This is already happening in any sane modern application.
- seanhunter 3y agoIt might be worth you reading https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html In particular: > Be kind. Don't be snarky. Converse curiously; don't cross-examine. Edit out swipes. > Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something.
- gouggoug 3y ago> This is already happening in any sane modern application. Probably in part thanks to 12factor.net
- sbjs 3y agoI genuinely feel like there's two camps of professionals: those who know how to do something and do it extremely well; and those who know how to describe something and break it down very well. The former are the ones making the magic happen, like Fabrice Bellard. The latter become very influential teachers like Bob Martin. Both are needed (although the former more so), but the latter usually get all the attention and credit.
- webscalist 3y agoI prefer to have known configuration presets (groups) version controlled. Documents known shared environments (db host, ... etc) well.
- vilunov 3y agoI feel like each of this points can be reasonably disputed. 1. One app - one repo. There is nothing fundamentally wrong with multiple apps (as in separate pods/containers/processes) being developed in a single repo, as long as they are tightly coupled functionally, share the release cycle, but still need to be deployed separately to get advantages of separate processes and independent scaling. The first example that comes to mind is separation of public api and worker processes, e.g. ruby's sidekiq, python's celery or a generic kafka consumer. 2. > A twelve-factor app never relies on implicit existence of system-wide packages. This is incredibly hard to achieve in practice if you're not running something nix, and even then there are leaks of dependencies on kernel syscall APIs. Every Rust app implicitly depends on a glibc (except musl ones), which is present in major Linux distros except slim ones such as Alpine. In fact, the whole Docker crutch IMO is needed specifically to alleviate this problem. 3. > The twelve-factor app stores config in environment variables It seems to me much more brittle, as it forces you to store secrets in envs, which could be less secure, and to abandon structured file configs, which may enjoy type safety, IDE autocomplete and automatic parsing. With envs you are forced to implement parses for non-trivial and non-string config vars yourself. In fact, envs are often committed into repos as well in form of dot-env files, so the point about commit-safety is moot as well.
- twic 3y ago> This is incredibly hard to achieve in practice if you're not running something nix, and even then there are leaks of dependencies on kernel syscall APIs. You're misinterpreting that guideline. The text says (my emphasis): > Most programming languages offer a packaging system for distributing support libraries, such as CPAN for Perl or Rubygems for Ruby. Libraries installed through a packaging system can be installed system-wide (known as “site packages”) It's not saying not to depend on the operating system (of which glibc is part), it's saying not to require particular language package manager packages to be installed on the machine. Agreed that the other ones you mention are silly, though. Especially the one about environment variables - that isn't actually reasoned advice, it was just the authors writing down what was most common in Ruby development at the time, on the assumption that that must be the best thing to do.
- 3y ago
- twic 3y agoThe big thing missing from this document is justifications for the rules. It's almost all just rules. Are the rules good? Hard to say, and this document won't help you find out.
- brodouevencode 3y agoIf you click on the rule it links to a page that is what you're asking for. For example: https://12factor.net/backing-services https://12factor.net/backing-services
- moondev 3y agoKubernetes basically forces you to deploy 12fa, using these guidelines is a pretty good playbook for migrating an existing application as well.
- sergioisidoro 3y agoTwelve factor app: use environment for config Docker: Don't use environment for config, it's insecure I like and use many of the patterns of 12 factor, but indeed some of these were written in the context of VPS, where environment is stable, secure and more constant (unlike containers, where environment might end up shipped in one of the layers). In the age of containers, this particular point has been a bit of a struggle for me. Docker secrets don't always play well, and you need to do some gymnastics to make it work.
- deleted 3y ago[deleted]
- samus 3y agoThe basic idea is still valid. You might just have to use a more appropriate mechanism when using containers. For example, the explanation of the Config factor explains that a config file can also be used.
- 89vision 3y agodo you have any examples of more secure ways of injecting secrets?
- dharmab 3y agoCheck out the Secrets Operator for Kubernetes. Injects your secrets from a secrets store as a file mounted into your container.
- marwis 3y agoAnd how is /var/run/secrets any more secure than env? I guess you avoid the risk of accidentally logging secrets with other env variables but otherwise it seems to be just as secure/insecure.
- deleted 3y ago[deleted]
- zemo 3y agothe twelve-factor app is a set of recommendations from 2011 that are based less on engineering principles and more on the capabilities of Heroku and containerized infrastructure in 2011. For example: > Another approach to config is the use of config files which are not checked into revision control, such as config/database.yml in Rails. This is a huge improvement over using constants which are checked into the code repo, but still has weaknesses: it’s easy to mistakenly check in a config file to the repo; there is a tendency for config files to be scattered about in different places and different formats, making it hard to see and manage all the config in one place. Further, these formats tend to be language- or framework-specific. > The twelve-factor app stores config in environment variables (often shortened to env vars or env). yeah the reason they argue for this is that the people that wrote this worked on Heroku, and the way that Heroku worked is you populated the environment variables from some fields in a web app. If you want your config history tracked in version control, or you do gitops, or you have k8s configmaps, or you want to have your configuration files on a mounted volume ... those things are all broadly fine; they keep the configuration state separate from the app deploy state. This document really confuses the forest with the trees and recommends things based less on actual engineering principles and more on the product capabilities of the corporation that produced it. It is an actively harmful set of guidelines.
- phendrenad2 3y agoHistory repeats itself. Netlify tried this with the MACH Alliance.
- morelisp 3y agoThe 12 factor app that stored config in env vars found it trivial to migrate to ConfigMaps. The LOB apps today that expect complex ConfigMap layouts (at worst case, directly from the k8s API) are going to be hell to migrate to whatever comes after k8s.
- throw1234651234 3y agoThe whole .env file holding all your configs thing is a disaster. People forget to update the template, it has to be passed around, etc. Configure your secrets in Azure Key Vault (AWS Secrets Manager, GCP-something-I-forget) or whatever the non-cloud Vault software is called. Devs only set client_id, client_secret, and env defaulted to dev. On startup, the app fetches configs for the secret store. The needed secrets are in source control. If the needed secret is missing from the secret store, there is a clear error on startup. You are welcome, I just saved you weeks of onboarding, confusion, and devs passing around .envs on Slack.
- zoogeny 3y agoSeems good as a general heuristic but bad as a requirements checklist.
- lhnz 3y agoIt's pretty bad that, more than a decade after this was released, there are detractors for some of the guidelines but yet nobody is able to point to a singular more modern set of principles. A lot of our advice is now nuanced and spread out all over the internet...
- qwerty4343 3y agoIt's good to see it coming up again. I hope there will be a platform among the new PAAS players that is 100% compatible with the 12-factor app principles.
- PaulKeeble 3y agoI have always disagreed most with the config advice. Config ends up getting defined by a lot of different parties and quite often by developers and its often best to ship the application with reasonable defaults and then allow overrides from environment specific files and then environment variables. This is the most flexible for most server side applications because you often know what a setup should be and its best source controlled but secrets need to be injected at run time. Configuration needs to have cascading levels to make life easiest in dev, test and production without enormous setup time.
- Kinrany 3y agoYou can, and probably should, have reasonable defaults. Configuration should be version controlled, but separately from source code because it describes deployments, not images used for deployments.
- willseth 3y agoWhy not also express the defaults as config, or is that what you meant? I don't think that runs afoul of 12 factor.
- schneems 3y ago> Configuration needs to have cascading levels to make life easiest in dev, test and production without enormous setup time. The backdrop of when the 12factor app was written is that you can toggle modes like that with RAILS_ENV=test (an environment variable). One way I think of the config section is “does your config strategy play well with containers?” Once you build an image you have a static disk state that doesn’t persist changes unless you make a new image. If your config is file based (only), then to toggle between test and production behavior you would have to build a whole new image. By allowing config to change without the underlying disk, it helps to isolate changes. Did the app break because something in the deploy (image generation) broke? Or is it a bad config? If you decouple image generation from configuration changes it eliminates that question.
- ahtihn 3y ago> Once you build an image you have a static disk state that doesn’t persist changes unless you make a new image. If your config is file based (only), then to toggle between test and production behavior you would have to build a whole new image. You could mount a volume with the config file. The nice thing is that this lets you change the config at runtime (assuming the app watches for file changes). Env vars cannot be changed at runtime.
- porsager 3y agoWould be nice to have a simple term for all these absolutist ideas about how to solve a Problem. I nominate the term "The Holy Grail Jail". I've actually had great success doing the opposite to most of these points.
- dathinab 3y agoevery time I read Twelve-Factor App I'm like who they set up twelve factor authentication, that's crazy isn't 2FA enough then I remember that it's about something else
- theusus 3y agoI hate things like these. They just make people architects and SEs create forceful indirection that creates nothing but burden.
- bdcravens 3y agoPerhaps in 2023, but in 2011, we were still at the edge of when people would deploy by moving folders and individual files via FTP, hard-coding credentials, and dropping in extracted zip files as a method of dependency management. Of course even then added layers were often considered "burden", but in general, we need some degree of order lest we create an unmanageable mess of index.old.3 and files edited directly on production.
- adolph 3y agoThe 12 factor app guidelines are still relevant today since many software teams are building distributed systems, which require careful attention to detail as to how services are coupled, configured, and deployed. The 12 factor app is a wonderful starting point for best practices. In my experience though they omit a few facets and leave room for antipatterns in certain facets. https://smallbatches.fm/5/transcript https://smallbatches.fm/5/transcript [00:03:52] Joe Kuttner: But today the infrastructure that we run on is disposable, right? It's much more like livestock or cattle where we can buy new ones when we need one at market and we can dispose of, you know, something's not working. Right. We get. Yeah. So it's a very, it's a very different model. [00:04:07] Joe Kuttner: And that was just emerging when the 12 factor app was first launched. I think it did a great job of helping bring people into that state of mind. https://smallbatches.fm/20/transcript https://smallbatches.fm/20/transcript
- nopurpose 3y agoI wish there was a generic control plane for generating and shipping config to apps. Kinda what Envoy proxy has, but capable of shipping arbitrary schema. Many complications of config management will go away then. Every app instances knows just its ID, then everything else (region, cluster, stage, current experiment, maintenance, anything else) is discovered by "configd" service and config suitable for that very instance is returned. Not unlike feature flags, but for backend services. OPA (Open Policy Agent) comes close: it has data sources (so it can fetch inventory from cloud providers, kubernetes, any other internal system) and Rego language to program config generation, but it doesn't have means to notify instances of config updates, so they have to resort to polling.
- wly_cdgr 3y ago"last updated 2017" if you look at the footer
- RyeCombinator 3y agoThought we are up to 15 now? https://domenicoluciani.com/2021/10/30/15-factor-app.html https://domenicoluciani.com/2021/10/30/15-factor-app.html
- freedomben 3y agoOver the years I've had a lot of discussions around 12 factor apps and encountered a lot of confusion. I find that the 12-factor site is great, but mainly for people that already understand why those things are important. For people that don't already understand the why behind the rules, it took more in-depth explanation. For that purpose, I made the video: "What are 12 Factor Apps and Why Should You Care?"[1]. I know of a couple of companies that use that for new-hire orientation for engineers/devops that have said it's been really helpful. Regardless where you get the information, it's worth taking an hour or two to learn about 12-factor apps. Most of these "rules" are just things that take awareness and aren't immediately obvious unless you've learned them all the hard way by making the mistakes and feeling the pain. [1] https://youtu.be/REbM4BDeua0 https://youtu.be/REbM4BDeua0
- commandlinefan 3y agoIt's like unit testing. The idea of writing code to test your code was so blindingly obvious to me from the time I started programming that I was shocked that it took so long for other people to pick up on it, but the concept of "unit testing" started to gain some traction around the late 90's and I was relieved that somebody with some authority was advocating it. But the reason I didn't just do it myself before Kent Beck published JUnit was because the code I worked on didn't lend itself well to being controlled by other code. Global variables everywhere, widely distributed state, dependence on external systems and specific file system layouts and general disregard for modularity made it impossible to run anything outside of the context it was designed in. All of this was "bad design", but it met deadlines, so everybody did it anyway. I was _hoping_ that if unit testing gained some traction, programmers would get away from monolithic design. After 25 years of observing unit tests for getters and setters and then one big unit test that creates an in-memory database because every function in the app needs a live database just to run and then gets commented out because it fails, I've lost any faith that unit testing will ever be more than a meaningless checkmark that everybody fills out because it's a "best practice" but nobody actually stops to think why.
- berkes 3y agoI'd like to believe that "everybody" and "nobody" in your story aren't as black-and-white. Many people see the value of unit testing. And see how TDD actually shapes your code to be testable and (accidentally, well, not really) therefore also cleaner and better organized. I've always said, and will continue to do so: good, maintainable code, is accidentally also very well testable code. And vice-versa. And it's not a real accident, if you think about it.
- deleted 3y ago[deleted]
- throwaway4good 3y agoHolds up well 12 years later.
- lakomen 3y agoNot this again. Let the horse die already
- commandlinefan 3y agoThis horse or any other horse. I read this and I think "this is good advice, we should follow it." You read it and you think "this is good advice, we should follow it." And then we both do, but our interpretations conflict completely so we end up arguing about the true definition of "implicit existence of system-wide packages".
- robbywashere_ 3y agoFun to compare this to with how insane deployments are now featuring kubernetes manifests which only big brains can understand
- mplewis 3y agoWhat part of the Kubernetes manifest do you think makes deployments insane?
- xdennis 3y agoNot GP, but here's an example I've seen. Kubernetes files are generated by Jsonnet. It is invoked from makefiles. Makefiles are spread across different git modules. The scripts require various Go binaries. Most are installed through a command, some have to be installed manually. If the version is wrong, mysterious errors happen. I once had a problem with the generated files so I deleted the generated directory and regenerated everything. Turns out you can't do that. The generated files are edited, and if you generate from scratch you've wiped out the manual edits. Note that I'm not criticizing Kubernetes here. But when you have a non-simple system with some missing aspects, some people take the opportunity to make it even more complex.
- pizzaknife 3y agomakefile cackles as it observes the exchange
- dang 3y agoRelated: Ask HN: Is 12factor.net Still Relevant? - https://news.ycombinator.com/item?id=36283702 https://news.ycombinator.com/item?id=36283702 - June 2023 (6 comments) 12 Factor App Revisited - https://news.ycombinator.com/item?id=33164407 https://news.ycombinator.com/item?id=33164407 - Oct 2022 (7 comments) Twelve-factor app anno 2022 - https://news.ycombinator.com/item?id=31225921 https://news.ycombinator.com/item?id=31225921 - May 2022 (35 comments) The Twelve-Factor App (2011) - https://news.ycombinator.com/item?id=31198956 https://news.ycombinator.com/item?id=31198956 - April 2022 (102 comments) Twelve-factor app development on Google Cloud - https://news.ycombinator.com/item?id=21415488 https://news.ycombinator.com/item?id=21415488 - Nov 2019 (63 comments) The Twelve-Factor App - https://news.ycombinator.com/item?id=19947507 https://news.ycombinator.com/item?id=19947507 - May 2019 (3 comments) 12 Factor CLI Apps - https://news.ycombinator.com/item?id=18172689 https://news.ycombinator.com/item?id=18172689 - Oct 2018 (247 comments) 12 factor app configuration vs. leaking environment variables (2014) - https://news.ycombinator.com/item?id=15869436 https://news.ycombinator.com/item?id=15869436 - Dec 2017 (2 comments) Ask HN: Alternative to Heroku that doesn't enforce 12-factor - https://news.ycombinator.com/item?id=10628961 https://news.ycombinator.com/item?id=10628961 - Nov 2015 (1 comment) The Twelve-Factor App - https://news.ycombinator.com/item?id=10288216 https://news.ycombinator.com/item?id=10288216 - Sept 2015 (3 comments) The Twelve-Factor App - https://news.ycombinator.com/item?id=9492120 https://news.ycombinator.com/item?id=9492120 - May 2015 (2 comments) Twelve-Factor Applications with Consul - https://news.ycombinator.com/item?id=7780249 https://news.ycombinator.com/item?id=7780249 - May 2014 (2 comments) The Twelve-Factor App - https://news.ycombinator.com/item?id=7547687 https://news.ycombinator.com/item?id=7547687 - April 2014 (1 comment) Building Twelve Factor Apps on Heroku - https://news.ycombinator.com/item?id=6219444 https://news.ycombinator.com/item?id=6219444 - Aug 2013 (1 comment) 12 Factor model for architecting SaaS applications - https://news.ycombinator.com/item?id=6060381 https://news.ycombinator.com/item?id=6060381 - July 2013 (1 comment) The Twelve-Factor App - https://news.ycombinator.com/item?id=5979452 https://news.ycombinator.com/item?id=5979452 - July 2013 (1 comment) 12factor: Methodology for Building Software-as-a-Service Apps - https://news.ycombinator.com/item?id=4027026 https://news.ycombinator.com/item?id=4027026 - May 2012 (1 comment) Twelve Factors of Web Application Development - https://news.ycombinator.com/item?id=3267187 https://news.ycombinator.com/item?id=3267187 - Nov 2011 (37 comments)
- oldbbsnickname 3y agoThe superset of 12factor is platform engineering, especially managing stateful backend components. I worked at a MAANG where both manual discovery (documentation) AND automated discoverability were completely haphazard and often missing because everything was home-grown. Important takeaways should be: - Don't reinvent everything without a specific problem that cannot be solved with reasonable scaling of what already exists. - Theory of least surprise. - Churn is bad. Don't change APIs or functionality for "refactoring" reasons. Replacing something with Rust is no excuse for leaving it broken for a month. - Don't duplicate functionality without differentiated purposes. - Make the purpose of a service and it's use-cases crystal clear. - Name things sensibly rather than giving them cute names. - Standardize stateful storage into a common API that allows for interchangeable and migratable backends. - Document everything into a wiki that's consistent, organized, and is maintained by dedicated, effective technical writers. Don't brain spew text into massive documents! - Following the previous, have a comprehensive service catalog. - Don't assume the reader knows what you know. Explain everything from primitives. - Aim for self-service. - Service owners and/or their operations support must be professional and available enough to provide input when self-service, board messaging, and peer discussions are unable to provide an authoritative resolution. - Reuse, reuse, reuse, and reuse some more. - Build tools that can visualize, chart, and record query results from almost anywhere. - Don't be Twitter: Build selective authorization into every internal API. - Least (but necessary) privilege: issue enough reason-tagged authority delegations such that people can do their jobs. Locking down every bit of internal telemetry and analytics preventing joins between datasources doesn't scale OR create a situation where people are perennially in technical violation of the wording of official policy to do their jobs. - Reward the reduction and efficient maintenance of code because too often, only adding features is rewarded. - Don't make the common cases difficult and awkward. - Avoid code generation. - Avoid unnecessary layers of abstraction. - Avoid building layer upon layer of CLI codegen or service commands lacking in documentation and missing man pages. - Hire some people with actual software engineering, software architecting, systems engineering, and systems operations experience. (The names and responsibilities of these roles seem to change every 5 years now without any particular logic.)