14 ms·

How to build software like an SRE

316 points by kiyanwang 4y ago

rcgorton 4y ago
tunesmith 4y ago
I like to think I'm a pretty good architect - my team respects me, I solve a lot of problems that they feel like they can't, I read lists like this and if I don't follow all the advice, at least most of the advice isn't surprising. But there's one thing that at this point makes me feel like I'm taking crazy pills because it seems like I disagree with just about everyone. I think unit tests should be first priority, ahead of integration tests and selenium-ish tests (whether you call them frontend tests, functional tests, whatever) I have experience with all three approaches, but slowly moving from backend to full-stack and frontend, I'm struck by how many people - it almost seems a consensus at this point - argue that integration tests should be your first line of defense. I've long since given up the battle on this; I'm not directly on the frontend teams, so I can't seagull my way in there and contradict the lead. But I think the point that gets lost on people is that the value of unit tests isn't chiefly the output of running the test suite. It's that the process of writing good unit tests forces you to write well-structured code. Code that is well-structured enough that it actually minimizes your need for integration and selenium-ish tests. But I feel like in the industry (not just my employer), I'm surrounded by people who are comfortable with writing integration tests and calling it good, even with spaghetti-code implementations underneath. I've always suspected that the ideal testing setup would be mostly unit tests, integration tests only to fill those gaps, and selenium-ish tests (hopefully rare!) only to fill in the final remaining gaps. But I think the system dynamics are set up such that unit tests are rare (due to wanting to finish the Jira story when the feature starts to work), integration tests are frequent since you can happy-path across a large surface area, and large expensive suites of selenium-ish tests are written by a completely different department since you can write them without having to understand the codebase. It just seems like a recipe for poor overall quality and a lot of wasted work/time/money. But I'm clearly in the minority. Maybe I'm missing something basic that I've somehow never learned in my long career.
- tibbon 4y ago
  I strongly agree. Most of this felt fine, except about testing. I'm frequently tasked with writing reliable services, and I always 100% of the time start with tests. Are they perfect? Absolutely not, but I am able to write, test, maintain and iterate on at-scale critical services with great confidence; and 90% of that confidence can come from well structured tests. My code is generally also easy to refactor, understand, port to other languages, etc. Testing is such a critical part of that. I only test things "in staging" as a last check; and very rarely am I disappointed with the behavior (sometimes the speed or scale, but not behavior)
- smrtinsert 4y ago
  100% in agreement. The largest projects that I've successfully completed have had extensive unit test and test suites in general.
- 3pt14159 4y ago
  What you're missing is that most people in web dev use a full featured framework like Django or Rails and do not need unit tests nearly as much as integration tests because very, very, very often most of the architectural decisions have been made for them and the framework linking it all together is where mismatches between expectations seep in. "Wups, I pluralized this thing that should have been singular in the routes / urls file" Also, when developing APIs for the front end it's pretty unlikely that I need to test Rabbit.permanent_url on its own and much more likely I need to test things like listing all the rabbits for a given rabbit farm that are candidates for sale in the local meat market. Where exactly should this test go? The framework handles all the magic SQL generation and the frontend folks really only care about input -> output. If you're building everything from the ground up, then of course there will be way more unit tests, but with an established framework you don't have to test everything. You trust the framework to get some things right for you.
- phtrivier 4y ago
  Your ideal is what is commonly described as the "testing pyramid". It's very well suited to "single artifact" code (a lib, an app with an easy way to drive interaction through a gui, etc...) It tends to turn into a "testing sandglass" with time (because most of the value of integration tests is also derived from gui/e2e tests.) Depending on the app, it might makes sense. Kent C Dodds (author of the poorly named "testing library") is embracing this sandglass shape by calling it the "testing trophy". Honestly, to me the hardest deterrent to testing are : * If it's not done from day 1, you end up with that one hard to test piece of code that makes every other piece of code hard to test * Few people enjoy writing tests (I do, but I reckon I'm part of a minority.) Do what helps you the most !!
AtlasBarfed 4y ago
cpach 4y ago
Good article. Seems like solid and useful advice.
nuker 4y ago
> Kubernetes gives infra teams scalability superpowers Gives nightmares too. Im sticking with Fargate, full, native IaaC support, feature parity with k8s. Or plain EC2s in ASG.
- dividedbyzero 4y ago
  Fargate has full feature parity with Kubernetes? Is that true? Also, what exactly is "full, native IaaC support" in this case, compared to e.g. EKS?
  - nuker 4y ago
    I meant ECS with fargate launch-type, of course. IaaC support is you can use single CFN template to deploy autoscaled service, with load balancing and run time parameters. No kubectl or helm needed.
    - dividedbyzero 4y ago
      Question stands for ECS too, does it actually have full feature parity by now? Last time I used it it was still more limited.
      - nuker 4y ago
        Give me one, not too esoteric plz
      - crgwbr 4y ago
        I’m sure it doesn’t have “full” feature parity, but do you really need 100% of k8s’ features? ECS is a lot simpler to work with and has plenty of features for most systems.
        jnsaff2 4y ago
        So like Full Self Driving?
- 4y ago
simplecto 4y ago
Great insights! I came to some similar conclusions as well. I really like the one about failing on missing config parameters. I used to set default params all the time (thinking I was clever). Another big win in the last few years has been feature flagging. I use it extensively to tune performance when in production. I can control batch sizes, delays, retries, and API key rotation. bookmarked!
falafelite 4y ago
Big +1 to “ Never give up on local testing”. The current code base I’m dealing with has many tests that require interaction with a dev environment on the cloud, and occasionally these tests fail due to a timeout or some other thing not related to actually testing the code (and instead reveals that I forgot to refresh my MFA). Additionally the dependency thing can be huge; we ran into a weird bug for installing a particular dependency on our CI system so our test there keeps failing, but being able to run it locally let’s us know that the changes did not break our actual code tests.
- anonymousDan 4y ago
  How does this work in a serverless world? As far as I can tell local dev for most serverless environments that people actually care about is a joke.
  - dijit 4y ago
    Usually there is some sort of local development framework for serveless functions. Most seem to use docker, here’s one for Google Cloud functions: https://cloud.google.com/functions/docs/running/overview https://cloud.google.com/functions/docs/running/overview If you use something like “cloud run” then it’s containers anyway.
  - koblas 4y ago
    For golang, but would work in any language. What's really worked is the idea that your main loads configuration and injects all of the configured interfaces into your handler. You then can build all of the testing on mocks/stubs to test the behavior. If you access a database you access in through the interface which can mimic the appropriate behavior for your code in test vs production. If you need to you can do local integration testing of the db access layer.
  - Hikikomori 4y ago
    With AWS SAM you can invoke functions locally, run api gateway and use step functions. There's also a 3rd party project to run various AWS services locally, can't remember the project name. There's also SAM sync that syncs changes you make quickly, so if you have your own dev copy in AWS you can quickly test changes.
arianvanp 4y ago
> Use Helm. Or some other tool for managing Kubernetes manifests, I’m not picky – the important thing is that you ~never directly use kubectl apply, edit, or delete. The resource lifecycle needs to be findable in version control. I'd say `kubectl apply` fits better with a Git oriented flow than the imperative `helm install` flow. Nothing about helm guarantees or forces you to use version control. Actually the opposite. If `helm install` fails. You need to `helm delete` first. It has no way of declaratively managing updates at all. Have a CI job call `kubectl apply -k ./deploy --prune` on every commit is way nicer than helm in my opinion.
- cassianoleal 4y ago
  Helm also hides complexity in the templates. There's a lot of valid complaints floating around YAML but what's worse than dealing with YAML is dealing with templated YAML. kubectl brings everything to the surface. Helm is an unnecessary abstraction.
  - tecleandor 4y ago
    I think Helm can be nice to template mildly complex stuff you cannot reach simply with Kustomize, but it's true there are lot of things you can't see until you apply them, even when `helm lint` doesn't fail. If you need this templating, a think you can do is use helm for that but not for install/deployment: render the charts with `helm template` and then apply them with `kubectl apply` and/or verify the changes with `kubectl diff`. IIRC Argo CD can show you the differences between the running helm chart and the new version you want to apply if you haven't enabled automated syncing, but I only use Argo CD with Kustomize, so I can't say for sure right now. edit: by reading the article again I think the spirit of that point is not 'use just helm' but 'don't apply stuff directly'. Long term, if there's more than one person working in Kubernetes, that's prone to failure and configuration drift. Doing the templating in whatever language or system that fills your needs and then using something like Argo CD or Octopus to apply them can remove most of those problems.
    - pickledish 4y ago
throwaway012282 4y ago
Sorry for the throwaway. > I’ve been doing this “reliability” stuff for a little while now (~5 years), at companies ranging from about 20 developers to over 2,000 I've been doing it for 20+ years including running critical services in FAANGs. > Use Docker > Use Kubernetes. Hard pass. There's a reason why most FAANGs developed their own packages management and deployment system and keep using them. They are simpler, less bloated, easier to debug. > Deploy everything all the time On your local testbed, maybe, unless you already know a commit is broken. On production, absolutely not.
- adamckay 4y ago
  > There's a reason why most FAANGs developed their own packages management and deployment system and keep using them. They are simpler, less bloated, easier to debug. And for every other company that isn't one of those 5, rolling your own deployment system won't be simpler, leaner or easier to debug. Nor will it, in all likelihood, be as stable, feature rich, observable, etc.
- greymalik 4y ago
  > There's a reason why most FAANGs developed their own packages management and deployment system and keep using them. They are simpler, less bloated, easier to debug. FAANG-appropriate solutions aren’t always appropriate for the rest of the world.
- evercast 4y ago
  > There's a reason why most FAANGs developed their own packages management and deployment system and keep using them. They are simpler, less bloated, easier to debug. Most FAANGs developed their computing infrastructure before Kubernetes gained popularity. After all the investment into building it and fine tuning for their software services (like ads), they probably have a lot more reasons to not migrate to something different. Not just because their systems are "simpler, less bloated, easier to debug".
  - nine_k 4y ago
    Interestingly, it's G in FAANG who came up with K8s in the first place.
ZephyrBlu 4y ago
> Enable limited “instant” config rollouts Hmm, I think "config" is too vague here. I've seen numerous incidents caused by NGINX config changes that would have been far worse if we had instant 100% rollouts, but I'm not sure this is the kind of config change the author is referring to. > Run 3 of everything What is a "thing" in this example? Is it an instance of a service? A database? Something else? If you're using Kubernetes and have autoscaling, what does "3 of everything" even mean? It sounds like the author is basically talking about databases..?
- pronik 4y ago
  > What is a "thing" in this example? Is it an instance of a service? A database? Something else? No, you just run three of everything. While only databases and similarly distributed data storages really need exactly three (or sometimes, just an odd number larger than two) for quorum and network partition detection, there is a good possibility that you'd have three availability zones because of your databases. And since you already have three availability zones, it's only logical to put every service in every zone at least once, so running "three of everything" becomes a rule of the thumb, easy to remember, easy to implement. > If you're using Kubernetes and have autoscaling, what does "3 of everything" even mean? It sounds like the author is basically talking about databases..? If you use autoscaling, you still need to start somewhere. And this somewhere is "three of everything".
- t3rabytes 4y ago
  I think they’re getting at something closer to feature flags?
rytis 4y ago
> Extremely strict RPC settings. I’m talking zero retries (or MAYBE one) [...] I disagree. If we're talking about distributed systems, then one thing is guaranteed - network is not going to be reliable. And if we have 10's or 100's of services, this policy means that at the smallest blip the whole thing collapses like a house of cards. If the concern is "it's hard to troubleshoot", well then perhaps implement better logging ("connectivity to service X has been unreliable with X% failure rate over the last Xhrs" instead of "connection terminated", or no logging at all).
- outworlder 4y ago
  > Extremely strict RPC settings. I’m talking zero retries (or MAYBE one) [...] > I disagree I too disagree with that too, pretty strongly. The network is unreliable. Technology has become pretty good, to the point that we have become spoiled and take the reliability for granted. But it is at its very core an unreliable medium. Applications should expect that and survive accordingly. Now, before packets go out to the network, or after they have made their way inside your network, they are in an environment that you have a degree of control. Errors and retries should be monitored. If they increase and remain elevated, they should be investigated. But guess what, if your services are resilient, you should have some time to investigate before things start breaking down - as they would if you treated everything as if it was _localhost_ Case in point: our app started breaking in horrendous ways once we deployed it in China and it tried to cross over the Great Firewall. Most issues would have been survivable if they just backed off and retried. The GFW doesn't usually like the first attempt to/from a new address, but it will usually allow the traffic after a while. We had to pay a company a not insignificant chunk of cash to get better connectivity (that would have been fine as an improvement, but not to make the system work at all). Retries are at the core of everything we do. TCP has retries (if the author followed their own advice, they should switch to UDP!). Kubernetes has a whole bunch of retries before reaching error states: CrashLoopBackOff, ImagePullBackOff. Your ethernet card has retries. But it also tracks errors. Track errors. And have retries. For as long as it makes sense to retry for your use-case. > well then perhaps implement better logging Not logging. Metrics! Logs are good for troubleshooting. For signaling issues, metrics are better. Errors can be a simple counter that you increment. Have Prometheus or similar scrape that and throw alerts as necessary.
bigbluedots 4y ago
Lots of YAML, some duct tape.
- keyle 4y ago
  Hah. That made me smile. But that's my issue with most "kids" these days. The code is in rust, takes dark wizard incantations to understand, a treasure map to even compile, ... And then the rest is built on yolo-yaml reproducible like that time I did pancakes that didn't suck.
kbrannigan 4y ago
> Use docker Heh nah, I’ve started learning ansible and it’s so much easier for me to understand. For some reason docker is a little to complex for my purposes. Ansible just runs ssh commands , no extra layer of abstraction I found that the fewest abstractions the better, because at some point you’ll have to debug both the abstraction and the bottom layer.
- pronik 4y ago
  You'll find yourself in the world of pain the next time you need to remove something. Over time, your playbook will be full with stuff that shouldn't be on the machines. At some point you'll consider re-installing the machines from scratch from time to time. At some later point you'd consider re-installing machines every time your Ansible playbook changes. And at that point you'll have reimplemented containers. Poorly.
- nine_k 4y ago
  It's very easy to understand. With Ansible, you first start a machine, then build a configuration on it. With Docker, you first build the configuration, then start the machine (not a real VM, but an isolated enough slice). The upside of a Docker image is that you build it once, and then start instantly, no need to wait for Ansible to run through steps. The downside, but also an upside, of Docker is that you can't make persistent changes inside it while it's running, and usually you avoid such changes. You always can restart it from a known state. Of course Ansible can handle more aspects than just application software configuration, basically any aspect at all. This is its upside, but also a downside: you can make an unintended system change along with some innocent-looking operation by mistake, and nothing will stop you, there's no separation of concerns.
  - kbrannigan 4y ago
    Thanks for that detailed explanation. So it’s possible to pair Ansible with docker? Like only the app on docker but all the runtime, Apache, dependencies configured using Ansible ?
    - 4y ago
KaiserPro 4y ago
I agree with most of these points. They are a good starting point. I'd probably steer clear of kubernetes until you actually really really need it. ECS is good enough for most things, and a fucktonne more simple to look after. but, where I hard diverge is the lack of metrics. Everything should be generating metrics. Logs are great, but they are crap for giving you near realtime trends. Everytime your container gets a connection: increment a counter. Every message processed, counter. every message failed, counter. Every KB of data sent out, counter. Every service call, counter by service. metric all the things, in a sensible, mostly automated way, with a decent schema. Then you can combine all those metrics into a dashboard that shows your system performing against business goals.
kasey_junk 4y ago
This list has things I agree with and things I don’t. You could probably make another similar list that didn’t overlap at all. That’s because the real way to build software that’s reliable has very little to do with a list of features/tools it has to do with designing for the sad paths at the same time as the happy paths. For each failure modality the app should have a conscious decision about what the app will do when the bad thing happens. This can be everything from “this will cause an outage” to “this will route to an alternate system”. What’s important is figuring out up front what those decisions mean for your data, your operations, your business and your users. Without that all the structured logs and version controlled configs in the world are just window dressing.
dimgl 4y ago
Good article, until: > Use Kubernetes. Dear God no lol If you're running a startup that's starting to scale dramatically, I think maybe this is valid advice. But nowadays you can keep things dead simple and still be able to accomplish most business use-cases.
- pgwhalen 4y ago
  Time and time again, commenters suggest that the reason to use Kubernetes is to enable easy scaling, but this is not the case in my organization. Kubernetes is valuable because it is a standardized and flexible deployment model. Easy scalability is a nice side effect of that.
- jstx1 4y ago
  Doing simple things in Kubernetes isn't that much more complicated than doing simple things outside of Kubernetes.
  - mikedelago 4y ago
    I agree - I think the hardest hump for developers is to learn Kubernetes first. After that, it's really not bad for most workloads.
- w0m 4y ago
  Kube has hit enough market/mind saturation that using it for more trivial use case isn't isn't necessarily a horrendous idea anymore. Just having a Kube available I can run a quick kron on is valuable.
  - rednerrus 4y ago
    How much overhead does it require to feed and water k8s?
    - w0m 4y ago
      After initial setup, a managed GKE/AKS is pretty hands-off.
chrisseaton 4y ago
I saw a great talk at EuRuKo last week about treating your CI like an SRE would and using metrics and tripwires and things, by Mel Kaulfuss.
Havoc 4y ago
>The highest-value-per-time-spent kind of test is just pushing your change to staging (or better, prod!) and showing it does what you wanted and doesn’t break everything Ah yes, the time honoured testing in prod strategy
- 411111111111111 4y ago
  "Everyone's got a test system, some also have a stable test system"
- pickledish 4y ago
  Well, "this but unironically", really! One of the main tenets of how I hold SRE is that you should be confident in sending pretty much _anything_ to prod, because you trust the system you've built around it to catch problems before they lead to an outage. And in a world like that, with good shadow/canary deployments etc -- there's no better way to validate your change, right?
  - harshaw 4y ago
    unless you run a service that stores data or does something that can't fail in production.
mkl95 4y ago
I have worked at some places where SRE guys were a joy to work with. But at my current place they take a cowboy approach and make my life harder. So I'm currently not interested in building software like an SRE.
- zaphar 4y ago
  SRE's should not be cowboys. They should be the least cowboy engineers in your company. Something is very wrong there.
  - mkl95 4y ago
    That used be my perception as well. I have now had back to back jobs where there is some maverick in the SRE team. A single guy who modifies some configuration at 2AM for services your team supposedly owns can ruin your week. Some people think lack of bureaucracy is a get out of jail card, and they are often right.
    - zaphar 4y ago
      As someone who has managed a team of SREs that kind of behavior would land someone on a PIP if done repeatedly. Sounds like you had a run of bad luck in teams.
      - mkl95 4y ago
        A PIP would be nice tbh. At my previous place they PIP'd one of those guys, but they ended up moving him to my team, where we taught him a bit about processes.
Bayart 4y ago
> Use Helm. Or some other tool for managing Kubernetes manifests, I’m not picky – the important thing is that you ~never directly use kubectl apply, edit, or delete. The resource lifecycle needs to be findable in version control. I have to partly disagree with that one. I find tools like Helm to obscure things that should be readily visible. My favoured method is to keep manifests in full (which you can source from `helm template`!) as pure yaml files and version that. If possible freeze versions and go through regular patch cycles to review updates. That you apply them through `kubectl apply` or through Argo is irrelevant. I treat the repo as the state and the running cluster as stateless. If it's borked, just redeploy. I don't see it as useful to care too much about the in-cluster resource life-cycle. But I completely agree that resources need to be version-controlled.
- thisiswrongggg 4y ago
  Agree 100%. Plus Helm runs against a basic tenet of microservices (the usual architecture for the deployed apps in k8s these days). People use to bundle services together when using Helm and the like which, in time, couples services together.
  - dividedbyzero 4y ago
    But don't you actually want a certain amount of coupling in the operations part? After all you need to ensure they run in a place and in a manner that allows them to find and talk to each other and usually in a combination determined by the goal of a specific deployment, i.e. sometimes you might not need a rail-cargo-service because the customer only ships by truck, etc. Then scaling/autoscaling (if any) needs to be compatible, versions need to be within a certain range, any central data store must be coordinated as well, not to speak of service meshes, chaos experiments and the like. It's a good thing to develop services with minimal coupling, but that stage has different risks and goals from devops/ops, at least in my experience on both sides of the dev/ops transition zone.
    - mikedelago 4y ago
      > But don't you actually want a certain amount of coupling in the operations part? In my opinion, for sure. There's a balance between "too coupled" and "too de-coupled" that should be stricken rather than too far on either side. It's good to say that this is also contextual; some projects may be fine with either more or less coupling than others, and that's OK.
jasonhansel 4y ago
> Never give up on local testing. It keeps dev cycle time much shorter than needing to rely on (and fiddle with) CI or remote workspaces. Excellent advice for improving productivity and testability. I wish more teams would be sure to follow this by not adopting technologies that are impossible to test locally.
pluc 4y ago
> How to Build Software Like an Site Reliability Engineer Acronyms don't excuse poor grammar.
- pxx 4y ago
  The "S" in "SRE" is pronounced "ess" which would take an "an" as the article. The title is grammatically correct. You would also say "an FBI agent" or sending "an SOS message."
- jasonhansel 4y ago
  No, "an SRE" is correct here; it's actually "a SRE" that would be bad grammar. Whether you use "a" or "an" depends on whether the next word starts with a vowel sound, not with a vowel letter; since the letter "S" is pronounced "ess" in an acronym, you would write "an SRE," but "a Site Reliability Engineer."
- unfunco 4y ago
  It's an initialism, not an acronym. And _an_ SRE is correct.
  - morelisp 4y ago
    I have bad news for you about something I heard a recruiter say a couple weeks ago!
gocartStatue 4y ago
I disagree so much with discarding unit tests, if you’re not dead sure of the base units, then you have a combinatoric explosion down the road. How do you even design code without unit tests :D
- bitpow 4y ago
  Discarding unit tests is such terrible advice. The author clearly never discovered regression bugs at build time, or developed using a dynamically typed language. Also, I would have expected to see "Use only strongly typed languages" as one of the bullet items. I get the sense that the author doesn't actually have a ton of enterprise experience.
pickledish 4y ago
Ah hello, author here! I wrote this up last week and posted it to Reddit (didn’t expect to find it on HN!) and I should probably say the same thing I did over there — this is a list of things I have learned are important, based on my own experiences. I’m sure others have had different experiences leading them to different opinions (sometimes even opposite to mine!), which is good, and I’m enjoying reading all the viewpoints here :)
jasonhansel 4y ago
> Structured logs are non-negotiable I disagree with this. Logs should only be used by human operators; if you want something machine-readable, you should expose metrics via some other channel or record errors in a dedicated service. Sure, some tools will let you read structured logs quite cleanly, but it will never be as nicely formatted as something designed, first and foremost, for humans. Of course, you may need to (say) tack on a request ID to avoid logs from different threads/goroutines/etc from getting interleaved, but that doesn't require everything to be in JSON.
- outworlder 4y ago
  > Logs should only be used by human operators I suspect that you will find a lot of resistance to the idea. At least I have. Machines should be consuming metrics, not logs, unless there aren't any alternatives. That statement never seems to be controversial, but people... just don't do it, for whatever reason. Also, the author seems to want to do a poor man's version of distributed tracing.
- Matthias247 4y ago
  There's some logs which you want machine readable that are not necessarily metrics. Let's say you have 10 services in serial, and you want to get precise information on how that request was performed - because your customer reported it performed super bad. To do that you might need to get logs from all those 10 services and align them. If they all are human readable and you can filter by a common request-id, that's easy to do. And you can even have tooling on top of it that automatically determines what might have happened by looking at other fields of the log entries. Metrics are mostly just pushed to a metrics system, and that moment you don't care about them or the relation to an end-to-end request anymore. So while all services might have emitted a metric for the request, looking at the overall metrics won't tell you if outliers in the services refer to the same end-to-end request or different ones.
- gravypod 4y ago
  Structured logs can still be human readable. Ex: [2022-01... INFO] trace=abcd123 <msg> There is a structure which can be parsed.
omginternets 4y ago
A colleague of mine once shared his lay sociological theory about dev vs ops, and if taken for what it is -- an essentialization -- it's an interesting perspective. The idea is that ops people have inherited a blue-collar culture, whereas devs have inherited an office-worker/academic culture. Ops people conceive of their work as fundamentally operational: progress is measured in terms of actions taken, and while automation is greatly valued, there is nothing inherently "messy" with one-off fixes; the objective is to get things working now. The pathological case for this mindset is that of constantly being on the back-foot, responding to incidents with one-off fixes without recognizing that many of them share a common cause that could be addressed. Dev people conceive of their work as fundamentally intellectual: progress is attained when the problem is correctly conceived, at which point the solution follows naturally. While writing code is greatly valued, most effort should be spent understanding the problem; the objective is to solve it correctly, once and for all. The pathological case for this mindset is that of over-engineering by an ivory-tower idealist, disconnected from the messiness of real-world praxis. In nearly all orgs I've seen, the proportion of first-generation college grads is greater in ops teams than in dev teams. So too is the proportion of people who come from blue-collar families, or are mechanically inclined (ex: look around and see who tinkers with cars). Likewise, the proportion of people holding graduate degrees is greater in the dev crowd (ex: look around and see who's into math). It should hopefully be clear that neither is superior to the other. The point is rather that the divide between dev and ops is partly sociological, which means it is largely based on values. Ops will tend to over-value "honest hard work" and dev will tend to over-value "clear articulate thought". There is also some latent, historical tension between these sociological groups, which has a funny way of masquerading as a technical problem. It is helpful to view arguments about "just ship it" vs "design it the right way" through this lens. Far from being a cute "just so" story, it's been my experience that this dynamic is very important for two reasons: (1) it's harder than you might expect to foster the sense of common destiny required for real "devops"-style collaboration, and (2) each side of the dev-ops divide has a lot to gain from learning when the other side's mindset is helpful, and how to cultivate it in themselves.
- slyall 4y ago
  Interesting idea but perhaps you are confusing attitudes to code with attitudes in general. eg Ops tends to be very attached to getting the infrastructure correct whereas developers care about that less. Developers will write an application that handles all sorts of data edge cases but doesn't react well to network errors.
molodec 4y ago
Great, but it is just a way to build reliable software. Many companies run reliable software without using half of the suggested points.
molodec 4y ago
> Avoid state like the plague This really depends on the problem domain. Application state is fine in many areas, and in the case the state is non trivial managing it in the application is a lot easier that pushing to databases.
dainiusse 4y ago
The title should be "how to build software for an SRE"
- pronik 4y ago
  I'd argue that without SRE you won't be delivering any software to the customer.
0xbadcafebee 4y ago
"Extremely strict RPC settings. I’m talking zero retries (or MAYBE one), and a timeout like 3x the p99. We are striving for predictability here, and sprinkling retries or long timeouts as a quick fix for a flaky downstream service will turn into a week-long investigation and a migraine a year from now. Fix the flaky service!" You want the 80%, right? Might be the network is flaky, might be a service, might be a load balancer. Use timeouts and retries to ignore the flakiness until it eats into your error budgets. "Never give up on local testing. It keeps dev cycle time much shorter than needing to rely on (and fiddle with) CI or remote workspaces. Containerizing the local test environment can make it easier to keep dependencies straight and consistent across machines." Until debugging the snowflakey local testing, or the difference between local dev and CI, ends up eating more time than just using a local<->remote sync tool. There's like a thousand tools now to combine local dev and remote envs, and they will save your organization a lot of time and hassle. "Your git repository is your point-in-time-recoverable source of truth." I call this "suggestion-of-truth". The truth is you never know what state things are actually in, but you hope it's like what's in Git. And there's never enough information kept in Git, like all the versions of all your services used at each deploy, or the state of the entire system, or filesystem snapshots, so recovery often fails. "For infra changes, make plans extremely obvious. This could mean “post the Terraform plan as a comment on the pull request”," Sadly this isn't enough as they might not be using a plan file, and when they apply something unexpected happens. Even if you have a plan file, there's a million other things you should be "making obvious" at deploy time, but people never do. Hard problem to solve. "Use Kubernetes. Assuming you have more than one service and more than one instance, you either need or will need stuff like service discovery, autoscaling, and deployment versioning, and you do not want to manage all that by yourself. Kubernetes gives infra teams scalability superpowers." Yeah, if you pay for it, in money, in expertise, in person-hours. Those things don't just come out of the box. Autoscaling: not at all. Service discovery: you have to set up a DNS controller with an external nameserver, define a naming + namespace convention and stick to it, and use external service definitions for stuff not hosted in K8s. Deployment versioning: you have to define a deployment pattern that stores the versions of everything at deploy time and then link that to your K8s deploys. There's still weeks/months of setting shit up, and then dealing with the complexity of it all and the constant upgrading. Do literally everything you can to abstract away K8s setup + maintenance. Pay anyone to do it all for you. If you're unwilling to pay for it, don't use it. If you do it yourself, even with a managed control plane, it's like building a car from scratch. People on HN sometimes say "oh setting up / maintaining K8s was easy" and I think of the guy who built his own 2-story wooden deck saying it was easy. I wouldn't stand near it.
ChrisMarshallNY 4y ago
> Use Git. Use it for everything – infrastructure, configuration, code, dashboards, on-call rotations. Your git repository is your point-in-time-recoverable source of truth. I'd amend that to "Use configuration management." Git is wonderful, but it isn't a Swiss Army knife. There are some things that are better archived using systems like Perforce (many game developers use it, because Very Big Assets), or even simple incremental file system backups. Back in the "dark ages," the Japanese release managers at my company used to clone an entire computer disk drive onto an external disk, and store that disk in a special file cabinet, so you not only had the source, you also had the entire system that was used to build it. This has some obvious issues, but it worked for them, for years. Also -and I cannot stress this enough- TEST YOUR BACKUP RECOVERY. I have been in the position of needing to restore old files (either partially, or whole-hog), and found out, the hard way, that my backups were pooched.
ryanthedev 4y ago
I couldn’t agree more on these bullets. People forgot what it means to build 12 factor applications.
daitangio 4y ago
I agree on most of the bullet points. As a Software Architect (the SRE mirror role :) I am following https://12factor.net/ https://12factor.net/ and that guidelines simplified a lot my CI/CD pipelines. Helm + K8s is complex but you can easily enable to change some param and redeploy in a very fast and relable way. And yes, docker outperform ansible on a lot of use-cases in my humble opinion.
sobriquet9 4y ago
> Don’t waste time on code coverage Is it really that useless? I was about to start doing code coverage and now wonder if I should reconsider.
- cgrealy 4y ago
  Code coverage doesn't tell you much IMO. I'll take a system with 70% coverage that's focused on the really complex pieces over a system with 99% coverage that's mostly checking that "person.Name" returns the name. Your engineers should know where be dragons in the code and unit testing appropriately.
- jjice 4y ago
  What do you mean by "doing code coverage"? Tracking it? Doesn't take too long to set up in some situations. Increasing it? Depends on if that's trying to get to 100% or just to have a metric to see where tests are needed. It's usually not productive to enforce it for long ongoing development projects (aside from maybe "stay about 70%" or something). It can be a neat metric, but I don't think it's very productive. Testing comes in the form over very unbalanced importance (see "Pareto distribution"). Start by testing the most important business critical functionality and then move out from there. 20% of your tests will cover 80% of your actual important use cases. The trade off would be easier to manager if tests didn't require maintenance, but to get the best bang for your buck, focus on the most critical stuff first. Code coverage is by no means a necessity.
  - sobriquet9 4y ago
    I was thinking about including the necessary instrumentation to check code coverage during regression tests. Something like coverage.py. The intent is not to get a specific number, but to understand which parts of the code might not be exercised enough by existing test cases.
- UncleSam 4y ago
  > My goal here isn’t “what is 100% the most reliability-oriented way we can build things”, it’s more like “what is the 80% of reliability we can get for 20% of the effort while still enabling devs to go fast” The author also points out that these bullet points will not apply to every unique situation. If your goal is to allow devs to go fast at the cost of more hardened code, then yes, skipping code coverage is a major way to code faster. There are definitely times when it is appropriate to skip writing tests when the primary goal is to go fast. It's difficult to answer your question without knowing your specific situation though.
jimcavel888 4y ago
ransom1538 4y ago
IMHO, How to build software like an SRE: 1. Let us know when something is broken. If a curl fails, log it. We want more logs not less. 2. Make the code configuration file free. Use ENV vars. If an ENV var is missing crash and say why. The environment will determine how the app acts. 3. Build process and Deploy process should be different. Github actions is great for the Build process, Jenkins is great for deployments.
bsnnkv 4y ago
As a counterpoint to part of this article, I just posted an article on HN in which an SRE turned solo dev ditched Kubernetes for Fly after 2 years and rising costs.[1] VMs are good enough for high-traffic sites like HN and Pinboard. I think they can be good enough for most solo devs as well. [1] https://news.ycombinator.com/item?id=33235042 https://news.ycombinator.com/item?id=33235042
woopwoop24 4y ago
helm is a bad solution to a lazy thought out problem. why would i template yaml from another yaml and without versioning and weird if else constructs just for deploying which clearly should be json? because we devops are obviously not able to abstract the problem. oh you switched from nginx to haproxy, lets change all the labels in my git comitted repos where my yaml lies. oh the author changed the way the helm chart works, lets do manual compare to see what changed. Oh k8s changed the api version again, lets change all the yaml. and helmfile on top of helm -> more abstraction in the wrong direction -> love it. i would much rather use something like starlark and program my deployment needs against the correct library version of the k8s api which my cluster has and get the final yaml for free instead of templating and forking and maintaining dead weight every damn version.

316 points by kiyanwang 4y ago

I like to think I'm a pretty good architect - my team respects me, I solve a lot of problems that they feel like they can't, I read lists like this and if I don't follow all the advice, at least most of the advice isn't surprising. But there's one thing that at this point makes me feel like I'm taking crazy pills because it seems like I disagree with just about everyone. I think unit tests should be first priority, ahead of integration tests and selenium-ish tests (whether you call them frontend tests, functional tests, whatever) I have experience with all three approaches, but slowly moving from backend to full-stack and frontend, I'm struck by how many people - it almost seems a consensus at this point - argue that integration tests should be your first line of defense. I've long since given up the battle on this; I'm not directly on the frontend teams, so I can't seagull my way in there and contradict the lead. But I think the point that gets lost on people is that the value of unit tests isn't chiefly the output of running the test suite. It's that the process of writing good unit tests forces you to write well-structured code. Code that is well-structured enough that it actually minimizes your need for integration and selenium-ish tests. But I feel like in the industry (not just my employer), I'm surrounded by people who are comfortable with writing integration tests and calling it good, even with spaghetti-code implementations underneath. I've always suspected that the ideal testing setup would be mostly unit tests, integration tests only to fill those gaps, and selenium-ish tests (hopefully rare!) only to fill in the final remaining gaps. But I think the system dynamics are set up such that unit tests are rare (due to wanting to finish the Jira story when the feature starts to work), integration tests are frequent since you can happy-path across a large surface area, and large expensive suites of selenium-ish tests are written by a completely different department since you can write them without having to understand the codebase. It just seems like a recipe for poor overall quality and a lot of wasted work/time/money. But I'm clearly in the minority. Maybe I'm missing something basic that I've somehow never learned in my long career.

tibbon 4y ago
I strongly agree. Most of this felt fine, except about testing. I'm frequently tasked with writing reliable services, and I always 100% of the time start with tests. Are they perfect? Absolutely not, but I am able to write, test, maintain and iterate on at-scale critical services with great confidence; and 90% of that confidence can come from well structured tests. My code is generally also easy to refactor, understand, port to other languages, etc. Testing is such a critical part of that. I only test things "in staging" as a last check; and very rarely am I disappointed with the behavior (sometimes the speed or scale, but not behavior)
smrtinsert 4y ago
100% in agreement. The largest projects that I've successfully completed have had extensive unit test and test suites in general.
3pt14159 4y ago
What you're missing is that most people in web dev use a full featured framework like Django or Rails and do not need unit tests nearly as much as integration tests because very, very, very often most of the architectural decisions have been made for them and the framework linking it all together is where mismatches between expectations seep in. "Wups, I pluralized this thing that should have been singular in the routes / urls file" Also, when developing APIs for the front end it's pretty unlikely that I need to test Rabbit.permanent_url on its own and much more likely I need to test things like listing all the rabbits for a given rabbit farm that are candidates for sale in the local meat market. Where exactly should this test go? The framework handles all the magic SQL generation and the frontend folks really only care about input -> output. If you're building everything from the ground up, then of course there will be way more unit tests, but with an established framework you don't have to test everything. You trust the framework to get some things right for you.
phtrivier 4y ago
Your ideal is what is commonly described as the "testing pyramid". It's very well suited to "single artifact" code (a lib, an app with an easy way to drive interaction through a gui, etc...) It tends to turn into a "testing sandglass" with time (because most of the value of integration tests is also derived from gui/e2e tests.) Depending on the app, it might makes sense. Kent C Dodds (author of the poorly named "testing library") is embracing this sandglass shape by calling it the "testing trophy". Honestly, to me the hardest deterrent to testing are : * If it's not done from day 1, you end up with that one hard to test piece of code that makes every other piece of code hard to test * Few people enjoy writing tests (I do, but I reckon I'm part of a minority.) Do what helps you the most !!

AtlasBarfed 4y ago

cpach 4y ago

Good article. Seems like solid and useful advice.

nuker 4y ago

> Kubernetes gives infra teams scalability superpowers Gives nightmares too. Im sticking with Fargate, full, native IaaC support, feature parity with k8s. Or plain EC2s in ASG.

dividedbyzero 4y ago
Fargate has full feature parity with Kubernetes? Is that true? Also, what exactly is "full, native IaaC support" in this case, compared to e.g. EKS?
- nuker 4y ago
  I meant ECS with fargate launch-type, of course. IaaC support is you can use single CFN template to deploy autoscaled service, with load balancing and run time parameters. No kubectl or helm needed.
  - dividedbyzero 4y ago
    Question stands for ECS too, does it actually have full feature parity by now? Last time I used it it was still more limited.
    - nuker 4y ago
      Give me one, not too esoteric plz
    - crgwbr 4y ago
      I’m sure it doesn’t have “full” feature parity, but do you really need 100% of k8s’ features? ECS is a lot simpler to work with and has plenty of features for most systems.
      - jnsaff2 4y ago
        So like Full Self Driving?
4y ago

simplecto 4y ago

Great insights! I came to some similar conclusions as well. I really like the one about failing on missing config parameters. I used to set default params all the time (thinking I was clever). Another big win in the last few years has been feature flagging. I use it extensively to tune performance when in production. I can control batch sizes, delays, retries, and API key rotation. bookmarked!

falafelite 4y ago

Big +1 to “ Never give up on local testing”. The current code base I’m dealing with has many tests that require interaction with a dev environment on the cloud, and occasionally these tests fail due to a timeout or some other thing not related to actually testing the code (and instead reveals that I forgot to refresh my MFA). Additionally the dependency thing can be huge; we ran into a weird bug for installing a particular dependency on our CI system so our test there keeps failing, but being able to run it locally let’s us know that the changes did not break our actual code tests.

anonymousDan 4y ago
How does this work in a serverless world? As far as I can tell local dev for most serverless environments that people actually care about is a joke.
- dijit 4y ago
  Usually there is some sort of local development framework for serveless functions. Most seem to use docker, here’s one for Google Cloud functions: https://cloud.google.com/functions/docs/running/overview https://cloud.google.com/functions/docs/running/overview If you use something like “cloud run” then it’s containers anyway.
- koblas 4y ago
  For golang, but would work in any language. What's really worked is the idea that your main loads configuration and injects all of the configured interfaces into your handler. You then can build all of the testing on mocks/stubs to test the behavior. If you access a database you access in through the interface which can mimic the appropriate behavior for your code in test vs production. If you need to you can do local integration testing of the db access layer.
- Hikikomori 4y ago
  With AWS SAM you can invoke functions locally, run api gateway and use step functions. There's also a 3rd party project to run various AWS services locally, can't remember the project name. There's also SAM sync that syncs changes you make quickly, so if you have your own dev copy in AWS you can quickly test changes.

arianvanp 4y ago

> Use Helm. Or some other tool for managing Kubernetes manifests, I’m not picky – the important thing is that you ~never directly use kubectl apply, edit, or delete. The resource lifecycle needs to be findable in version control. I'd say `kubectl apply` fits better with a Git oriented flow than the imperative `helm install` flow. Nothing about helm guarantees or forces you to use version control. Actually the opposite. If `helm install` fails. You need to `helm delete` first. It has no way of declaratively managing updates at all. Have a CI job call `kubectl apply -k ./deploy --prune` on every commit is way nicer than helm in my opinion.

cassianoleal 4y ago
Helm also hides complexity in the templates. There's a lot of valid complaints floating around YAML but what's worse than dealing with YAML is dealing with templated YAML. kubectl brings everything to the surface. Helm is an unnecessary abstraction.
- tecleandor 4y ago
  I think Helm can be nice to template mildly complex stuff you cannot reach simply with Kustomize, but it's true there are lot of things you can't see until you apply them, even when `helm lint` doesn't fail. If you need this templating, a think you can do is use helm for that but not for install/deployment: render the charts with `helm template` and then apply them with `kubectl apply` and/or verify the changes with `kubectl diff`. IIRC Argo CD can show you the differences between the running helm chart and the new version you want to apply if you haven't enabled automated syncing, but I only use Argo CD with Kustomize, so I can't say for sure right now. edit: by reading the article again I think the spirit of that point is not 'use just helm' but 'don't apply stuff directly'. Long term, if there's more than one person working in Kubernetes, that's prone to failure and configuration drift. Doing the templating in whatever language or system that fills your needs and then using something like Argo CD or Octopus to apply them can remove most of those problems.
  - pickledish 4y ago

throwaway012282 4y ago

Sorry for the throwaway. > I’ve been doing this “reliability” stuff for a little while now (~5 years), at companies ranging from about 20 developers to over 2,000 I've been doing it for 20+ years including running critical services in FAANGs. > Use Docker > Use Kubernetes. Hard pass. There's a reason why most FAANGs developed their own packages management and deployment system and keep using them. They are simpler, less bloated, easier to debug. > Deploy everything all the time On your local testbed, maybe, unless you already know a commit is broken. On production, absolutely not.

adamckay 4y ago
> There's a reason why most FAANGs developed their own packages management and deployment system and keep using them. They are simpler, less bloated, easier to debug. And for every other company that isn't one of those 5, rolling your own deployment system won't be simpler, leaner or easier to debug. Nor will it, in all likelihood, be as stable, feature rich, observable, etc.
greymalik 4y ago
> There's a reason why most FAANGs developed their own packages management and deployment system and keep using them. They are simpler, less bloated, easier to debug. FAANG-appropriate solutions aren’t always appropriate for the rest of the world.
evercast 4y ago
> There's a reason why most FAANGs developed their own packages management and deployment system and keep using them. They are simpler, less bloated, easier to debug. Most FAANGs developed their computing infrastructure before Kubernetes gained popularity. After all the investment into building it and fine tuning for their software services (like ads), they probably have a lot more reasons to not migrate to something different. Not just because their systems are "simpler, less bloated, easier to debug".
- nine_k 4y ago
  Interestingly, it's G in FAANG who came up with K8s in the first place.

ZephyrBlu 4y ago

> Enable limited “instant” config rollouts Hmm, I think "config" is too vague here. I've seen numerous incidents caused by NGINX config changes that would have been far worse if we had instant 100% rollouts, but I'm not sure this is the kind of config change the author is referring to. > Run 3 of everything What is a "thing" in this example? Is it an instance of a service? A database? Something else? If you're using Kubernetes and have autoscaling, what does "3 of everything" even mean? It sounds like the author is basically talking about databases..?

pronik 4y ago
> What is a "thing" in this example? Is it an instance of a service? A database? Something else? No, you just run three of everything. While only databases and similarly distributed data storages really need exactly three (or sometimes, just an odd number larger than two) for quorum and network partition detection, there is a good possibility that you'd have three availability zones because of your databases. And since you already have three availability zones, it's only logical to put every service in every zone at least once, so running "three of everything" becomes a rule of the thumb, easy to remember, easy to implement. > If you're using Kubernetes and have autoscaling, what does "3 of everything" even mean? It sounds like the author is basically talking about databases..? If you use autoscaling, you still need to start somewhere. And this somewhere is "three of everything".
t3rabytes 4y ago
I think they’re getting at something closer to feature flags?

rytis 4y ago

> Extremely strict RPC settings. I’m talking zero retries (or MAYBE one) [...] I disagree. If we're talking about distributed systems, then one thing is guaranteed - network is not going to be reliable. And if we have 10's or 100's of services, this policy means that at the smallest blip the whole thing collapses like a house of cards. If the concern is "it's hard to troubleshoot", well then perhaps implement better logging ("connectivity to service X has been unreliable with X% failure rate over the last Xhrs" instead of "connection terminated", or no logging at all).

outworlder 4y ago
> Extremely strict RPC settings. I’m talking zero retries (or MAYBE one) [...] > I disagree I too disagree with that too, pretty strongly. The network is unreliable. Technology has become pretty good, to the point that we have become spoiled and take the reliability for granted. But it is at its very core an unreliable medium. Applications should expect that and survive accordingly. Now, before packets go out to the network, or after they have made their way inside your network, they are in an environment that you have a degree of control. Errors and retries should be monitored. If they increase and remain elevated, they should be investigated. But guess what, if your services are resilient, you should have some time to investigate before things start breaking down - as they would if you treated everything as if it was _localhost_ Case in point: our app started breaking in horrendous ways once we deployed it in China and it tried to cross over the Great Firewall. Most issues would have been survivable if they just backed off and retried. The GFW doesn't usually like the first attempt to/from a new address, but it will usually allow the traffic after a while. We had to pay a company a not insignificant chunk of cash to get better connectivity (that would have been fine as an improvement, but not to make the system work at all). Retries are at the core of everything we do. TCP has retries (if the author followed their own advice, they should switch to UDP!). Kubernetes has a whole bunch of retries before reaching error states: CrashLoopBackOff, ImagePullBackOff. Your ethernet card has retries. But it also tracks errors. Track errors. And have retries. For as long as it makes sense to retry for your use-case. > well then perhaps implement better logging Not logging. Metrics! Logs are good for troubleshooting. For signaling issues, metrics are better. Errors can be a simple counter that you increment. Have Prometheus or similar scrape that and throw alerts as necessary.

bigbluedots 4y ago

Lots of YAML, some duct tape.

keyle 4y ago
Hah. That made me smile. But that's my issue with most "kids" these days. The code is in rust, takes dark wizard incantations to understand, a treasure map to even compile, ... And then the rest is built on yolo-yaml reproducible like that time I did pancakes that didn't suck.

kbrannigan 4y ago

> Use docker Heh nah, I’ve started learning ansible and it’s so much easier for me to understand. For some reason docker is a little to complex for my purposes. Ansible just runs ssh commands , no extra layer of abstraction I found that the fewest abstractions the better, because at some point you’ll have to debug both the abstraction and the bottom layer.

pronik 4y ago
You'll find yourself in the world of pain the next time you need to remove something. Over time, your playbook will be full with stuff that shouldn't be on the machines. At some point you'll consider re-installing the machines from scratch from time to time. At some later point you'd consider re-installing machines every time your Ansible playbook changes. And at that point you'll have reimplemented containers. Poorly.
nine_k 4y ago
It's very easy to understand. With Ansible, you first start a machine, then build a configuration on it. With Docker, you first build the configuration, then start the machine (not a real VM, but an isolated enough slice). The upside of a Docker image is that you build it once, and then start instantly, no need to wait for Ansible to run through steps. The downside, but also an upside, of Docker is that you can't make persistent changes inside it while it's running, and usually you avoid such changes. You always can restart it from a known state. Of course Ansible can handle more aspects than just application software configuration, basically any aspect at all. This is its upside, but also a downside: you can make an unintended system change along with some innocent-looking operation by mistake, and nothing will stop you, there's no separation of concerns.
- kbrannigan 4y ago
  Thanks for that detailed explanation. So it’s possible to pair Ansible with docker? Like only the app on docker but all the runtime, Apache, dependencies configured using Ansible ?
  - 4y ago

KaiserPro 4y ago

I agree with most of these points. They are a good starting point. I'd probably steer clear of kubernetes until you actually really really need it. ECS is good enough for most things, and a fucktonne more simple to look after. but, where I hard diverge is the lack of metrics. Everything should be generating metrics. Logs are great, but they are crap for giving you near realtime trends. Everytime your container gets a connection: increment a counter. Every message processed, counter. every message failed, counter. Every KB of data sent out, counter. Every service call, counter by service. metric all the things, in a sensible, mostly automated way, with a decent schema. Then you can combine all those metrics into a dashboard that shows your system performing against business goals.

kasey_junk 4y ago

This list has things I agree with and things I don’t. You could probably make another similar list that didn’t overlap at all. That’s because the real way to build software that’s reliable has very little to do with a list of features/tools it has to do with designing for the sad paths at the same time as the happy paths. For each failure modality the app should have a conscious decision about what the app will do when the bad thing happens. This can be everything from “this will cause an outage” to “this will route to an alternate system”. What’s important is figuring out up front what those decisions mean for your data, your operations, your business and your users. Without that all the structured logs and version controlled configs in the world are just window dressing.

dimgl 4y ago

Good article, until: > Use Kubernetes. Dear God no lol If you're running a startup that's starting to scale dramatically, I think maybe this is valid advice. But nowadays you can keep things dead simple and still be able to accomplish most business use-cases.

pgwhalen 4y ago
Time and time again, commenters suggest that the reason to use Kubernetes is to enable easy scaling, but this is not the case in my organization. Kubernetes is valuable because it is a standardized and flexible deployment model. Easy scalability is a nice side effect of that.
jstx1 4y ago
Doing simple things in Kubernetes isn't that much more complicated than doing simple things outside of Kubernetes.
- mikedelago 4y ago
  I agree - I think the hardest hump for developers is to learn Kubernetes first. After that, it's really not bad for most workloads.
w0m 4y ago
Kube has hit enough market/mind saturation that using it for more trivial use case isn't isn't necessarily a horrendous idea anymore. Just having a Kube available I can run a quick kron on is valuable.
- rednerrus 4y ago
  How much overhead does it require to feed and water k8s?
  - w0m 4y ago
    After initial setup, a managed GKE/AKS is pretty hands-off.

chrisseaton 4y ago

I saw a great talk at EuRuKo last week about treating your CI like an SRE would and using metrics and tripwires and things, by Mel Kaulfuss.

Havoc 4y ago

>The highest-value-per-time-spent kind of test is just pushing your change to staging (or better, prod!) and showing it does what you wanted and doesn’t break everything Ah yes, the time honoured testing in prod strategy

411111111111111 4y ago
"Everyone's got a test system, some also have a stable test system"
pickledish 4y ago
Well, "this but unironically", really! One of the main tenets of how I hold SRE is that you should be confident in sending pretty much _anything_ to prod, because you trust the system you've built around it to catch problems before they lead to an outage. And in a world like that, with good shadow/canary deployments etc -- there's no better way to validate your change, right?
- harshaw 4y ago
  unless you run a service that stores data or does something that can't fail in production.

mkl95 4y ago

I have worked at some places where SRE guys were a joy to work with. But at my current place they take a cowboy approach and make my life harder. So I'm currently not interested in building software like an SRE.

zaphar 4y ago
SRE's should not be cowboys. They should be the least cowboy engineers in your company. Something is very wrong there.
- mkl95 4y ago
  That used be my perception as well. I have now had back to back jobs where there is some maverick in the SRE team. A single guy who modifies some configuration at 2AM for services your team supposedly owns can ruin your week. Some people think lack of bureaucracy is a get out of jail card, and they are often right.
  - zaphar 4y ago
    As someone who has managed a team of SREs that kind of behavior would land someone on a PIP if done repeatedly. Sounds like you had a run of bad luck in teams.
    - mkl95 4y ago
      A PIP would be nice tbh. At my previous place they PIP'd one of those guys, but they ended up moving him to my team, where we taught him a bit about processes.

Bayart 4y ago

> Use Helm. Or some other tool for managing Kubernetes manifests, I’m not picky – the important thing is that you ~never directly use kubectl apply, edit, or delete. The resource lifecycle needs to be findable in version control. I have to partly disagree with that one. I find tools like Helm to obscure things that should be readily visible. My favoured method is to keep manifests in full (which you can source from `helm template`!) as pure yaml files and version that. If possible freeze versions and go through regular patch cycles to review updates. That you apply them through `kubectl apply` or through Argo is irrelevant. I treat the repo as the state and the running cluster as stateless. If it's borked, just redeploy. I don't see it as useful to care too much about the in-cluster resource life-cycle. But I completely agree that resources need to be version-controlled.

thisiswrongggg 4y ago
Agree 100%. Plus Helm runs against a basic tenet of microservices (the usual architecture for the deployed apps in k8s these days). People use to bundle services together when using Helm and the like which, in time, couples services together.
- dividedbyzero 4y ago
  But don't you actually want a certain amount of coupling in the operations part? After all you need to ensure they run in a place and in a manner that allows them to find and talk to each other and usually in a combination determined by the goal of a specific deployment, i.e. sometimes you might not need a rail-cargo-service because the customer only ships by truck, etc. Then scaling/autoscaling (if any) needs to be compatible, versions need to be within a certain range, any central data store must be coordinated as well, not to speak of service meshes, chaos experiments and the like. It's a good thing to develop services with minimal coupling, but that stage has different risks and goals from devops/ops, at least in my experience on both sides of the dev/ops transition zone.
  - mikedelago 4y ago
    > But don't you actually want a certain amount of coupling in the operations part? In my opinion, for sure. There's a balance between "too coupled" and "too de-coupled" that should be stricken rather than too far on either side. It's good to say that this is also contextual; some projects may be fine with either more or less coupling than others, and that's OK.

jasonhansel 4y ago

> Never give up on local testing. It keeps dev cycle time much shorter than needing to rely on (and fiddle with) CI or remote workspaces. Excellent advice for improving productivity and testability. I wish more teams would be sure to follow this by not adopting technologies that are impossible to test locally.

pluc 4y ago

> How to Build Software Like an Site Reliability Engineer Acronyms don't excuse poor grammar.

pxx 4y ago
The "S" in "SRE" is pronounced "ess" which would take an "an" as the article. The title is grammatically correct. You would also say "an FBI agent" or sending "an SOS message."
jasonhansel 4y ago
No, "an SRE" is correct here; it's actually "a SRE" that would be bad grammar. Whether you use "a" or "an" depends on whether the next word starts with a vowel sound, not with a vowel letter; since the letter "S" is pronounced "ess" in an acronym, you would write "an SRE," but "a Site Reliability Engineer."
unfunco 4y ago
It's an initialism, not an acronym. And _an_ SRE is correct.
- morelisp 4y ago
  I have bad news for you about something I heard a recruiter say a couple weeks ago!

gocartStatue 4y ago

I disagree so much with discarding unit tests, if you’re not dead sure of the base units, then you have a combinatoric explosion down the road. How do you even design code without unit tests :D

bitpow 4y ago
Discarding unit tests is such terrible advice. The author clearly never discovered regression bugs at build time, or developed using a dynamically typed language. Also, I would have expected to see "Use only strongly typed languages" as one of the bullet items. I get the sense that the author doesn't actually have a ton of enterprise experience.

pickledish 4y ago

Ah hello, author here! I wrote this up last week and posted it to Reddit (didn’t expect to find it on HN!) and I should probably say the same thing I did over there — this is a list of things I have learned are important, based on my own experiences. I’m sure others have had different experiences leading them to different opinions (sometimes even opposite to mine!), which is good, and I’m enjoying reading all the viewpoints here :)

jasonhansel 4y ago

> Structured logs are non-negotiable I disagree with this. Logs should only be used by human operators; if you want something machine-readable, you should expose metrics via some other channel or record errors in a dedicated service. Sure, some tools will let you read structured logs quite cleanly, but it will never be as nicely formatted as something designed, first and foremost, for humans. Of course, you may need to (say) tack on a request ID to avoid logs from different threads/goroutines/etc from getting interleaved, but that doesn't require everything to be in JSON.

outworlder 4y ago
> Logs should only be used by human operators I suspect that you will find a lot of resistance to the idea. At least I have. Machines should be consuming metrics, not logs, unless there aren't any alternatives. That statement never seems to be controversial, but people... just don't do it, for whatever reason. Also, the author seems to want to do a poor man's version of distributed tracing.
Matthias247 4y ago
There's some logs which you want machine readable that are not necessarily metrics. Let's say you have 10 services in serial, and you want to get precise information on how that request was performed - because your customer reported it performed super bad. To do that you might need to get logs from all those 10 services and align them. If they all are human readable and you can filter by a common request-id, that's easy to do. And you can even have tooling on top of it that automatically determines what might have happened by looking at other fields of the log entries. Metrics are mostly just pushed to a metrics system, and that moment you don't care about them or the relation to an end-to-end request anymore. So while all services might have emitted a metric for the request, looking at the overall metrics won't tell you if outliers in the services refer to the same end-to-end request or different ones.
gravypod 4y ago
Structured logs can still be human readable. Ex: [2022-01... INFO] trace=abcd123 <msg> There is a structure which can be parsed.

omginternets 4y ago

A colleague of mine once shared his lay sociological theory about dev vs ops, and if taken for what it is -- an essentialization -- it's an interesting perspective. The idea is that ops people have inherited a blue-collar culture, whereas devs have inherited an office-worker/academic culture. Ops people conceive of their work as fundamentally operational: progress is measured in terms of actions taken, and while automation is greatly valued, there is nothing inherently "messy" with one-off fixes; the objective is to get things working now. The pathological case for this mindset is that of constantly being on the back-foot, responding to incidents with one-off fixes without recognizing that many of them share a common cause that could be addressed. Dev people conceive of their work as fundamentally intellectual: progress is attained when the problem is correctly conceived, at which point the solution follows naturally. While writing code is greatly valued, most effort should be spent understanding the problem; the objective is to solve it correctly, once and for all. The pathological case for this mindset is that of over-engineering by an ivory-tower idealist, disconnected from the messiness of real-world praxis. In nearly all orgs I've seen, the proportion of first-generation college grads is greater in ops teams than in dev teams. So too is the proportion of people who come from blue-collar families, or are mechanically inclined (ex: look around and see who tinkers with cars). Likewise, the proportion of people holding graduate degrees is greater in the dev crowd (ex: look around and see who's into math). It should hopefully be clear that neither is superior to the other. The point is rather that the divide between dev and ops is partly sociological, which means it is largely based on values. Ops will tend to over-value "honest hard work" and dev will tend to over-value "clear articulate thought". There is also some latent, historical tension between these sociological groups, which has a funny way of masquerading as a technical problem. It is helpful to view arguments about "just ship it" vs "design it the right way" through this lens. Far from being a cute "just so" story, it's been my experience that this dynamic is very important for two reasons: (1) it's harder than you might expect to foster the sense of common destiny required for real "devops"-style collaboration, and (2) each side of the dev-ops divide has a lot to gain from learning when the other side's mindset is helpful, and how to cultivate it in themselves.

slyall 4y ago
Interesting idea but perhaps you are confusing attitudes to code with attitudes in general. eg Ops tends to be very attached to getting the infrastructure correct whereas developers care about that less. Developers will write an application that handles all sorts of data edge cases but doesn't react well to network errors.

molodec 4y ago

Great, but it is just a way to build reliable software. Many companies run reliable software without using half of the suggested points.

molodec 4y ago

> Avoid state like the plague This really depends on the problem domain. Application state is fine in many areas, and in the case the state is non trivial managing it in the application is a lot easier that pushing to databases.

dainiusse 4y ago

The title should be "how to build software for an SRE"

pronik 4y ago
I'd argue that without SRE you won't be delivering any software to the customer.

0xbadcafebee 4y ago

"Extremely strict RPC settings. I’m talking zero retries (or MAYBE one), and a timeout like 3x the p99. We are striving for predictability here, and sprinkling retries or long timeouts as a quick fix for a flaky downstream service will turn into a week-long investigation and a migraine a year from now. Fix the flaky service!" You want the 80%, right? Might be the network is flaky, might be a service, might be a load balancer. Use timeouts and retries to ignore the flakiness until it eats into your error budgets. "Never give up on local testing. It keeps dev cycle time much shorter than needing to rely on (and fiddle with) CI or remote workspaces. Containerizing the local test environment can make it easier to keep dependencies straight and consistent across machines." Until debugging the snowflakey local testing, or the difference between local dev and CI, ends up eating more time than just using a local<->remote sync tool. There's like a thousand tools now to combine local dev and remote envs, and they will save your organization a lot of time and hassle. "Your git repository is your point-in-time-recoverable source of truth." I call this "suggestion-of-truth". The truth is you never know what state things are actually in, but you hope it's like what's in Git. And there's never enough information kept in Git, like all the versions of all your services used at each deploy, or the state of the entire system, or filesystem snapshots, so recovery often fails. "For infra changes, make plans extremely obvious. This could mean “post the Terraform plan as a comment on the pull request”," Sadly this isn't enough as they might not be using a plan file, and when they apply something unexpected happens. Even if you have a plan file, there's a million other things you should be "making obvious" at deploy time, but people never do. Hard problem to solve. "Use Kubernetes. Assuming you have more than one service and more than one instance, you either need or will need stuff like service discovery, autoscaling, and deployment versioning, and you do not want to manage all that by yourself. Kubernetes gives infra teams scalability superpowers." Yeah, if you pay for it, in money, in expertise, in person-hours. Those things don't just come out of the box. Autoscaling: not at all. Service discovery: you have to set up a DNS controller with an external nameserver, define a naming + namespace convention and stick to it, and use external service definitions for stuff not hosted in K8s. Deployment versioning: you have to define a deployment pattern that stores the versions of everything at deploy time and then link that to your K8s deploys. There's still weeks/months of setting shit up, and then dealing with the complexity of it all and the constant upgrading. Do literally everything you can to abstract away K8s setup + maintenance. Pay anyone to do it all for you. If you're unwilling to pay for it, don't use it. If you do it yourself, even with a managed control plane, it's like building a car from scratch. People on HN sometimes say "oh setting up / maintaining K8s was easy" and I think of the guy who built his own 2-story wooden deck saying it was easy. I wouldn't stand near it.

ChrisMarshallNY 4y ago

> Use Git. Use it for everything – infrastructure, configuration, code, dashboards, on-call rotations. Your git repository is your point-in-time-recoverable source of truth. I'd amend that to "Use configuration management." Git is wonderful, but it isn't a Swiss Army knife. There are some things that are better archived using systems like Perforce (many game developers use it, because Very Big Assets), or even simple incremental file system backups. Back in the "dark ages," the Japanese release managers at my company used to clone an entire computer disk drive onto an external disk, and store that disk in a special file cabinet, so you not only had the source, you also had the entire system that was used to build it. This has some obvious issues, but it worked for them, for years. Also -and I cannot stress this enough- TEST YOUR BACKUP RECOVERY. I have been in the position of needing to restore old files (either partially, or whole-hog), and found out, the hard way, that my backups were pooched.

ryanthedev 4y ago

I couldn’t agree more on these bullets. People forgot what it means to build 12 factor applications.

daitangio 4y ago

I agree on most of the bullet points. As a Software Architect (the SRE mirror role :) I am following https://12factor.net/ https://12factor.net/ and that guidelines simplified a lot my CI/CD pipelines. Helm + K8s is complex but you can easily enable to change some param and redeploy in a very fast and relable way. And yes, docker outperform ansible on a lot of use-cases in my humble opinion.

sobriquet9 4y ago

> Don’t waste time on code coverage Is it really that useless? I was about to start doing code coverage and now wonder if I should reconsider.

cgrealy 4y ago
Code coverage doesn't tell you much IMO. I'll take a system with 70% coverage that's focused on the really complex pieces over a system with 99% coverage that's mostly checking that "person.Name" returns the name. Your engineers should know where be dragons in the code and unit testing appropriately.
jjice 4y ago
What do you mean by "doing code coverage"? Tracking it? Doesn't take too long to set up in some situations. Increasing it? Depends on if that's trying to get to 100% or just to have a metric to see where tests are needed. It's usually not productive to enforce it for long ongoing development projects (aside from maybe "stay about 70%" or something). It can be a neat metric, but I don't think it's very productive. Testing comes in the form over very unbalanced importance (see "Pareto distribution"). Start by testing the most important business critical functionality and then move out from there. 20% of your tests will cover 80% of your actual important use cases. The trade off would be easier to manager if tests didn't require maintenance, but to get the best bang for your buck, focus on the most critical stuff first. Code coverage is by no means a necessity.
- sobriquet9 4y ago
  I was thinking about including the necessary instrumentation to check code coverage during regression tests. Something like coverage.py. The intent is not to get a specific number, but to understand which parts of the code might not be exercised enough by existing test cases.
UncleSam 4y ago
> My goal here isn’t “what is 100% the most reliability-oriented way we can build things”, it’s more like “what is the 80% of reliability we can get for 20% of the effort while still enabling devs to go fast” The author also points out that these bullet points will not apply to every unique situation. If your goal is to allow devs to go fast at the cost of more hardened code, then yes, skipping code coverage is a major way to code faster. There are definitely times when it is appropriate to skip writing tests when the primary goal is to go fast. It's difficult to answer your question without knowing your specific situation though.

jimcavel888 4y ago

ransom1538 4y ago

IMHO, How to build software like an SRE: 1. Let us know when something is broken. If a curl fails, log it. We want more logs not less. 2. Make the code configuration file free. Use ENV vars. If an ENV var is missing crash and say why. The environment will determine how the app acts. 3. Build process and Deploy process should be different. Github actions is great for the Build process, Jenkins is great for deployments.

bsnnkv 4y ago

As a counterpoint to part of this article, I just posted an article on HN in which an SRE turned solo dev ditched Kubernetes for Fly after 2 years and rising costs.[1] VMs are good enough for high-traffic sites like HN and Pinboard. I think they can be good enough for most solo devs as well. [1] https://news.ycombinator.com/item?id=33235042 https://news.ycombinator.com/item?id=33235042

woopwoop24 4y ago

helm is a bad solution to a lazy thought out problem. why would i template yaml from another yaml and without versioning and weird if else constructs just for deploying which clearly should be json? because we devops are obviously not able to abstract the problem. oh you switched from nginx to haproxy, lets change all the labels in my git comitted repos where my yaml lies. oh the author changed the way the helm chart works, lets do manual compare to see what changed. Oh k8s changed the api version again, lets change all the yaml. and helmfile on top of helm -> more abstraction in the wrong direction -> love it. i would much rather use something like starlark and program my deployment needs against the correct library version of the k8s api which my cluster has and get the final yaml for free instead of templating and forking and maintaining dead weight every damn version.