4 ms·

Automation is the serialization of understanding

173 points by zdw 5y ago

epgui 5y ago
I read the title, and I thought I knew what the main ideas of the article would be. And then I read the article, and I have no idea what this is supposed to be about. Anyone smarter than me mind fleshing out just a little bit of context?
- deleted 5y ago
  [deleted]
- Jtsummers 5y ago
  If you build out a layered system with reusable components you're better off. But if you forget about that and create a precious gem of a script for your Docker deployment that can't be used anywhere else, you may be losing out (if you need to do anything besides use Docker). If Docker is all you need, you're fine. If you need to deploy anywhere else, you end up with multiple precious gem scripts that duplicate effort and maybe miss elements (creating subtle differences and incompatibilities between the multiple deployments). Unix philosophy, DRY.
  - aesyondu 5y ago
    Thanks for the summary. I read the article and to me this is more about creating appropriate abstractions than automation.
trabant00 5y ago
I think I know what the article wanted to say but it did a really poor job of saying it. The discussion is interesting though. As we pile ever more abstractions on top of existing ones we forget what they are based on. Take young "devops" for example, there's plenty of them who don't understand what AWS has under the hood. A lot of them miss fundamental Linux knowledge and never touched virtualization tools directly. And it can bite them hard sometimes. On the other hand I think it's always been like this. I grew up doing a lot of Linux networking, virtualization, packaging, etc. But I don't know any underlying tech for that. I can't even read C code, don't know the Linux system calls. And of course there's an xkcd about this https://xkcd.com/435/ https://xkcd.com/435/
- thinkmassive 5y ago
  It's an excerpt from a podcast, not really an "article" so maybe that's why you're disappointed?
- lillecarl 5y ago
  I started my DevOps career last November, but having done Ops before. Since DevOps these days should just mean run everything in Kubernetes I don't see why you need to know all the things, just bolt controllers onto your K8s cluster to integrate the features you need with your cloud provider, get some simple CI/CD running and you're set to make your salary for awhile. Make sure to assist your developers by making it easy to do things right. Honestly the easiest career transition ever.
blowski 5y ago
To paraphrase the maxim, working automated systems evolve from working manual systems. But only some manual systems work. I start CI/CD by doing the whole process manually. For example, type the commands to build a Docker image, or spawn a VM, or obtain a secret. I encode all this in pseudo code, then put it in Bash (or Python). When a conditional branch appears (a different environment with different credentials), I treat it like any other code. Separate the bits that stay the same, and inject the bits that change. The problem with most CI/CD systems is that people tightly couple themselves to the tool without really understanding it - the point the article is making. They over-complicate the solution because the documentation encourages you to do that. When they want to customise, debug, or even migrate away from it, it’s very difficult.
- codeflo 5y ago
  > Separate the bits that stay the same, and inject the bits that change. Basically, treat your build/deployment scripts like you would any other code. Strive to have a single source of truth for any fact that can change etc. That seems obvious, but in my experience was at first counterintuitive for many people who otherwise had no trouble applying these principles in application code. A lot of modern CI/CD principles boil down to “it’s just code”.
  - fennecfoxen 5y ago
    That includes the part where “wouldn’t it be great if we could run just this one piece of code by itself sometimes, instead of having it embedded in a big ball of mud with tons of shared state and invisible dependencies?”
- harperlee 5y ago
  Very related to the idea of “Do-nothing scripting”: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=Do%20nothing%20script&sort=byPopularity&type=story https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
- kqr 5y ago
  People vastly underestimate manual processes. A properly optimised, trimmed, practiced manual process can be executed much more cheaply than most people think. What's the point? Flexibility. Humans are amazing at adapting to circumstances that invalidate portions of the process. In a fast-changing world, this is critical. Really solid automation is notoriously expensive to set up, and about as expensive to change when the circumstances fundamentally change. When, not if. (You can usually still scrape by on an old automated process that's no longer relevant, but you're leaving oodles of value on the floor.) Why do people underestimate manual processes? They usually don't know how to optimise them (hint: it involves restructuring things you think of as outside the process in question) so they associate manual with inefficient and expensive. When you have a time-tested optimised manual process, it might not even make economic sense to automate it. But if it does, you don't have to do it all at once. You can start by automating the dumbest, simplest, most tedious, and most stable step of the process first. Then you can continue to introduce automated steps only for as long as it makes economic sense to do so. ---- Another common complaint about manual processes is that they invite mistakes. I have three things to say on that: - There are ways to mistake-proof manual processes. "Poka-yoke" is a good starting word for googling. - Automated processes make mistakes too. The difference is that an automated process makes mistakes more consistently. (Which may or may not be a good thing, depending on context.) - Manual processes can avoid mistakes that are about to happen. In fact, I'd argue this happens at about the same rate as mistakes are made.
moring 5y ago
While I generally agree with the article, IMHO there is more to automation than understanding, and I regularly encounter problems with automation that cannot be solved by just understanding things. Two examples just because I tripped over them recently, so I still remember: 1. In general, things that are easy for a human (especially a developer) but hard to formalize. The particular example is "fixing typescript imports" where WebStorm thinks the import should not have a filename extension but tsc thinks they should end in ".js" (yes, .js for a typescript import, that's a well-known TS quirk). As a human, I can easily spot these places, but writing a script that is guaranteed to recognize exactly the import lines that must be fixed is hard -- it's putting into formal rules, not understanding, that is the hard part here. 2. Tools that make an explicit distinction between being invoked by a human vs. being invoked by a script. Particular example: Google Cloud, user account vs. service account authentication.
koffiezet 5y ago
I think one key aspect is lost here. A thing automation does is compartmentalise, abstract and remove the need for in-depth understanding by/for everyone involved. It's the reason why container images are the latest form of packaging for certain types of applications: it streamlines the interfaces towards applications even more. Configuration is usually done with environment variables, or even abstracted further with something like helm charts to deploy on an even more abstracted platform like Kubernetes. Things like configuration, temporary and persistent/temporary storage, health checks, services/ports, ... are all presented with a unified interface.
infologics 5y ago
[dead]
jasfi 5y ago
There are some things which seems obvious candidates to me, such as trading. However I have been surprised at how frequently and intensely people fight this idea. However not everything automated is fought, why? My auto crypto trading system: https://tradecast.one https://tradecast.one

173 points by zdw 5y ago

I read the title, and I thought I knew what the main ideas of the article would be. And then I read the article, and I have no idea what this is supposed to be about. Anyone smarter than me mind fleshing out just a little bit of context?

deleted 5y ago
[deleted]
Jtsummers 5y ago
If you build out a layered system with reusable components you're better off. But if you forget about that and create a precious gem of a script for your Docker deployment that can't be used anywhere else, you may be losing out (if you need to do anything besides use Docker). If Docker is all you need, you're fine. If you need to deploy anywhere else, you end up with multiple precious gem scripts that duplicate effort and maybe miss elements (creating subtle differences and incompatibilities between the multiple deployments). Unix philosophy, DRY.
- aesyondu 5y ago
  Thanks for the summary. I read the article and to me this is more about creating appropriate abstractions than automation.

trabant00 5y ago

I think I know what the article wanted to say but it did a really poor job of saying it. The discussion is interesting though. As we pile ever more abstractions on top of existing ones we forget what they are based on. Take young "devops" for example, there's plenty of them who don't understand what AWS has under the hood. A lot of them miss fundamental Linux knowledge and never touched virtualization tools directly. And it can bite them hard sometimes. On the other hand I think it's always been like this. I grew up doing a lot of Linux networking, virtualization, packaging, etc. But I don't know any underlying tech for that. I can't even read C code, don't know the Linux system calls. And of course there's an xkcd about this https://xkcd.com/435/ https://xkcd.com/435/

thinkmassive 5y ago
It's an excerpt from a podcast, not really an "article" so maybe that's why you're disappointed?
lillecarl 5y ago
I started my DevOps career last November, but having done Ops before. Since DevOps these days should just mean run everything in Kubernetes I don't see why you need to know all the things, just bolt controllers onto your K8s cluster to integrate the features you need with your cloud provider, get some simple CI/CD running and you're set to make your salary for awhile. Make sure to assist your developers by making it easy to do things right. Honestly the easiest career transition ever.

blowski 5y ago

To paraphrase the maxim, working automated systems evolve from working manual systems. But only some manual systems work. I start CI/CD by doing the whole process manually. For example, type the commands to build a Docker image, or spawn a VM, or obtain a secret. I encode all this in pseudo code, then put it in Bash (or Python). When a conditional branch appears (a different environment with different credentials), I treat it like any other code. Separate the bits that stay the same, and inject the bits that change. The problem with most CI/CD systems is that people tightly couple themselves to the tool without really understanding it - the point the article is making. They over-complicate the solution because the documentation encourages you to do that. When they want to customise, debug, or even migrate away from it, it’s very difficult.

codeflo 5y ago
> Separate the bits that stay the same, and inject the bits that change. Basically, treat your build/deployment scripts like you would any other code. Strive to have a single source of truth for any fact that can change etc. That seems obvious, but in my experience was at first counterintuitive for many people who otherwise had no trouble applying these principles in application code. A lot of modern CI/CD principles boil down to “it’s just code”.
- fennecfoxen 5y ago
  That includes the part where “wouldn’t it be great if we could run just this one piece of code by itself sometimes, instead of having it embedded in a big ball of mud with tons of shared state and invisible dependencies?”
harperlee 5y ago
Very related to the idea of “Do-nothing scripting”: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=Do%20nothing%20script&sort=byPopularity&type=story https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
kqr 5y ago
People vastly underestimate manual processes. A properly optimised, trimmed, practiced manual process can be executed much more cheaply than most people think. What's the point? Flexibility. Humans are amazing at adapting to circumstances that invalidate portions of the process. In a fast-changing world, this is critical. Really solid automation is notoriously expensive to set up, and about as expensive to change when the circumstances fundamentally change. When, not if. (You can usually still scrape by on an old automated process that's no longer relevant, but you're leaving oodles of value on the floor.) Why do people underestimate manual processes? They usually don't know how to optimise them (hint: it involves restructuring things you think of as outside the process in question) so they associate manual with inefficient and expensive. When you have a time-tested optimised manual process, it might not even make economic sense to automate it. But if it does, you don't have to do it all at once. You can start by automating the dumbest, simplest, most tedious, and most stable step of the process first. Then you can continue to introduce automated steps only for as long as it makes economic sense to do so. ---- Another common complaint about manual processes is that they invite mistakes. I have three things to say on that: - There are ways to mistake-proof manual processes. "Poka-yoke" is a good starting word for googling. - Automated processes make mistakes too. The difference is that an automated process makes mistakes more consistently. (Which may or may not be a good thing, depending on context.) - Manual processes can avoid mistakes that are about to happen. In fact, I'd argue this happens at about the same rate as mistakes are made.

moring 5y ago

While I generally agree with the article, IMHO there is more to automation than understanding, and I regularly encounter problems with automation that cannot be solved by just understanding things. Two examples just because I tripped over them recently, so I still remember: 1. In general, things that are easy for a human (especially a developer) but hard to formalize. The particular example is "fixing typescript imports" where WebStorm thinks the import should not have a filename extension but tsc thinks they should end in ".js" (yes, .js for a typescript import, that's a well-known TS quirk). As a human, I can easily spot these places, but writing a script that is guaranteed to recognize exactly the import lines that must be fixed is hard -- it's putting into formal rules, not understanding, that is the hard part here. 2. Tools that make an explicit distinction between being invoked by a human vs. being invoked by a script. Particular example: Google Cloud, user account vs. service account authentication.

koffiezet 5y ago

I think one key aspect is lost here. A thing automation does is compartmentalise, abstract and remove the need for in-depth understanding by/for everyone involved. It's the reason why container images are the latest form of packaging for certain types of applications: it streamlines the interfaces towards applications even more. Configuration is usually done with environment variables, or even abstracted further with something like helm charts to deploy on an even more abstracted platform like Kubernetes. Things like configuration, temporary and persistent/temporary storage, health checks, services/ports, ... are all presented with a unified interface.

infologics 5y ago

[dead]

jasfi 5y ago

There are some things which seems obvious candidates to me, such as trading. However I have been surprised at how frequently and intensely people fight this idea. However not everything automated is fought, why? My auto crypto trading system: https://tradecast.one https://tradecast.one