6 ms·

The Art of Automation

91 points by asicsp 6y ago

tuatoru 6y ago
Randall Munroe has done at least two cartoons about this. My own experience has led me to the point where if I'm thinking about automating something, I stop and ask "is there a way that this can be done away with completely?" The most reliable, easiest to maintain scripts are those with zero lines of code. They also perform best and consume least resources. With a bit of thought and/or flexibility in how things are achieved, it's surprising how often this question produces a good result.
- codegladiator 6y ago
  https://xkcd.com/1319/ https://xkcd.com/1319/ https://xkcd.com/1205/ https://xkcd.com/1205/
- jka 6y ago
  Your comment predicts the next few waves of no/low-code automation, I think. Early phases will involve the introduction of no/low-code automations that integrate existing APIs, services, and systems within businesses for the purposes of automating existing processes. That's good for the APIs, systems and automation platforms because they become essential and encumbent components of operating those businesses. The later, and I suspect much longer, phases will involve identification of business processes that aren't really needed and that can be removed by thinking about the ways that organizations function in entirely new ways, or that can be streamlined. By that point some of the integrations may have become so baked-in that they'll require significant maintenance to update. And as we see with some cloud-based IoT devices, it's possible for service providers to encounter outages or cease operating, in which case internal businesses processes may halt. All this is similar to the gradual automation and improvement of physical assembly lines; it's a case of applying the same principles to organizations. And yep, the cheapest, most efficient and reliable component is one that doesn't have to exist.
  - ethbr0 6y ago
    So, there's something of a story behind this. To simplify things, the early phase you mentioned is simply computing, and has been proceeding since the 1960s. The latter phase is called business process management (aka BPM) [1] and has been a thing since ~1980(?). In promise, BPM says "Encode all your business processes and their relationships into our tool, and we'll optimize and automate them." In practice, BPM is doomed to failure because (1) it's invariably implemented in an organization by management consultant types who care more about billable hours than actual solutions & (2) it's fundamentally an imperfect abstraction over your actual business processes, leading to leaks / hacks in implementation, leading to yet another system that further slows down the business. SAP is illustrative here, with the adage that smart businesses change their business to fit SAP, rather than try to change SAP to fit their business. [1] https://en.m.wikipedia.org/wiki/Business_process_management https://en.m.wikipedia.org/wiki/Business_process_management
- tyingq 6y ago
  This certainly holds true for a lot of approval workflows. I've been in big companies where my ~$300 Visio purchase flowed to a VP for approval. They could have easily done an auto-approve based on job title. It's not like the VP, who had hundreds of people on his team, would know if I needed it or not.
  - jrott 6y ago
    Approval workflows have so many problems. I've seen people that have to make tons of approvals but it's not there real job just approve everything that comes through since they don't want to break things. Almost completely destroyed the security point of making people review stuff for access control. It does fix things in a pure compliance sense so it'll continue.
- mathattack 6y ago
  This is an old idea in the quality movement. Reducing unnecessary steps improves both speed and quality.
- SimonDorfman 6y ago
  I love this one: https://xkcd.com/974/ https://xkcd.com/974/ It questions whether one should automate something that will only need to be done a few times.
wombatmobile 6y ago
> Time is one of the most valuable resources in the world. If there was something you could do to free more time for yourself, why wouldn’t you? You can automate a task and thereby cause the task to happen without your involvement, but that doesn't give you more time. It just automates a task. The amount of time you have is still that which happens between being born and dying.
- cinntaile 6y ago
  It is defined in the article what she means by it. time gained = (time doing task manually)-(time to automate task)
  - wombatmobile 6y ago
    What you are describing is experiential change. Instead of feeling beholden to a task, which in your equation is negative time, automation frees you to have opportunity for other activities that you might experience as positive time. Another way to view these experiences is that they all take place in the finite amount of time that you have here on Earth. That time doesn't change. Only your experience of it changes. You can control your experiences by cultivating your thoughts. For example, you might enjoy the time it takes you to design and build a system for automating a task. Contrast that experience with that of somebody who hates the idea of doing work to automate a task, and would rather just get the task over and done with, and never consider automation. Who is right, and who is wrong? You could calculate an answer by applying your maths, and that would be one answer, which might come out in your favour, or might not, depending on whether you could automate a task in less time than it would take to perform the task. But you already know that you're coming out ahead, even if it takes you 10 times longer to design and implement the automation than it does to perform the task manually, because you enjoy the process of automating tasks. That's because you've cultivated your thoughts.
    - samus 6y ago
      > But you already know that you're coming out ahead, even if it takes you 10 times longer to design and implement the automation than it does to perform the task manually, because you enjoy the process of automating tasks. That's because you've cultivated your thoughts. This is only true under the assumption that people automate stuff for fun. Businesses have a different incentive. They want automation to enable their pricey employees to take care of more tricky tasks, or to simply require fewer of them in the first place. A business won't give leave to their employees to automate processes just because they enjoy automating stuff.
rubin55 6y ago
A thing I usually find lacking in others’ “automate-all-the-things” scripts/apps/pipelines/etc is documentation. Without good documentation you only temporarily kept time-consuming chaos/manual-labor at bay at the cost of more complexity/entropy in the future. In my opinion, without docs, it eventually becomes a burden on (future) colleagues instead of a nicely consistent automated-away process. Don’t automate (just) for yourself, automate for others, where docs empower others to be autonomous and able to manage said automation without your head around.
- summitsummit 6y ago
  assuming in a capitalist society, this is not a choice that maximizes roi
- dgudkov 6y ago
  Good point. EasyMorph (I'm the founder) auto-generates human-readable HTML documentation on workflows in plain English. It's not perfect, but not bad either. https://easymorph.com https://easymorph.com
- DavidPeiffer 6y ago
  I agree documentation is super important! It's I recently started at a new company and have again been implementing the framework described in Manual Work is a Bug[1], summarized as: 1) Document the Steps 2) Create Automation Equivalents 3) Create Automation 4) Self-Service and Autonomous Systems When I started going through stuff with my coworker, I made a Word document with bullets and sub-bullets for all the steps in a process. When they unexpectedly needed to take a week leave of absence a month into the job, I was able to complete almost every routine part of the job. It took a ton of stress off of them! Across all my professional jobs, I find #1 to be the hardest. My process is to write down the high-level steps, then strictly follow the documentation next time I do the process, adding more and more detail every time. Once it's decently documented, I record how long each step takes, and add comments like "This needs to be better documented" or "This is the most tedious step, and would be pretty simple to automate". Timing each step lets me start and complete something between meetings, filling in gaps. Overall I've found the system to be really helpful. It makes transitioning job duties to new employees trivial and has saved me literally hundreds of hours over just a few years in the professional workforce. [1] https://queue.acm.org/detail.cfm?id=3197520 https://queue.acm.org/detail.cfm?id=3197520
monkeydust 6y ago
Been on an automation drive during lockdown. Highly recommend Automate the Boring Stuff with Python (https://amzn.to/3mVuKSx https://amzn.to/3mVuKSx) if you are interested in automation + coding (from beginner level). Have automated banking tasks and programmatically getting information from the internet (aka scrapping). Amazing what you can do with python libraries - beautiful soup, selenium...
- asicsp 6y ago
  ATBS is also free to read online [0] and the author has _automated_ to provide free coupons [1] (applicable during first week of a month) for the online course on udemy I'd also suggest realpython [2] for resources on web scraping [0] https://automatetheboringstuff.com/2e/ https://automatetheboringstuff.com/2e/ [1] https://inventwithpython.com/automateudemy https://inventwithpython.com/automateudemy [2] https://realpython.com/tutorials/web-scraping/ https://realpython.com/tutorials/web-scraping/
iamwil 6y ago
I started curating a bunch of automation stories, use-cases, and ideas at https://automationcookbook.io https://automationcookbook.io. It is usually more business process automation, rather than personal automation, though.
Dolios 6y ago
An improved timeGained equation. timeGained = (manualTimeSpent * n) - (timeToAutomate)
- Rexxar 6y ago
  This forget (at least) three factors: - Repetitive manual labor is prone to failure. If failure is expansive and automated task could be better just for the additional safety. - Once it's automatized the frequency at which the task is done can be increased. - Once something is automatized, other things become easier to automatized.
nimrody 6y ago
There was a post here a few weeks ago reminding us of the dangers of this approach. I don't have the link but the essence is here: After a process gets automated, people forget the details of how to do the work manually. Now, when the automation breaks (and it will break at some point. It takes a huge amount of work to make these things failsafe and scripting doesn't really encourage robustness) -- no one remembers the details and fixing the issue is more difficult and requires specialized knowledge that is no longer common in the organization.
- mxuribe 6y ago
  Totally agree, and i've seen this many times (and have had my teams "clean up" after others). however, what can help mitigate - maybe not fully prevent - in some cases are the following 2 approaches: 1. Build automation so that it is NOT such a blackbox. I even go so far as to build automation with scripting languages...that allows myself/my team to at least peek at the source code to see what is happening. 2. And, of course documentation. i know this is simplistic of me to state...but it is invaluable nonethelss. I should also clarify that i mean there should exist documentation both for users, or by extension those who benefit from the automation, as well as, for developers/sys admins/sys maintainers (say, within the code itself).
- gglitch 6y ago
  This sounds to me like not a failure of automation as such, but of process/knowledge management by the team that builds the automation. No process is permanently exempt from entropy.
walterbell 6y ago
Low-cost NFC tag stickers can be used to trigger iPhone shortcuts, with an inch of proximity. iOS shortcuts can make SSH connections to issue commands via shell scripts on local Linux servers.
- contingencies 6y ago
  That would be a closed source stack and device dependent solution. You could also use QR codes to trigger URI access which in turn ran arbitrary code on arbitrary devices. That would be an open source stack and device agnostic solution.
  - walterbell 6y ago
    NFC can trigger an interrupt to launch code/app, so proximity is sufficient. NFC exists on multiple platforms. QR codes would require launching an app to read the code, it would be faster for the user to launch the automation shortcut directly. There is also the issue of security. Which platform hardware+software would you entrust to pull the trigger on an automation sequence with physical side-effects?
    - contingencies 6y ago
      NFC costs more to set up. QR you can stick on anything easily, forward or share, etc. Trust? NIH. Something I wrote myself, only available on the local wireless network segment, probably.

91 points by asicsp 6y ago

Randall Munroe has done at least two cartoons about this. My own experience has led me to the point where if I'm thinking about automating something, I stop and ask "is there a way that this can be done away with completely?" The most reliable, easiest to maintain scripts are those with zero lines of code. They also perform best and consume least resources. With a bit of thought and/or flexibility in how things are achieved, it's surprising how often this question produces a good result.

codegladiator 6y ago
https://xkcd.com/1319/ https://xkcd.com/1319/ https://xkcd.com/1205/ https://xkcd.com/1205/
jka 6y ago
Your comment predicts the next few waves of no/low-code automation, I think. Early phases will involve the introduction of no/low-code automations that integrate existing APIs, services, and systems within businesses for the purposes of automating existing processes. That's good for the APIs, systems and automation platforms because they become essential and encumbent components of operating those businesses. The later, and I suspect much longer, phases will involve identification of business processes that aren't really needed and that can be removed by thinking about the ways that organizations function in entirely new ways, or that can be streamlined. By that point some of the integrations may have become so baked-in that they'll require significant maintenance to update. And as we see with some cloud-based IoT devices, it's possible for service providers to encounter outages or cease operating, in which case internal businesses processes may halt. All this is similar to the gradual automation and improvement of physical assembly lines; it's a case of applying the same principles to organizations. And yep, the cheapest, most efficient and reliable component is one that doesn't have to exist.
- ethbr0 6y ago
  So, there's something of a story behind this. To simplify things, the early phase you mentioned is simply computing, and has been proceeding since the 1960s. The latter phase is called business process management (aka BPM) [1] and has been a thing since ~1980(?). In promise, BPM says "Encode all your business processes and their relationships into our tool, and we'll optimize and automate them." In practice, BPM is doomed to failure because (1) it's invariably implemented in an organization by management consultant types who care more about billable hours than actual solutions & (2) it's fundamentally an imperfect abstraction over your actual business processes, leading to leaks / hacks in implementation, leading to yet another system that further slows down the business. SAP is illustrative here, with the adage that smart businesses change their business to fit SAP, rather than try to change SAP to fit their business. [1] https://en.m.wikipedia.org/wiki/Business_process_management https://en.m.wikipedia.org/wiki/Business_process_management
tyingq 6y ago
This certainly holds true for a lot of approval workflows. I've been in big companies where my ~$300 Visio purchase flowed to a VP for approval. They could have easily done an auto-approve based on job title. It's not like the VP, who had hundreds of people on his team, would know if I needed it or not.
- jrott 6y ago
  Approval workflows have so many problems. I've seen people that have to make tons of approvals but it's not there real job just approve everything that comes through since they don't want to break things. Almost completely destroyed the security point of making people review stuff for access control. It does fix things in a pure compliance sense so it'll continue.
mathattack 6y ago
This is an old idea in the quality movement. Reducing unnecessary steps improves both speed and quality.
SimonDorfman 6y ago
I love this one: https://xkcd.com/974/ https://xkcd.com/974/ It questions whether one should automate something that will only need to be done a few times.

wombatmobile 6y ago

> Time is one of the most valuable resources in the world. If there was something you could do to free more time for yourself, why wouldn’t you? You can automate a task and thereby cause the task to happen without your involvement, but that doesn't give you more time. It just automates a task. The amount of time you have is still that which happens between being born and dying.

cinntaile 6y ago
It is defined in the article what she means by it. time gained = (time doing task manually)-(time to automate task)
- wombatmobile 6y ago
  What you are describing is experiential change. Instead of feeling beholden to a task, which in your equation is negative time, automation frees you to have opportunity for other activities that you might experience as positive time. Another way to view these experiences is that they all take place in the finite amount of time that you have here on Earth. That time doesn't change. Only your experience of it changes. You can control your experiences by cultivating your thoughts. For example, you might enjoy the time it takes you to design and build a system for automating a task. Contrast that experience with that of somebody who hates the idea of doing work to automate a task, and would rather just get the task over and done with, and never consider automation. Who is right, and who is wrong? You could calculate an answer by applying your maths, and that would be one answer, which might come out in your favour, or might not, depending on whether you could automate a task in less time than it would take to perform the task. But you already know that you're coming out ahead, even if it takes you 10 times longer to design and implement the automation than it does to perform the task manually, because you enjoy the process of automating tasks. That's because you've cultivated your thoughts.
  - samus 6y ago
    > But you already know that you're coming out ahead, even if it takes you 10 times longer to design and implement the automation than it does to perform the task manually, because you enjoy the process of automating tasks. That's because you've cultivated your thoughts. This is only true under the assumption that people automate stuff for fun. Businesses have a different incentive. They want automation to enable their pricey employees to take care of more tricky tasks, or to simply require fewer of them in the first place. A business won't give leave to their employees to automate processes just because they enjoy automating stuff.

rubin55 6y ago

A thing I usually find lacking in others’ “automate-all-the-things” scripts/apps/pipelines/etc is documentation. Without good documentation you only temporarily kept time-consuming chaos/manual-labor at bay at the cost of more complexity/entropy in the future. In my opinion, without docs, it eventually becomes a burden on (future) colleagues instead of a nicely consistent automated-away process. Don’t automate (just) for yourself, automate for others, where docs empower others to be autonomous and able to manage said automation without your head around.

summitsummit 6y ago
assuming in a capitalist society, this is not a choice that maximizes roi
dgudkov 6y ago
Good point. EasyMorph (I'm the founder) auto-generates human-readable HTML documentation on workflows in plain English. It's not perfect, but not bad either. https://easymorph.com https://easymorph.com
DavidPeiffer 6y ago
I agree documentation is super important! It's I recently started at a new company and have again been implementing the framework described in Manual Work is a Bug[1], summarized as: 1) Document the Steps 2) Create Automation Equivalents 3) Create Automation 4) Self-Service and Autonomous Systems When I started going through stuff with my coworker, I made a Word document with bullets and sub-bullets for all the steps in a process. When they unexpectedly needed to take a week leave of absence a month into the job, I was able to complete almost every routine part of the job. It took a ton of stress off of them! Across all my professional jobs, I find #1 to be the hardest. My process is to write down the high-level steps, then strictly follow the documentation next time I do the process, adding more and more detail every time. Once it's decently documented, I record how long each step takes, and add comments like "This needs to be better documented" or "This is the most tedious step, and would be pretty simple to automate". Timing each step lets me start and complete something between meetings, filling in gaps. Overall I've found the system to be really helpful. It makes transitioning job duties to new employees trivial and has saved me literally hundreds of hours over just a few years in the professional workforce. [1] https://queue.acm.org/detail.cfm?id=3197520 https://queue.acm.org/detail.cfm?id=3197520

monkeydust 6y ago

Been on an automation drive during lockdown. Highly recommend Automate the Boring Stuff with Python (https://amzn.to/3mVuKSx https://amzn.to/3mVuKSx) if you are interested in automation + coding (from beginner level). Have automated banking tasks and programmatically getting information from the internet (aka scrapping). Amazing what you can do with python libraries - beautiful soup, selenium...

asicsp 6y ago
ATBS is also free to read online [0] and the author has _automated_ to provide free coupons [1] (applicable during first week of a month) for the online course on udemy I'd also suggest realpython [2] for resources on web scraping [0] https://automatetheboringstuff.com/2e/ https://automatetheboringstuff.com/2e/ [1] https://inventwithpython.com/automateudemy https://inventwithpython.com/automateudemy [2] https://realpython.com/tutorials/web-scraping/ https://realpython.com/tutorials/web-scraping/

iamwil 6y ago

I started curating a bunch of automation stories, use-cases, and ideas at https://automationcookbook.io https://automationcookbook.io. It is usually more business process automation, rather than personal automation, though.

Dolios 6y ago

An improved timeGained equation. timeGained = (manualTimeSpent * n) - (timeToAutomate)

Rexxar 6y ago
This forget (at least) three factors: - Repetitive manual labor is prone to failure. If failure is expansive and automated task could be better just for the additional safety. - Once it's automatized the frequency at which the task is done can be increased. - Once something is automatized, other things become easier to automatized.

nimrody 6y ago

There was a post here a few weeks ago reminding us of the dangers of this approach. I don't have the link but the essence is here: After a process gets automated, people forget the details of how to do the work manually. Now, when the automation breaks (and it will break at some point. It takes a huge amount of work to make these things failsafe and scripting doesn't really encourage robustness) -- no one remembers the details and fixing the issue is more difficult and requires specialized knowledge that is no longer common in the organization.

mxuribe 6y ago
Totally agree, and i've seen this many times (and have had my teams "clean up" after others). however, what can help mitigate - maybe not fully prevent - in some cases are the following 2 approaches: 1. Build automation so that it is NOT such a blackbox. I even go so far as to build automation with scripting languages...that allows myself/my team to at least peek at the source code to see what is happening. 2. And, of course documentation. i know this is simplistic of me to state...but it is invaluable nonethelss. I should also clarify that i mean there should exist documentation both for users, or by extension those who benefit from the automation, as well as, for developers/sys admins/sys maintainers (say, within the code itself).
gglitch 6y ago
This sounds to me like not a failure of automation as such, but of process/knowledge management by the team that builds the automation. No process is permanently exempt from entropy.

walterbell 6y ago

Low-cost NFC tag stickers can be used to trigger iPhone shortcuts, with an inch of proximity. iOS shortcuts can make SSH connections to issue commands via shell scripts on local Linux servers.

contingencies 6y ago
That would be a closed source stack and device dependent solution. You could also use QR codes to trigger URI access which in turn ran arbitrary code on arbitrary devices. That would be an open source stack and device agnostic solution.
- walterbell 6y ago
  NFC can trigger an interrupt to launch code/app, so proximity is sufficient. NFC exists on multiple platforms. QR codes would require launching an app to read the code, it would be faster for the user to launch the automation shortcut directly. There is also the issue of security. Which platform hardware+software would you entrust to pull the trigger on an automation sequence with physical side-effects?
  - contingencies 6y ago
    NFC costs more to set up. QR you can stick on anything easily, forward or share, etc. Trust? NIH. Something I wrote myself, only available on the local wireless network segment, probably.