7 ms·

Almost 10 years ago, when I just started to work, I had to do a repetitive task. Something in IIS about 30-50 subdomain bindings. I told to myself “I’m not goin

Almost 10 years ago, when I just started to work, I had to do a repetitive task. Something in IIS about 30-50 subdomain bindings. I told to myself “I’m not going to do that manually, such a waste of time, let’s write a script!”. 5 hours later, I was still working on this script and I was stuck. I asked some help to a senior colleague and his reply was: “you know, it would have taken you only 10 min to do it manually”. :)

dmortin 8y ago
If it's a one time task then there is no need for a script, but if you have to do it again and again then it's very much worth it to write a script.
- lloeki 8y ago
  Scripts/automation can be worth it even for one-shot tasks for which time to create a script > time to do the task manually: a) They help in applying forethought to a process. Maybe you can even dry-run the thing. b) They help documenting what has been done, and how, which can be invaluable in a team setting. c) They train you into getting incrementally better at automating, and at sensing when to automate and when not to†. If it's painful, you're not doing enough of it. † I find that for the ones that would take ages to properly script, more often than not a combination of manual (like preparing a normalised input for the script using vim) and automated subtasks can do wonders, leveraging the best of wetware and software worlds.
  - scarface74 8y ago
    The only thing that I will do manually in any environment is putting sensitive data into configuration. With IAM roles, I can usually avoid that. I will create the roles manually. If you go by the old mantra of treating servers like cattle and not pets, you have to automate everything. My VMs (EC2 instances) can be destroyed, replicated, auto scaled, etc. at any time. I have to automate no matter how long it takes to figure out.
dougmwne 8y ago
You are so right from a business standpoint. But what about personal development? This is one of the ways developers are born. Sure the first one or ten attempts to automate something might be a net loss to the business, but eventually that junior employee has leveled up their programming skills and developed an estimating sense of when to automate and when to suck it up. It's easy to sit at our place of knowledge and tut at the beginers but we were all beginers once.
- shostack 8y ago
  I'm an "early" programmer and still poking my way around various toy side projects to learn concepts and such. Ubuntu and Bash had been somewhat of a mystery to me previously, but I eventually got a virtual dev environment setup and was comfortable getting in with navigating to my Vagrant folder, then executing 'vagrant up', 'vagrant ssh', etc. and then navigating to my project. That comfort turned to annoyance as I realized how unnecessary those few keystrokes were. And despite how little time they took, I found myself getting angry at the repetition of having to enter them. So I decided to conquer my Bash scripting fears, and wrote what many would consider to be a trivial script to navigate to the Vagrant folder on my host machine, boot up the VM, and then navigate to the appropriate project folder in the VM. Along the way I learned more about aliases, and also how to pass commands from a host to a guest machine which was one of the roadblocks I encountered. I now can boot up and get into my project folder with a simple three character alias from within any directory on my host machine. Again--any experienced developer would be able to script this in minutes most likely. Back then it took hours. And now those are the most satisfying three characters I type, as every time I enter them, I get reminded of the progress I'm making and that even trivial things may be candidates for automation.
  - sgwae 8y ago
    You could still save yourself the misery of typing one or two characters if you fix that alias.
    - otisfunkmeyer 8y ago
maxxxxx 8y ago
From a short time perspective you wasted your time. But in the long run you learned valuable skills that later may have enabled you to automate other more important things.
snerbles 8y ago
From a pure time savings perspective, this makes sense - naturally, links to XKCD are already popping up. From a maintenance perspective, a well-written script can serve as a reference after the original task was long forgotten. A poor substitute for documentation, but when performing software archaeology on legacy systems you really need everything you can get.
z0r 8y ago
Often the time invested in writing an automation script far outweighs the time spent doing it manually, even when you account for every time the script will ever be run. And it can still be worth it if the script never makes a mistake that human might.
- masukomi 8y ago
  even when it doesn't. I find that if you are careful about adhering to the unix philosophy of tiny scripts that do one thing well then you end up with lots of tools to solve problems specific to your problem domain that get reused in future tools.
- matthberg 8y ago
  This xkcd is really relevant, being a chart of the time invested vs time saved, not even including the accuracy and learning value: https://xkcd.com/1205/ https://xkcd.com/1205/
jrockway 8y ago
I always look at it as decreasing a dependency on me personally. If I'm sick, the task can't be done at all, for example. So even if you spend an hour automating something that can be done manually in 10 minutes, it's a good use of time because the script doesn't get sick, doesn't go on vacation, doesn't leave for another company, etc.
- taneq 8y ago
  The flip side of this is your company gradually becoming dependent on a sprawling and ever-growing tangle of likely undocumented scripts. This can be a time bomb (no matter how good your intentions) waiting to explode if you ever leave.
  - ivan_ah 8y ago
    Not necessarily. If the script is documented and written cleanly (e.g Fabric3 which has very little magic, and reads almost like Bash), then in can server as "runnable documentation". In this case automation could be really good for continuity and busfactor. But yeah... got to put some effort into it. Like all docs, script-docs worn't write themselves.
    - taneq 8y ago
      Agree that it's not unavoidable, but the problem is deeper than just the scripts themselves being well documented and straightforward. The main risk (IME, anyway) is awareness / discovery of the scripts in a timely manner - the fact that everything 'just works' means that it can be months or years before something breaks, at which point it may be too late to ask the person who set it up. Hopefully there's good handover from the outgoing staff member to a technically proficient colleague. If so then everything might be OK but if not then a catastrophic failure may well be the first sign that something is wrong. (A recent example of much simpler automation going horribly wrong was a company losing their top-level domain name because it was set to auto-renew using an ex-employee's personal credit card, and their email address as contact. This worked for years until that card was cancelled and the warning emails bounced off the closed account.)
deleted 8y ago
[deleted]
vkjv 8y ago
I can think of a few benefits, even if it takes longer: you may need to do something similar again and now you have the script, you've gained experience and you might right the next script faster, correctness. I think correctness is the most important one. Even if a manual task is quick, ensuring that you do it correctly and exactly the same way for all 30-50 things is extremely tedious. If one out of those 50 subdomains was behaving differently and you needed to investigate exactly why, that could quickly erode any time savings.
- jessaustin 8y ago
  Correctness is huge. The other day a colleague and I were figuring out some things about scheduled work days over the next few years. After I had futzed about in a spreadsheet for five minutes, he said "this is ridiculous!" and started furiously paging through a paper calendar. We had our results nearly simultaneously, but he was off by over a month. After that we just used the spreadsheet.
  - hectormalot 8y ago
    Correctness also works the other way around. We needed to link to databases with non-matching customer IDs, the analyst on the team spent several hours automating fuzzy matching of customer names in R before we concluded it was easier and more correct to build a mapping table by hand for the few 100 that didn’t match.
defertoreptar 8y ago
My rationale is that every time I automate a task, I get better at automating tasks. Losing a battle here and there helps me win more battles later on, and by the end of my life I will have netted more time saved than lost. Also, automation work is much more interesting than doing stupid, repetitive tasks.
meej 8y ago
I am reminded of this Brian Marick classic: http://www.exampler.com/testing-com/writings/automate.pdf http://www.exampler.com/testing-com/writings/automate.pdf
ajnin 8y ago
Relevant : https://xkcd.com/1319/ https://xkcd.com/1319/ and : https://xkcd.com/1205/ https://xkcd.com/1205/
TomK32 8y ago
sure, but mess it up once and you spent time to figure the error and fix it. or go on holidays and your colleagues just gotta run the script. the big benefits in scripts that is often overlooked: Given the same conditions and input, the output will always meet your expectations. There's no chance for human error.
- aaron695 8y ago
  It's worse. If the employee has no idea what's going on, there's more chance if something goes wrong they'll have no idea and it's an easy out for them. "I ran the script as asked" Unlike manual where it might be longer but if there's a issue they will know and it's their responsibility to move forward. What do you trust more, asking a emoloyee to ensure the backups are being done or that they have run the backup script....
agumonkey 8y ago
A classic. Someone else might have said: next script will only take 15min. Maybe partial automation would have been a good roi. Or maybe being deep into manual is also neat for your brain.
- mattkrause 8y ago
  For data analysis, there’s a lot to be said for examining the data instead of blindly piping your data through something.
jillesvangurp 8y ago
I make these tradeoffs a lot. Automating is fun but time consuming. Doing things manually is tedious. However, automating has one benefit: you make things easier for the next time you do something similar because you learn something. Usually requests for tedious things are not isolated incidents. So I have a bunch of commandline stuff that I know how to glue together like csvkit, jq, xargs, and a simple bash script that adds functions to do rest against our api server. Using that I can do easy bulk operations against our db without investing in product features, ui, etc.
alifaziz 8y ago
Upfront time investment to build automation script sometimes more than manual. But most of the time it's a one time effort to save more of your future time.
charlieflowers 8y ago
Terrance Parr's motto (the creator of ANTLR) is “Why program by hand in five days what you can spend five years of your life automating?”
lbotos 8y ago
XKCD always delivers: https://xkcd.com/1205/ https://xkcd.com/1205/
sh87 8y ago
I am a script junkie. I have to, just have to script every task out. Here's what I've realized. First do it manually, then automate it. Doing manual tasks is frustrating enough to motivate me to automate it. Plus I have a much better understanding of the inputs, the required outputs and constraints for the script to be more useful. Let your scripts evolve. Don't force everything in the first go. Simplicity is the key. Let them be rigid, readable and in a single file at first. Then, introduce flexibility as required. Abstract out core functions into individual scripts/functions/modules. This builds a certain sense of familiarity with the code that is just too endearing and eventually very productive. Reuse your mini-scripts. Each of my scripts does one thing. There are super-scripts that combine many of those for a task that I am performing at the moment. DRY principle. The more simple, atomic and mutually agreeable your scripts are the more powerful they get. This is just the unix way of doing things. Nothing new. Write consistent code comments You will write. You will forget. Make the code comments personal and consistent enough for you to immediately recognize them. Use version control Absolutely use it. There will be many versions of your core scripts/functions that dont agree with each other. My personal preference is a version number (semver) suffix on each file to identify which core script is used by which task script. Reap the rewards The rewards of putting in the effort to me have been akin to compound interest on an early investment. I have reused core scripts so heavily that prototyping any new idea is pretty easy. If there is something that a script does not do, I know what I need to read over the next weekend :)

dmortin 8y ago

If it's a one time task then there is no need for a script, but if you have to do it again and again then it's very much worth it to write a script.

lloeki 8y ago
Scripts/automation can be worth it even for one-shot tasks for which time to create a script > time to do the task manually: a) They help in applying forethought to a process. Maybe you can even dry-run the thing. b) They help documenting what has been done, and how, which can be invaluable in a team setting. c) They train you into getting incrementally better at automating, and at sensing when to automate and when not to†. If it's painful, you're not doing enough of it. † I find that for the ones that would take ages to properly script, more often than not a combination of manual (like preparing a normalised input for the script using vim) and automated subtasks can do wonders, leveraging the best of wetware and software worlds.
- scarface74 8y ago
  The only thing that I will do manually in any environment is putting sensitive data into configuration. With IAM roles, I can usually avoid that. I will create the roles manually. If you go by the old mantra of treating servers like cattle and not pets, you have to automate everything. My VMs (EC2 instances) can be destroyed, replicated, auto scaled, etc. at any time. I have to automate no matter how long it takes to figure out.

dougmwne 8y ago

You are so right from a business standpoint. But what about personal development? This is one of the ways developers are born. Sure the first one or ten attempts to automate something might be a net loss to the business, but eventually that junior employee has leveled up their programming skills and developed an estimating sense of when to automate and when to suck it up. It's easy to sit at our place of knowledge and tut at the beginers but we were all beginers once.

shostack 8y ago
I'm an "early" programmer and still poking my way around various toy side projects to learn concepts and such. Ubuntu and Bash had been somewhat of a mystery to me previously, but I eventually got a virtual dev environment setup and was comfortable getting in with navigating to my Vagrant folder, then executing 'vagrant up', 'vagrant ssh', etc. and then navigating to my project. That comfort turned to annoyance as I realized how unnecessary those few keystrokes were. And despite how little time they took, I found myself getting angry at the repetition of having to enter them. So I decided to conquer my Bash scripting fears, and wrote what many would consider to be a trivial script to navigate to the Vagrant folder on my host machine, boot up the VM, and then navigate to the appropriate project folder in the VM. Along the way I learned more about aliases, and also how to pass commands from a host to a guest machine which was one of the roadblocks I encountered. I now can boot up and get into my project folder with a simple three character alias from within any directory on my host machine. Again--any experienced developer would be able to script this in minutes most likely. Back then it took hours. And now those are the most satisfying three characters I type, as every time I enter them, I get reminded of the progress I'm making and that even trivial things may be candidates for automation.
- sgwae 8y ago
  You could still save yourself the misery of typing one or two characters if you fix that alias.
  - otisfunkmeyer 8y ago

maxxxxx 8y ago

From a short time perspective you wasted your time. But in the long run you learned valuable skills that later may have enabled you to automate other more important things.

snerbles 8y ago

From a pure time savings perspective, this makes sense - naturally, links to XKCD are already popping up. From a maintenance perspective, a well-written script can serve as a reference after the original task was long forgotten. A poor substitute for documentation, but when performing software archaeology on legacy systems you really need everything you can get.

z0r 8y ago

Often the time invested in writing an automation script far outweighs the time spent doing it manually, even when you account for every time the script will ever be run. And it can still be worth it if the script never makes a mistake that human might.

masukomi 8y ago
even when it doesn't. I find that if you are careful about adhering to the unix philosophy of tiny scripts that do one thing well then you end up with lots of tools to solve problems specific to your problem domain that get reused in future tools.
matthberg 8y ago
This xkcd is really relevant, being a chart of the time invested vs time saved, not even including the accuracy and learning value: https://xkcd.com/1205/ https://xkcd.com/1205/

jrockway 8y ago

I always look at it as decreasing a dependency on me personally. If I'm sick, the task can't be done at all, for example. So even if you spend an hour automating something that can be done manually in 10 minutes, it's a good use of time because the script doesn't get sick, doesn't go on vacation, doesn't leave for another company, etc.

taneq 8y ago
The flip side of this is your company gradually becoming dependent on a sprawling and ever-growing tangle of likely undocumented scripts. This can be a time bomb (no matter how good your intentions) waiting to explode if you ever leave.
- ivan_ah 8y ago
  Not necessarily. If the script is documented and written cleanly (e.g Fabric3 which has very little magic, and reads almost like Bash), then in can server as "runnable documentation". In this case automation could be really good for continuity and busfactor. But yeah... got to put some effort into it. Like all docs, script-docs worn't write themselves.
  - taneq 8y ago
    Agree that it's not unavoidable, but the problem is deeper than just the scripts themselves being well documented and straightforward. The main risk (IME, anyway) is awareness / discovery of the scripts in a timely manner - the fact that everything 'just works' means that it can be months or years before something breaks, at which point it may be too late to ask the person who set it up. Hopefully there's good handover from the outgoing staff member to a technically proficient colleague. If so then everything might be OK but if not then a catastrophic failure may well be the first sign that something is wrong. (A recent example of much simpler automation going horribly wrong was a company losing their top-level domain name because it was set to auto-renew using an ex-employee's personal credit card, and their email address as contact. This worked for years until that card was cancelled and the warning emails bounced off the closed account.)

deleted 8y ago

[deleted]

vkjv 8y ago

I can think of a few benefits, even if it takes longer: you may need to do something similar again and now you have the script, you've gained experience and you might right the next script faster, correctness. I think correctness is the most important one. Even if a manual task is quick, ensuring that you do it correctly and exactly the same way for all 30-50 things is extremely tedious. If one out of those 50 subdomains was behaving differently and you needed to investigate exactly why, that could quickly erode any time savings.

jessaustin 8y ago
Correctness is huge. The other day a colleague and I were figuring out some things about scheduled work days over the next few years. After I had futzed about in a spreadsheet for five minutes, he said "this is ridiculous!" and started furiously paging through a paper calendar. We had our results nearly simultaneously, but he was off by over a month. After that we just used the spreadsheet.
- hectormalot 8y ago
  Correctness also works the other way around. We needed to link to databases with non-matching customer IDs, the analyst on the team spent several hours automating fuzzy matching of customer names in R before we concluded it was easier and more correct to build a mapping table by hand for the few 100 that didn’t match.

defertoreptar 8y ago

My rationale is that every time I automate a task, I get better at automating tasks. Losing a battle here and there helps me win more battles later on, and by the end of my life I will have netted more time saved than lost. Also, automation work is much more interesting than doing stupid, repetitive tasks.

meej 8y ago

I am reminded of this Brian Marick classic: http://www.exampler.com/testing-com/writings/automate.pdf http://www.exampler.com/testing-com/writings/automate.pdf

ajnin 8y ago

Relevant : https://xkcd.com/1319/ https://xkcd.com/1319/ and : https://xkcd.com/1205/ https://xkcd.com/1205/

TomK32 8y ago

sure, but mess it up once and you spent time to figure the error and fix it. or go on holidays and your colleagues just gotta run the script. the big benefits in scripts that is often overlooked: Given the same conditions and input, the output will always meet your expectations. There's no chance for human error.

aaron695 8y ago
It's worse. If the employee has no idea what's going on, there's more chance if something goes wrong they'll have no idea and it's an easy out for them. "I ran the script as asked" Unlike manual where it might be longer but if there's a issue they will know and it's their responsibility to move forward. What do you trust more, asking a emoloyee to ensure the backups are being done or that they have run the backup script....

agumonkey 8y ago

A classic. Someone else might have said: next script will only take 15min. Maybe partial automation would have been a good roi. Or maybe being deep into manual is also neat for your brain.

mattkrause 8y ago
For data analysis, there’s a lot to be said for examining the data instead of blindly piping your data through something.

jillesvangurp 8y ago

I make these tradeoffs a lot. Automating is fun but time consuming. Doing things manually is tedious. However, automating has one benefit: you make things easier for the next time you do something similar because you learn something. Usually requests for tedious things are not isolated incidents. So I have a bunch of commandline stuff that I know how to glue together like csvkit, jq, xargs, and a simple bash script that adds functions to do rest against our api server. Using that I can do easy bulk operations against our db without investing in product features, ui, etc.

alifaziz 8y ago

Upfront time investment to build automation script sometimes more than manual. But most of the time it's a one time effort to save more of your future time.

charlieflowers 8y ago

Terrance Parr's motto (the creator of ANTLR) is “Why program by hand in five days what you can spend five years of your life automating?”

lbotos 8y ago

XKCD always delivers: https://xkcd.com/1205/ https://xkcd.com/1205/

sh87 8y ago

I am a script junkie. I have to, just have to script every task out. Here's what I've realized. First do it manually, then automate it. Doing manual tasks is frustrating enough to motivate me to automate it. Plus I have a much better understanding of the inputs, the required outputs and constraints for the script to be more useful. Let your scripts evolve. Don't force everything in the first go. Simplicity is the key. Let them be rigid, readable and in a single file at first. Then, introduce flexibility as required. Abstract out core functions into individual scripts/functions/modules. This builds a certain sense of familiarity with the code that is just too endearing and eventually very productive. Reuse your mini-scripts. Each of my scripts does one thing. There are super-scripts that combine many of those for a task that I am performing at the moment. DRY principle. The more simple, atomic and mutually agreeable your scripts are the more powerful they get. This is just the unix way of doing things. Nothing new. Write consistent code comments You will write. You will forget. Make the code comments personal and consistent enough for you to immediately recognize them. Use version control Absolutely use it. There will be many versions of your core scripts/functions that dont agree with each other. My personal preference is a version number (semver) suffix on each file to identify which core script is used by which task script. Reap the rewards The rewards of putting in the effort to me have been akin to compound interest on an early investment. I have reused core scripts so heavily that prototyping any new idea is pretty easy. If there is something that a script does not do, I know what I need to read over the next weekend :)