6 ms·
Automating processes with software is hard
- qsort 2y agoThis reminds me of Seeing Like a State[0] (I know, it's not new... but I only got around to reading it earlier this year.) Automating a process makes it more standardized and legible, but takes away a lot of the nuance and of the resilience that human institutions tend to bake in. Do it too little, you get chaos; do it too much, you're destroying the very thing you were trying to nurture. It's certainly deformation professionnelle, but the parallel with software is eerily relevant. --- [0] https://en.wikipedia.org/wiki/Seeing_Like_a_State https://en.wikipedia.org/wiki/Seeing_Like_a_State
- veggieroll 2y agoOne thing on my list for CRUD software is an „I know what I’m doing” button that bypasses validation. Usually it’s behind manager or admin approval. It’s pretty tricky to get right and is very case by case in what it specifically means. But it’s been critical to handling edge cases. For example, in 99% of cases, you can’t progress to stage X without doing Y. But in this case „I know what I’m doing”. Though this puts a lot of pressure on downstream processes to handle missing data. Fortunately that hasn’t been an issue too much because validation grows and changes over time and must naturally deal with records that were created under prior validation when their state was originally valid.
- christkv 2y agoThis is so important. You don’t want to stop flows from happening because of a bug or bad config. Just keep track of the override who did it when and if possible a quick blurb. Then ensure those exceptions are caught and fed to dev.
- 0cf8612b2e1e 2y agoI like this idea, but seems like a bug factory. Going to break lots of downstream assumptions and create duplicate workflows for these (hopefully) rare events.
- duckmysick 2y agoThe "tricky to get right" part is so true. One time I was involved in automating a business process. Nothing super complicated - scrapping data from a portal, filling forms, some data wrangling. It had to handle a few edge cases and involve a human in the loop. I tried to include as much data validation as possible, but this being a business in real world there were of course some exceptions. I implemented a Superuser Override which was basically a button that launched a popup with another form where the app users requested an approval from their manager to accept the unvalidated data. It was red and scary and it had Bootstrap .danger all over because it was a cool thing to do back then. Don't judge, ok? Things worked as they should have. The automation saved tons of time, even if some fields were missing. Occasionally, there was a Superuser Override request which sent an email to the manager overseeing the process. Actually, things worked too well - the volume of processed data increased and so did the number of the Override requests. The manager - busy as they usually are - started approving all the requests. Manually at first and then they ran a macro that did it for them. Eventually the app users got wind of it and they started using the Superuser Override liberally. Before they shown more restraint. I wish there was some clever punch line to this story but I don't know what happened after. If I had to do it again, I'd be more aggressive with validations, make the approval process more like a code review, involve the most experienced app users as the Superuser, and have audit logs. Oh and Tailwind instead of Bootstrap.
- veggieroll 2y agoYeah. My first choice for approval is admins because some managers are better than others on this for sure. But that really is only possible because I build for a fairly small user base. Audit logging is definitely another must for me. I literally won't make an app without it. The app I'm working on now had a case where a PM promised up and down that edit history wasn't needed for a particular section, and lo-and-behold 2 years later it turns out that actually we do need the history.
- leetrout 2y agoRelated - I highly recommend reading Lisanne Bainbridge's paper "Ironies of Automation" which points out why we need to keep human factors in mind when designing automation. http://www.complexcognition.co.uk/2021/06/ironies-of-automation.html?m=1 http://www.complexcognition.co.uk/2021/06/ironies-of-automat...
- yen223 2y agoWe don't automate things because it's easy. We automate things because we thought it would be easy
- supportengineer 2y agoI see it as trading one class of problems for another set of problems.
- ethbr1 2y agoAs someone who spent most of a career in process automation, I've decided doing it well is mostly about state limitation. Exceptions or edge cases add additional states. To fight the explosion of state count (and the intermediate states those generate), you have a couple powerful tools: 1. Identifying and routing out divergent items (aka ensuring items get more similar as they progress through automation) 2. Reunifying divergent paths, instead of building branches Well-designed automation should look like a funnel, rather than a subway map. If you want to go back and automate a class of work that's being routed out, write a new automation flow explicitly targeting it. Don't try and kludge into into some giant spaghetti monolith that can handle everything. PS: This also has the side effect of simplifying and concluding discussions about "What should we do in this circumstance?" with other stakeholders. Which for more complex multi-type cases can be never-ending. PPS: And for god's sake, never target automation of 100% of incoming workload. Ever. Iteratively approach it, but accept reaching it may be impossible.
- HatchedLake721 2y agoWith your experience, anything you'd recommend to read in the process automation space? (I'm a founder of an automation SaaS where we've made "human interface" one of the core features of the product)
- trod123 2y agoI'm not aware of any books that have covered this appropriately. The biggest part of automation in my experience is boiling down the inputs to a 'unique' state that automation can then use as inputs and be run on. For computation to do work, it requires a property of consistency, and computers can only operate accurately and do work when properties of determinism are met. Also as the OP mentions, state can explode leading to sphagetti, this is why he mentions a sieve like approach based on similarity. Some problem spaces can be fundamentally inconsistent, such as with some approximations (common methods used for such), which falls back to what amounts to guesses, heuristics, and checks in terms of exception handling. There are problem scopes that cannot be characterized too so no amount of exception handling will resolve the entire scope, which is why you need fallbacks in a resilient design. If inputs cannot be controlled and uniquely differentiated, the automation fails in brittle ways, especially with regards to external change. The main interface (with regards to your core features) would be language, or communication. There exists words right now that can have contradictory, and different meanings, where the same word may mean the opposite depending on context, and this is not a general consensus but an individual one (where the individual may be misusing it). That breaks the 1:1 mapping required for determinism, and AI weights mimicking neurons have a narrow approximation where it may work under a narrow set of circumstances but no computer today can differentiate when the inputs are the same but have two or more, different states mixed in (many people forget that absence of a state is a state too) and then decompose them. Abstract decomposition seems to be something only humans are good at, and I'm glad this is the case otherwise none of us would have jobs.
- sholladay 2y agoIt doesn’t have to be so hard, though. One thing you can do is learn to just say “no”. Such as no to cash on delivery. Being flexible is good, but it comes at a cost. When that cost is too high, don’t do it. Realize that the customer who wants that workflow probably isn’t going to be your make-or-break moment. And in fact, they might have flexibility of their own. For example, you don’t accept Amex but they have a backup credit card or cash. It might be annoying to them if they are fussy, but it’s normal enough that the consequences are minimal. And yes, you may occasionally get a customer who doesn’t have that flexibility, but you shouldn’t be pinning your business on rare events. Figure out what’s most common among your target customers and support a few simple workflows. Say no to everything else until you have the resources to do it properly. Another thing you can do is have a hybrid low tech/high tech solution. Automate structured inputs with software. Write down the rest as notes in a logbook. Over time you will probably see patterns in the logbook that you can automate. Lastly, remember what Morpheus says in The Matrix, “Some [rules] can be bent. Others can be broken.” For example, you could simply pay the bill on behalf of the customer who wants cash on delivery. Now you assume some personal risk but the computer system doesn’t have to support their workflow. Is it worth it? Maybe, maybe not, but it’s a choice you can make.
- the_sleaze_ 2y agoThis is of course the right answer, reality is often easier to change than software when the goal is to keep reality and software in sync. So much harder to say no that it sounds though - sales is saying yes to everything they can, board is pressuring for more sales, the users are taking every possible opportunity to shift blame onto the software, and engineering is buckling under the churn of changing business goals every quarter.
- SoftTalker 2y agoDepends on the customer too though. Exceptions are always made for big/valuable customers, even if they are a PITA. If you have a long term big customer who has always been able to do COD and you suddenly pull that option with the explanation "it's easier for us" that's not going to go over well. Now you might lose them and they will tell their network "Wow, Foo Corp used to be good now they are just being unreasonable."
- omarhaneef 2y agoOne of the best pieces on here recently (and didn’t realize it was by Sinofsky!) Anyway, one other insight I would add is that the issues tend to come up at the interface between the systems. Your automation has to get some input and send some output and both those are pain points. That’s why we sometimes prefer imperfect monolith software.
- from-nibly 2y agoAlways reconsider the process itself before you automate it. Human based processes are different than computer based ones. Trying to directly emulate human actions is what can make automation fragile.
- bob1029 2y agoMuch of the pain can be reduced by considering if you should bother automating the thing in the first place. Some kinds of automation are absolutely essential. Aligning wafer stages during photolithography, shipping logistics, high frequency trading, etc. The business absolutely wouldn't work without it. The other kinds of automation are more questionable. Developing E2E CI/CD automation for an internal tool that is redeployed once every quarter might be a more difficult sell to management. For these cases, even if the rate of manual process invocation is somewhat high (i.e. frustrating for some employees), the customer can't see it. They won't pay one additional cent for this. There is also this entire meta time wasting game where the mere debate about "should we automate" takes up so much resources you could have manually invoked the thing more times than it would ever be feasibly invoked automatically during its projected lifetime. Alternatively, you could have just let your developers build the damn thing to get it out of their systems.
- awesomeMilou 2y ago> The other kinds of automation are more questionable. Developing E2E CI/CD automation for an internal tool that is redeployed once every quarter might be a more difficult sell to management. For these cases, even if the rate of manual process invocation is somewhat high (i.e. frustrating for some employees), the customer can't see it. They won't pay one additional cent for this. I've generally had the experience that it's the exact opposite. Best practices dictate that all processes should be automated to the limit.
- uoaei 2y agoThe thing about software is that it can only ever define its own process. "Software as spec". The goal of automation is to find processes whose architecture and dynamics are already amenable to being represented in software. Public reaction to automation has been mixed for obvious reasons, but also because when software is applied to structure- and/or determinism-resistant systems it fails to capture some essential components that ends up degrading the services that system can provide.
- hotspot_one 2y agohttps://xkcd.com/1205/ https://xkcd.com/1205/ and https://xkcd.com/974/ https://xkcd.com/974/
- kayge 2y agoDon't forget https://xkcd.com/1319/ https://xkcd.com/1319/ !
- saulpw 2y agoMidwits treat these as gospel. But when you automate something, you make it 10x more efficient, and this opens up the possibility of using the automation 10x more often. Adherents to xkcd/1205 are like "don't waste your time automating the full build process, we only do a full build a few times at the end of a release."
- bjornsing 2y ago> Then one day the business owner (my father) said “hey this customer is going to come by and pick up the order so don’t print a shipping label.” Yikes had to figure that out. Then one day another customer said “hey can we pay cash on delivery?” C.O.D. used to be a thing. Then I had to add that. What happens if you just say no? I have a feeling a lot of complexity stems from the fact that these exceptions are easy to handle when you’re doing things manually, so it’s hard to say no. But if you think about the business case I would not be surprised if the correct answer is no more often than not.
- immibis 2y agoNot in these cases. It doesn't make sense to deny taking money from a customer unless you're near capacity and can replace the customer.
- bjornsing 2y agoI’d say that depends. If it will cost you more money to handle a specific exception than what you can generate in gross profit from those sales, then you should just say no.
- marcosdumay 2y ago> What happens if you just say no? Do you want to say no? That's not an easy question to answer. You should think about it. Is it worth alienating some customers to handle them in a larger scale? Naively, this looks like an automatic "no", but it's not.
- TomNomNom 2y agoNot directly relevant to the post, but seems like a good place to share. My team and I once took on a very tricky automation project. At the time we had a complex software deployment done about once per month that involved a team of about a dozen people showing up at 4am to do it while traffic was low. The deployment involved many manual steps and coordination of everybody involved. The person leading each deployment followed the documented list of steps and got each person to do their bit at the right time; people to run database migrations, people to install RPMs on particular servers, people to test and verify functionality. Mistakes and missed steps were not uncommon. The very first thing we did was take the documentation and write a Jenkins job to post each step into a Slack channel specifically for coordinating the deployments. Someone clicked "go" and each step was posted as a message in that channel with a 'done' button to be clicked when that step was done. Clicking the button caused the next step to be posted. The next release we did used that instead of one person reading the steps out of confluence. Everyone involved in the release could always see what step we were at, and when it was their turn to do their bit. This helped ensure no steps were ever missed too. Over the following months we chipped away at that job a bit at a time. We'd pick a step in the process and automate just that step, starting with the low-hanging fruit first. The Slack message for that step went from "click to confirm you've done it" to "click to do it", with the result posted once it was done; followed by the next step to perform. It was a long process, but it allowed the rest of the business (and us!) to gradually gain confidence in the automation, and lowered the risk of the project dramatically. Once several steps had been automated and battle-tested we removed the 'click to do' bits in between and the whole release became a couple of clicks followed by the odd bit of manual QA.
- GarnetFloride 2y agoChecklist automation is a very powerful tool. Making a script that just calls the checklist that you walk through is the first step because you have to debug the checklist. It's really hard to automate something if you don't really known what it is you are doing, and steps are always missed. Once you debug the checklist you can start automating the steps and then you find the things that are easy for humans to do but hard for computers. That's the fun part. In one case I worked on, they automated the sales funnel, but then customers started asking for refunds and complaining online. Turns out the automation didn't work. I got a team together to get the customers happy and then did a Stalin's Postman test where we walked through the whole process. All but one of the steps failed. Now that we knew what was failing, we could start the process for fixing it.
- j45 2y agoIt's important to systemize (get it working manually) before automating. Premature automation can create more complexity in the process as edge cases are usually added in, or needing routing. If you're building process management and automation from scratch as a way to learn, it's also a red flag of technical debt waiting.
- prometheus76 2y agoThis is an excellent and insightful article, but it feels like it was speech-to-text and the author didn't take the time to clean it up before posting it. It's a little distracting.
- teddyh 2y agoNo kidding. “ate ice create 14 hours ago”. I’m guessing they meant “ice cream”, but it’s very hard to push yourself through reading sloppy writing like this. The author should have taken their own advice and not automated speech-to-text, spellchecking, and editing.
- jakub_g 2y agoThe author is probably a rather busy person (https://en.m.wikipedia.org/wiki/Steven_Sinofsky https://en.m.wikipedia.org/wiki/Steven_Sinofsky) and I appreciate they took time to share their thoughts, even with typos. Yeah I also got caught with the one above, but didn't spot any other.
- taeric 2y agoThis is demonstrably correct, to me, but I'm curious what makes this unique to software? Hardware automation has had obvious boons to places it works. And often we will force the places we want it to be to change to accommodate the automation. Agriculture is an easy example of what I mean here. Take a look at some fields that are optimized for giant machinery. Very impressive and can manage a lot. My favorite recently was seeing the "kickout" effort to prevent bad fruit from entering a hopper. To that end, is the main issue with software automation that it thinks it can be a total solution and often tries to start by not imposing changes on what is being automated?
- ninalanyon 2y agoThe biggest problems with any automation are describing what the current process really is and discovering which parts are essential and which are simply accidental. This is true whether one is creating a purely mechanical system or a purely software system The part that is unique to software is that companies often expect people whose only expertise is in software to do both of these tasks when the second often requires deep domain knowledge. When one mechanises something in hardware it is generally taken for granted that domain experts will be central to the effort but when the result is principally software, domain experts are often left out of the process.
- troupe 2y agoThe Do Nothing Scripting [1] is an approach that I feel recognizes the difficulty of automation by starting out with a script that just tells you what to do like a checklist. Once you've used it to fully document all the exceptions, etc. you can then start building the automation. An approach like this seems to give more weight to the fact that just figuring out a way to document exactly what needs to be done is often the hardest part and if you get that right before you start writing automation code, it might make make the automation much more efficient. [1] https://blog.danslimmon.com/2019/07/15/do-nothing-scripting-the-key-to-gradual-automation/ https://blog.danslimmon.com/2019/07/15/do-nothing-scripting-...
- rambambram 2y agoOu est mon scrollbar?
- ozim 2y agoThat is something that comes from experience - automate happy flow so you can handle all normal cases super fast - handle edge cases manually. It always was the biggest issue I have seen “sorry cannot help you with that because our system says so”, which should never be the case. There should always be way around the system. Then of course way around the system needs approval from some higher up anyways.
- akira2501 2y ago> My favorite example of the latter is how the arrival of IBM computing in the 60s and 70s totally changed the definition of accounting, inventory control, and business operations. Every process that was "computerized" ultimately looked nothing at all like what was going on under those green eyeshades in accounting. How did you arrive at that conclusion? Most of those applications were written by the businesses themselves. IBM provided you a programming environment not a set of predefined business software. Which is why COBOL jobs still exist. What changed with the mainframe was that instead of having a bunch of disparate and disconnected processes you now had a centralized set of dependent processes and a database that provided a single source of truth. Businesses were going to want this capability regardless of how limited humans with green eyeshades were previously. > Much of the early internet (and still most bank and insurance) look like HTML front ends to mainframe 3270 screens. Well, precisely, those are custom applications. If they weren't we wouldn't have this issue. You talk about automation but you seem to have not noticed that mainframes have CICS, why it exists, and why businesses still use it. The "old school" ways are actually still exceptionally powerful.
- kmoser 2y agoAlso: > As it turns out the entire flow at the post office (or DMV or tax office) is about exception handling. No amount of software is going to get you out of there because it is piecing together a bunch of inputs and outputs that are outside the bounds of a system. Not sure how the author arrived at this conclusion, either. Both entities (USPS and DMV) have elaborate rules for how business gets done. The USPS in particular has rules that cover shipping everything imaginable, often via different methods. The outputs are quite simple: USPS ships your package; DMV provides your license or vehicle registration.
- bokohut 2y agoAutomation is about saving one's time and time is something that no amount of money can directly buy more of. One can invest their time and money however to indirectly "buy" one more time through reduction of mundane tasks that many humans have come to accept as normal. Software only solutions aside my own personal and professional experiences have involved building several automation sensor networks and one of the first I built was over 20 years ago that was within my own house. In 2004 my master electrician father and I wired my entire house with both high voltage, 220v/110v, as well as low voltage, 24v/12v, in my plan to build a smart home. My automation planning choices then were focused on exactly this, saving me sub seconds of time here, there, and everywhere. As a former electrician I was acutely aware of light switch placement in a room and the time it takes to turn it on and off each and every time but I asked myself then if my creative awareness could successfully design out the need to turn on a light switch as I enter a room? I did just that using the automation sensor network I built that interacts with source code I wrote which manipulates other devices based on logic of sensor states and any other sensors or inputs I deemed inclusive for the logic's objective. In the last 20 years I have rarely turned on light switches as they come on by "magic" when anyone enters a room and this "magic" is actually real world human time saved for that individual entering said room. I encourage those intrigued by these words to do some of your own math on the time you spend where you reside turning on and off light switches. There will be those here that see it foolish to automate anything, any longshore men present here? There will be those here also that see some benefits to it but having lived in it for 20 years I will never live without it again. Stay Healthy!
- moandcompany 2y agoThe second order costs of automation should consider time spent diagnosing and repairing said automation, and consequences of automation failure. For example, should an automation fail, will manual processes still work? Does execution of a manual correction, or repair, require the knowledge or skills of a particular person or persons that may not be available, etc, or can it be done by the normal persons that the automation is intended to serve/benefit? In the Smart Home context, when the automation or other cleverness fails, will ____ still be operable normally by a regular person?
- hinkley 2y agoI don't set out to do automation but the automation eventually finds me. I start making a tool for myself to track a tedious or error prone process. Even after you've gotten the wiki to have the correct steps in the correct order, you can still run into human error. I use it and work on it until I stop burning my fingers. Then I share it with a few people who complain about the same thing, and see how they burn their fingers. Then I roll it out to a wider team, wait for more problems, then suggest it for general use. Usually by this point we are seeing fewer problems with the tools than with humans doing things manually, so we escalate to teasing people to use the tool so they don't end up with an Incident. Then it becomes mandatory and anyone who still refuses to use it is now risking PIP-level trouble if they don't join the party. People who try to skip steps make not only a lot of problems for themselves but also for anyone else who suggests automation tools. Running toward the finish line makes everything worse.
- pwojnaro 2y agoThis becomes even harder if your execution environment is non-deterministic by definition. Imagine test automation on physical devices (ex-founder of a mobile app testing SaaS company here) where system0level events that can't be programmatically detected or dismissed can delay, interrupt or alter the outcome of your tests. Dealing with exceptions to make execution outcomes more stable seem like an obvious choice, but what if it's not feasible due to the complexity of the environment? We found that accepting variance and extrapolating test results from multiple executions in the same (or identical) environment was the only way forward.
- svieira 2y ago> AI as a tool is regulated just as a PC or calculator or PDR is regulated — the regulation is not the tool but the person using the tool. Another gem from the article that I wanted to surface. It's yet-another-take on the general sentiment here, but it's very succinct. Automation is good, but automation is a tool that serves people (and not in a Soylent Green kind of way).
- gcanyon 2y agoOnly tangentially related, but I like the quote so I'll share it: a boss of mine once said, "You can't software your way out of a process problem."
- airbreather 2y agoMedical diagnosis is not a form of state based behaviour, therefore it can't be "automated" in the strict sense. Medical diagnosis is interpretation, you can't predict all possible states involved in the diagnosis and outcome, it is inherently unsuitable to be treated in such a way.
- akhileshwar09 2y agowell ,we cant say automation can make mistakes these days. cus the current machienes and the prototypes are soo complex than before and doin accurate jobs.