6 ms·

As someone who spent most of a career in process automation, I've decided doing it well is mostly about state limitation. Exceptions or edge cases add addition

by ethbr1 2y ago

As someone who spent most of a career in process automation, I've decided doing it well is mostly about state limitation. Exceptions or edge cases add additional states. To fight the explosion of state count (and the intermediate states those generate), you have a couple powerful tools: 1. Identifying and routing out divergent items (aka ensuring items get more similar as they progress through automation) 2. Reunifying divergent paths, instead of building branches Well-designed automation should look like a funnel, rather than a subway map. If you want to go back and automate a class of work that's being routed out, write a new automation flow explicitly targeting it. Don't try and kludge into into some giant spaghetti monolith that can handle everything. PS: This also has the side effect of simplifying and concluding discussions about "What should we do in this circumstance?" with other stakeholders. Which for more complex multi-type cases can be never-ending. PPS: And for god's sake, never target automation of 100% of incoming workload. Ever. Iteratively approach it, but accept reaching it may be impossible.

HatchedLake721 2y ago
With your experience, anything you'd recommend to read in the process automation space? (I'm a founder of an automation SaaS where we've made "human interface" one of the core features of the product)
- trod123 2y ago
  I'm not aware of any books that have covered this appropriately. The biggest part of automation in my experience is boiling down the inputs to a 'unique' state that automation can then use as inputs and be run on. For computation to do work, it requires a property of consistency, and computers can only operate accurately and do work when properties of determinism are met. Also as the OP mentions, state can explode leading to sphagetti, this is why he mentions a sieve like approach based on similarity. Some problem spaces can be fundamentally inconsistent, such as with some approximations (common methods used for such), which falls back to what amounts to guesses, heuristics, and checks in terms of exception handling. There are problem scopes that cannot be characterized too so no amount of exception handling will resolve the entire scope, which is why you need fallbacks in a resilient design. If inputs cannot be controlled and uniquely differentiated, the automation fails in brittle ways, especially with regards to external change. The main interface (with regards to your core features) would be language, or communication. There exists words right now that can have contradictory, and different meanings, where the same word may mean the opposite depending on context, and this is not a general consensus but an individual one (where the individual may be misusing it). That breaks the 1:1 mapping required for determinism, and AI weights mimicking neurons have a narrow approximation where it may work under a narrow set of circumstances but no computer today can differentiate when the inputs are the same but have two or more, different states mixed in (many people forget that absence of a state is a state too) and then decompose them. Abstract decomposition seems to be something only humans are good at, and I'm glad this is the case otherwise none of us would have jobs.
bloopernova 2y ago
> Well-designed automation should look like a funnel, rather than a subway map. Do you have any examples of this? Like an example project? > never target automation of 100% of incoming workload. Ever. A new product owner came in last year and wanted to "automate everything". They wanted a web page where builds, branches, commits, and more were all on one page that showed exactly which commit was deployed where, by whom, etc etc. They wanted this extravaganza for a 2-person application that was in maintenance with no new features. They're also the kind of person who consults chatgpt or copilot on a subject while you're explaining that subject to them, to check that the LLM agrees with what you are saying. They'll even challenge people to prove some copilot output is incorrect. It seems to me that they consider LLMs more reliable than people.
- shakna 2y ago
  > They wanted a web page where builds, branches, commits, and more were all on one page that showed exactly which commit was deployed where, by whom, etc etc. Sounds like they would have been a fan of ungit [0], which I have seen used for that kind of flow overview, though it has looked more impressive than actually proved helpful in my experience. [0] https://github.com/FredrikNoren/ungit https://github.com/FredrikNoren/ungit
- chrsig 2y ago
  > A new product owner came in last year and wanted to "automate everything". They wanted a web page where builds, branches, commits, and more were all on one page that showed exactly which commit was deployed where, by whom, etc etc. They wanted this extravaganza for a 2-person application that was in maintenance with no new features. you know, this seems very reasonable until that last sentence.
  - ykonstant 2y ago
    Why a web page and not directly the git log? If style is necessary, reformat the log data with some fancy ASCII art?
    - 2y ago
chrsig 2y ago
I wouldn't consider my career being in process automation, but I feel like you just described my approach to managing new product development on top of existing machinery while ensuring no breakage of what exists.
InDubioProRubio 2y ago
or: https://fsharpforfunandprofit.com/rop/ https://fsharpforfunandprofit.com/rop/
davedx 2y ago
This has a corollary in software and refactoring: don’t refactor duplicated code too early; keep it separate until it’s very clear the refactoring is correct
seanlinehan 2y ago
Spot on. I was an early eng and first VP of Product at Flexport. Global logistics is inherently complicated and involves coordinating many disparate parties. To complete any step in the workflow, you're generally taking in input data from a bunch of different companies, each of which have varying formats and quality of data. A very challenging context if your goal is process automation. The only way to make progress was exactly the way you described. At each step of the workflow, you need to design at least 2 potential resolution pathways: 1. Automated 2. Manual For the manual case, you have to actually build the interfaces for an operator to do the manual work and encode the results of their work as either: 1. Input into the automated step 2. Or, in the same format as the output of the automated case In either case, this is precisely aligned with your "reuinifying divergent paths" framing. In the automated case, you actually may wind up with N different automation pathways for each workflow step. For an example at Flexport: if we needed to ingest some information from an ocean carrier, we often had to build custom processors for each of the big carriers. And if the volume with a given trading partner didn't justify that investment, then it went to the manual case. From the software engineering framing, it's not that different from building a micro-services architecture. You encapsulate complexity and expose standard inputs and outputs. This avoids creating an incomprehensible mess and also allows the work to be subdivided for individual teams to solve. All that said – doing this in practice at a scaling organization is tough. The micro-services framing is hard to explain to people who haven't internalized the message. But yeah, 100% automation is a wild-goose chase. Maybe you eventually get it, maybe not. But you have to start with the assumption that you won't or you never will.
- initplus 2y ago
  Sounds like a really interesting problem space. I'm curious if you have any comments about how you approached dealing with inconsistencies between information sources? System A says X, system B says Y. I suppose best approach is again just to bail out to manual resolution?
  - seanlinehan 2y ago
    In the early days, we bailed out to manual resolution. In the later days, we had enough disparate data sources that we built oracles to choose which of the conflicting data was most likely to be correct. For example, we integrated with a data source that used OCR to scan container numbers as they passed through various way points while they were on trains. The tech wasn't perfect. We frequently got reports from the rail data source that a train was, for example, passing through the middle of the country when we knew with 100% certainty that it was currently in the middle of the pacific ocean on a boat. That spurious data could be safely thrown out on logical grounds. Other cases were not as straightforward!
perrygeo 2y ago
> never target automation of 100% of incoming workload... Iteratively approach it. This is critical. So many people jump from "it works" to "let's automate it entirely" without understanding all the nuances, all the various conditions and intermediate states that exist in the real world. The result is a brittle system that is unreliable and requires constant manual intervention. A better approach is to semi-automate things first. Write scripts with manual QA checkpoints and playbooks that get executed directly by devs. Do the thing yourself until it's so boring it hurts. Then automate it.
carlmr 2y ago
I also like the way the Toyota production system puts it with Jidoka / automate with a human touch. [1] 1. Only automate the steps you know how to execute manually very well. This is a prerequisite. This kind of goes in the direction of not automating everything but looking at the parts you can automate well first. 2. Enable human intervention in the automation. Enable humans to stop the line when something is wrong. Then you figure out what's wrong and improve your automation. This implies the interative approach. [1] https://global.toyota/en/company/vision-and-philosophy/production-system/ https://global.toyota/en/company/vision-and-philosophy/produ...
- MichaelZuo 2y ago
  Or to put it more concisely, trust but verify the ‘manual process’, that includes any verbal or written explanations of the process. Only 100% trust the automation after every possible step, procedure, and word has been 100% verified. (Which is to say almost never…)
screye 2y ago
This is genius. I skimmed it the first time, and it took me a good 30 minutes to appreciate the wide applicability of your insight. A whole family of problems across domains : supply chains, manufacturing lines, AI pipelines, resilient software, physical security, etc. come down to effective state limitation. In my day job, the vast bulk of my arguments with PMs come down to a lack of planning allocation for post-launch code cleanup. I haven't been able to find a succinct articulation for the utility of 'code cleanup' to a wider audience. 'State-limitation' fits nicely. Exactly the concept I was looking for. It draws from the better known 'less-is-more' adage, but adds a lot of implicit detail to the generic (almost cliche) adage.
- jalapenos 2y ago
  To be honest, for most low or moderately performing organisations, the best technique is just to not talk about it and just do it. So long as it's done silently, blended in with other things, and cloaked under clever wording (e.g. "this blocks that other thing you want" rather than "this will improve the codebase"), things will go quite well. As soon as you speak to them as you would another engineer, you provide them material to use against you in prevention of you taking proper action.
  - screye 2y ago
    That only works if the whole team coordinates. If one person writes broken code in half the time, while you take twice as much cleaning the mess.....then you're going to be perceived as ineffective.
- achillesheels 2y ago
  “A whole family of problems across domains…come down to effective state limitation.” A fancy way of saying, “simplicity is the mark of truth.” Or “Less mechanical points of failure the better.” I concur. Eliminate design risk.
airbreather 2y ago
Implicit state is the enemy.

by ethbr1 2y ago

HatchedLake721 2y ago

With your experience, anything you'd recommend to read in the process automation space? (I'm a founder of an automation SaaS where we've made "human interface" one of the core features of the product)

trod123 2y ago
I'm not aware of any books that have covered this appropriately. The biggest part of automation in my experience is boiling down the inputs to a 'unique' state that automation can then use as inputs and be run on. For computation to do work, it requires a property of consistency, and computers can only operate accurately and do work when properties of determinism are met. Also as the OP mentions, state can explode leading to sphagetti, this is why he mentions a sieve like approach based on similarity. Some problem spaces can be fundamentally inconsistent, such as with some approximations (common methods used for such), which falls back to what amounts to guesses, heuristics, and checks in terms of exception handling. There are problem scopes that cannot be characterized too so no amount of exception handling will resolve the entire scope, which is why you need fallbacks in a resilient design. If inputs cannot be controlled and uniquely differentiated, the automation fails in brittle ways, especially with regards to external change. The main interface (with regards to your core features) would be language, or communication. There exists words right now that can have contradictory, and different meanings, where the same word may mean the opposite depending on context, and this is not a general consensus but an individual one (where the individual may be misusing it). That breaks the 1:1 mapping required for determinism, and AI weights mimicking neurons have a narrow approximation where it may work under a narrow set of circumstances but no computer today can differentiate when the inputs are the same but have two or more, different states mixed in (many people forget that absence of a state is a state too) and then decompose them. Abstract decomposition seems to be something only humans are good at, and I'm glad this is the case otherwise none of us would have jobs.

bloopernova 2y ago

> Well-designed automation should look like a funnel, rather than a subway map. Do you have any examples of this? Like an example project? > never target automation of 100% of incoming workload. Ever. A new product owner came in last year and wanted to "automate everything". They wanted a web page where builds, branches, commits, and more were all on one page that showed exactly which commit was deployed where, by whom, etc etc. They wanted this extravaganza for a 2-person application that was in maintenance with no new features. They're also the kind of person who consults chatgpt or copilot on a subject while you're explaining that subject to them, to check that the LLM agrees with what you are saying. They'll even challenge people to prove some copilot output is incorrect. It seems to me that they consider LLMs more reliable than people.

shakna 2y ago
> They wanted a web page where builds, branches, commits, and more were all on one page that showed exactly which commit was deployed where, by whom, etc etc. Sounds like they would have been a fan of ungit [0], which I have seen used for that kind of flow overview, though it has looked more impressive than actually proved helpful in my experience. [0] https://github.com/FredrikNoren/ungit https://github.com/FredrikNoren/ungit
chrsig 2y ago
> A new product owner came in last year and wanted to "automate everything". They wanted a web page where builds, branches, commits, and more were all on one page that showed exactly which commit was deployed where, by whom, etc etc. They wanted this extravaganza for a 2-person application that was in maintenance with no new features. you know, this seems very reasonable until that last sentence.
- ykonstant 2y ago
  Why a web page and not directly the git log? If style is necessary, reformat the log data with some fancy ASCII art?
  - 2y ago

chrsig 2y ago

I wouldn't consider my career being in process automation, but I feel like you just described my approach to managing new product development on top of existing machinery while ensuring no breakage of what exists.

InDubioProRubio 2y ago

or: https://fsharpforfunandprofit.com/rop/ https://fsharpforfunandprofit.com/rop/

davedx 2y ago

This has a corollary in software and refactoring: don’t refactor duplicated code too early; keep it separate until it’s very clear the refactoring is correct

seanlinehan 2y ago

Spot on. I was an early eng and first VP of Product at Flexport. Global logistics is inherently complicated and involves coordinating many disparate parties. To complete any step in the workflow, you're generally taking in input data from a bunch of different companies, each of which have varying formats and quality of data. A very challenging context if your goal is process automation. The only way to make progress was exactly the way you described. At each step of the workflow, you need to design at least 2 potential resolution pathways: 1. Automated 2. Manual For the manual case, you have to actually build the interfaces for an operator to do the manual work and encode the results of their work as either: 1. Input into the automated step 2. Or, in the same format as the output of the automated case In either case, this is precisely aligned with your "reuinifying divergent paths" framing. In the automated case, you actually may wind up with N different automation pathways for each workflow step. For an example at Flexport: if we needed to ingest some information from an ocean carrier, we often had to build custom processors for each of the big carriers. And if the volume with a given trading partner didn't justify that investment, then it went to the manual case. From the software engineering framing, it's not that different from building a micro-services architecture. You encapsulate complexity and expose standard inputs and outputs. This avoids creating an incomprehensible mess and also allows the work to be subdivided for individual teams to solve. All that said – doing this in practice at a scaling organization is tough. The micro-services framing is hard to explain to people who haven't internalized the message. But yeah, 100% automation is a wild-goose chase. Maybe you eventually get it, maybe not. But you have to start with the assumption that you won't or you never will.

initplus 2y ago
Sounds like a really interesting problem space. I'm curious if you have any comments about how you approached dealing with inconsistencies between information sources? System A says X, system B says Y. I suppose best approach is again just to bail out to manual resolution?
- seanlinehan 2y ago
  In the early days, we bailed out to manual resolution. In the later days, we had enough disparate data sources that we built oracles to choose which of the conflicting data was most likely to be correct. For example, we integrated with a data source that used OCR to scan container numbers as they passed through various way points while they were on trains. The tech wasn't perfect. We frequently got reports from the rail data source that a train was, for example, passing through the middle of the country when we knew with 100% certainty that it was currently in the middle of the pacific ocean on a boat. That spurious data could be safely thrown out on logical grounds. Other cases were not as straightforward!

perrygeo 2y ago

> never target automation of 100% of incoming workload... Iteratively approach it. This is critical. So many people jump from "it works" to "let's automate it entirely" without understanding all the nuances, all the various conditions and intermediate states that exist in the real world. The result is a brittle system that is unreliable and requires constant manual intervention. A better approach is to semi-automate things first. Write scripts with manual QA checkpoints and playbooks that get executed directly by devs. Do the thing yourself until it's so boring it hurts. Then automate it.

carlmr 2y ago

I also like the way the Toyota production system puts it with Jidoka / automate with a human touch. [1] 1. Only automate the steps you know how to execute manually very well. This is a prerequisite. This kind of goes in the direction of not automating everything but looking at the parts you can automate well first. 2. Enable human intervention in the automation. Enable humans to stop the line when something is wrong. Then you figure out what's wrong and improve your automation. This implies the interative approach. [1] https://global.toyota/en/company/vision-and-philosophy/production-system/ https://global.toyota/en/company/vision-and-philosophy/produ...

MichaelZuo 2y ago
Or to put it more concisely, trust but verify the ‘manual process’, that includes any verbal or written explanations of the process. Only 100% trust the automation after every possible step, procedure, and word has been 100% verified. (Which is to say almost never…)

screye 2y ago

This is genius. I skimmed it the first time, and it took me a good 30 minutes to appreciate the wide applicability of your insight. A whole family of problems across domains : supply chains, manufacturing lines, AI pipelines, resilient software, physical security, etc. come down to effective state limitation. In my day job, the vast bulk of my arguments with PMs come down to a lack of planning allocation for post-launch code cleanup. I haven't been able to find a succinct articulation for the utility of 'code cleanup' to a wider audience. 'State-limitation' fits nicely. Exactly the concept I was looking for. It draws from the better known 'less-is-more' adage, but adds a lot of implicit detail to the generic (almost cliche) adage.

jalapenos 2y ago
To be honest, for most low or moderately performing organisations, the best technique is just to not talk about it and just do it. So long as it's done silently, blended in with other things, and cloaked under clever wording (e.g. "this blocks that other thing you want" rather than "this will improve the codebase"), things will go quite well. As soon as you speak to them as you would another engineer, you provide them material to use against you in prevention of you taking proper action.
- screye 2y ago
  That only works if the whole team coordinates. If one person writes broken code in half the time, while you take twice as much cleaning the mess.....then you're going to be perceived as ineffective.
achillesheels 2y ago
“A whole family of problems across domains…come down to effective state limitation.” A fancy way of saying, “simplicity is the mark of truth.” Or “Less mechanical points of failure the better.” I concur. Eliminate design risk.

airbreather 2y ago

Implicit state is the enemy.