6 ms·

The plan-execute pattern

154 points by surprisetalk 2y ago

PaulHoule 2y ago
There are a wide range of “patterns” that derive from compilers that are the higher teachings behind functional programming and OO patterns. People don’t usually call them patterns. (Scheme isn’t special because it has closures, it is special because you can write functions that write functions.) Reminds me of a thread a few days back when someone was talking about state machines as if they were obscure. At a systems programming level they aren’t obscure at all (how do you write non-blocking network code without async/await? how does regex and lexers work?). Application programmers are usually “parsing” with regex.match() or index() or split() and frequently run into brick walls when it gets complex (say parsing email headers that can span several lines) but state machines make it easy.
- Spivak 2y ago
  > Scheme isn’t special because it has closures, it is special because you can write functions that write functions. I swear this informational parasite will never die, any language with eval can write functions that write functions. What makes Scheme and other homoiconic languages special is that the code that is currently executing can be manipulated live like any other data structure and it works.
  - hollerith 2y ago
    >What makes Scheme and other homoiconic languages special is that the code that is currently executing can be manipulated live like any other data structure I think I know Scheme well, and I don't catch your meaning. Most Scheme implementations are compilers that output a machine-language executable. How can this executable be "manipulated live" in a way that differs from an executable produced by any other language? What am I not seeing?
    - Spivak 2y ago
      At the level where you're executing assembly every language is the same. Which is a way of saying that assembly is homoiconic. And even in C you're more than capable of W&Xing your binary to change behavior at runtime but you're doing it at a level below the language. It's the (not completely unrestricted) self-modifying bit that's cool and unique. You can go into an existing function and change what it does by manipulating or outright replacing its code. Other languages you end up having to evaluate and re-point/redefine the reference and it gets you where you need to go but not nearly as cool. For folks that liked Dijkstra's whole structured programming thing but didn't want to give up the power entirely.
      - PaulHoule 2y ago
        How about monkey patching in languages like Javascript and Python? I'd also say "homoiconic" is a less important concept than people would think. A while back I was writing a lodash-like library for Java because I don't like the Streams API (though in the end I just used the collectors from the Stream API because they were easy to fit into my system) One thing I found frustrating about that was that I didn't want to write 8 different functions for the 8 primitive types (never mind 64 functions for the direct product of the primitive types) and conceived of a code generator that would let you write Java-in-Java with a somewhat Lispy looking API that builds up something like S-expressions with typed static imports, here is some sample code https://github.com/paulhoule/ferocity/blob/main/ferocity0/src/test/java/com/ontology2/ferocity/TestUrFierce.java https://github.com/paulhoule/ferocity/blob/main/ferocity0/sr... The plan was to write a ferocity0 sufficient to write a code generator that could generate stubs for the standard library plus a persistent collection library, then write most of ferocity1 in ferocity0 and maybe write a ferocity2 in ferocity1 and ferocity2 would be the master tool for code generation in Java. Look how nice the stub generator is https://github.com/paulhoule/ferocity/blob/main/ferocity0/src/main/java/com/ontology2/ferocity/WrapperGenerator.java https://github.com/paulhoule/ferocity/blob/main/ferocity0/sr... Now many people would think this combines all the drawbacks of Common Lisp and Java and they might be right. On the other hand, I think if you're aggressive in organizing your code and using code generation you might cut your code volume enough to make up for the bulking that happens when you write Java-in-Java. The stubs to write Java-in-Java are a huge amount of code but they're 99% generated and once you have Java-in-Java you can keep recursing and messing around with quote and eval and do all the stuff you'd do in a homoiconic language except you are having to write code in a DSL.
  - PaulHoule 2y ago
    Back in the 1980s I found a book at the public library where somebody had a BASIC program that would read a BASIC program without line numbers and with structured programming loops and rewrite it an ordinary BASIC. It was oriented towards CP/M so I typed the program in and modified it to run on my TRS-80 Coco. It was kinda like https://en.wikipedia.org/wiki/Ratfor https://en.wikipedia.org/wiki/Ratfor Similarly I've written awk programs that write shell scripts (guilty pleasure), Python programs that write AVR-8 assembly, toolkits to write Java ASTs with static imports like @Test public void buildAList() { var stream = callBuild(callAdd(callAdd(callAdd(callBuilder(), Literal.of(7)), Literal.of(11)),Literal.of(15))); var expr = callToList(stream); var l = expr.evaluateRT(); assertEquals(3,l.size()); assertEquals(7, l.get(0)); assertEquals(11, l.get(1)); assertEquals(15, l.get(2)); } and evaluate with a tree-walking interpreter or use as a code generator in maven, do transformations on the ASTs, etc. There are many paths to metaprogramming but the Lisp family leads in putting it on your fingertips. Once you learn those methods you can apply them in the trashiest programming languages, but I think often people miss the point.
    - touisteur 2y ago
      Very common for me to write shell scripts that generate shell scripts to run. Instead of creating gnuplot scripts to loaded, sometimes a good sequence of echo "set terminal png 800,600; plot..." | gnuplot > my.png or a very long imagemagick drawing command or a simple svg generated then converted to png then assembled (thank montage), can do the job. The whole advantage of generating intermediate shell scripts, is to replay one part specifically to debug a small part of a script pipeline.
  - andai 2y ago
    >code that is currently executing can be manipulated live like any other data structure and it works. I heard from someone who did this with x86(-64?) assembly in production code, but I don't know the details.
    - deleted 2y ago
      [deleted]
  - lispm 2y ago
    > What makes Scheme and other homoiconic languages special is that the code that is currently executing can be manipulated live like any other data structure and it works. That would be true for source level interpreters, but not for compiled implementations.
  - kazinator 2y ago
    No, Scheme does not make self-modifying code defined. See this example in R7RS for the set-car! function: (define (f) (list 'not-a-constant-list)) (define (g) '(constant-list)) (set-car! (f) 3) ⟹ unspecified (set-car! (g) 3) ⟹ error It is an error in Scheme to modify a constant list. The implementation is not required to diagnose it (an exception is not required), which basically means that it's undefined behavior. In Common Lisp, it is likewise undefined behavior. Running Lisp code is not necessarily still made of nested lists. That may be the case if it is interpreted, but Lisp code is often compiled. Modifying the original nested lists that comprise the source code will do nothing to the object code, except if those parts are modified that serve as literals, which the compiled code may be sharing with the source code. In short, your understanding is wrong. When Lisp people talking about modifying the running program, that just refers to redefining functions and other entities. Common Lisp has strong support for redefining classes of which there are existing instances. Existing objects can be migrated to new representations in the redefined classes. None of that is related to homoiconicity or related topics.
- lispm 2y ago
  > Scheme isn’t special because it has closures, it is special because you can write functions that write functions. When Scheme was developed, it was special, because it used lexical binding and provided lexical closures. That was new. Writing functions that write functions would not have been new, since LISP had that already -> including Maclisp, which was used for the first SCHEME implementation. Later Scheme got used for exploring different macro systems.
- deleted 2y ago
  [deleted]
skrebbel 2y ago
I love this pattern. I agree with the author that design patterns that only exist because, say, your programming languages doesn't have function types, aren’t worth writing a book on. This pattern seems like a specific application of the general idea to favor data structures over logic. If you manage to make a clear datastructure that encapsulates all the different ways in which your data/users/etc might behave, the implementation of the logic is often very straight forward. Conversely, if your data structure is either too generic or too specific, you end up having to code a lot of special cases, exceptions, mutable state and so on in the logic to deal with that.
advael 2y ago
I spent the first bit of this article going "Wait isn't that just what a finite state machine is?" and then was subsequently super satisfied by the next bit where they say that this is indeed a really good way to handle the general case Reducing program logic to a small state machine and business logic that interacts with it in predefined ways is a pattern I learned while doing asynchronous netcode for a game project, but gradually realized was applicable in a ton of places, and framing it as a robust version of a "plan-execute" pattern is really intuitive but also really powerful. Great article
- scotty79 2y ago
  It's quite sad that generał purpose language designers are not yet at the stage of daring to provide any convenient syntax for high level concepts like state machines or entity component systems.
  - rdtsc 2y ago
    At least in Erlang there is a standard state machine built-in module: https://www.erlang.org/doc/system/statem.html https://www.erlang.org/doc/system/statem.html Most of TLS support in the built-in library is implemented using gen_statem, on top of the lower level crypto primitives provided by OpenSSL, same for SSH support. Others have built Raft using it. It's quite nice and I appreciate that it comes as a standard module.
  - advael 2y ago
    For both cases I think it's less that there isn't convenient syntax for it, and more that the generality of the syntax doesn't restrict you enough to enforce the pattern. For example, you can do a decent job approximating a monadic state machine system in C++14 with just optional and structs, so long as you follow the rule that all mutable state is kept in a state struct and all business logic must be in the form of functions with the signature optional<state>(const state&). Hell, you could even remove the optional if you don't consider there to be any "failure" cases, just cases that don't mutate the state Then, you can use any container you like to keep track of your valid state transitions and not even have to use polymorphism. But of course C++ gives you many ways in which to trivially break this nice system of guarantees, including mutable lambdas, or even just implicit reliance on fixed memory addresses via explicit allocation and pointers in an external scope. This pattern is very general, so it's hard to capture in a guardrail
gfaure 2y ago
I've also successfully used this in production — another side effect is that you can inspect the exact information that each step is using to compute its own output, if you ensure that the output plan is a pure function of the input plan.
rmckayfleming 2y ago
"I feel uneasy about design patterns. On the one hand, my university class on design patterns revived my interest in programming. On the other hand, I find most patterns in the Gang of Four book to be irrelevant to my daily work; they solve problems that a choice of programming language or paradigm creates." My relationship with design patterns changed when I stopped viewing them as prescriptive and started viewing them as descriptive.
- codeflo 2y ago
  Descriptive of the kinds of workarounds that a lack of proper abstraction capabilities made necessary in ancient Java.
  - layer8 2y ago
    The GoF book uses C++ and was written before the first version of Java existed. Furthermore, the purpose of design patterns is not to introduce anything new, it’s to provide a catalog of repeatedly occurring existing patterns in software development, including the respective situations each pattern is applicable in, and their individual benefits and drawbacks. An important motivation was to name each pattern, so that when one software engineer says “builder”, the other engineers know what it means. Previously developers encountered certain patterns sooner or later, but didn’t know what to call each pattern, or each developer invented their own names, or meant different things by the same name. The set of relevant patterns will change over time to some extent as software development changes, but that is orthogonal to the concept of design patterns. It’s a mistake to think of design patterns as specifically the set of 23 GoF patterns. Though some GoF patterns are unlikely to ever become irrelevant, like Adapter, Decorator, Composite, Proxy, Interpreter, Strategy.
  - groby_b 2y ago
    Or any other major commercially used language right now? Look, I'm glad you work in a language where you don't need to describe those things any more, but the vast majority of people still do. And for that group of people, understanding that "Design Patterns" is a dictionary, not a recipe book, is a really important thing.
BoppreH 2y ago
This pattern has several other advantages that are worth mentioning: - You can attach the plan to your Change Management, both for documentation and approval. - You can manually edit the plan. - The plan can be processed by other tools (linting, optimizations). - The "complexity" of the pair planner+executor is often lower than a monolithic algorithm (above a certain size). - Having a side-effect-free planner makes testing so much easier. The big downside, of course, is that you're building a VM. You'll have to design operations, parameters, encoding. And stacktraces will make much less sense. My rule of thumb is to use plan-execute for scripts that combine business logic with destructive operations or long runtime.
JohnMakin 2y ago
This is a similar pattern to Terraform, correct? (Plan/Apply). I like this pattern, however, when implementing it into automation it presents a conundrum. Often, code will be checked into a repository with a "plan" attached to it. Great. Merge after plan is approved, then automation executes the plan. However, lots of providers will only show errors on the "execute" phase (e.g., on apply, "resource already exists" errors, etc.), leading to a ridiculously slow feedback loop of having to open new PR's, go through all approval chains again, merge, execute plan again. Debugging a simple issue like a misnamed variable can take days this way. I have yet to see a graceful work flow here in a production environment and would love to hear about one. (No I will not use terraform cloud)
- DougBTX 2y ago
  I’d recommend testing the Apply step in a staging environment before rolling out. For releases, the plans don’t need to be committed to version control. Instead they can be generated in CI just before release, then put reviewing the plan on the checklist before confirming the release to prod.
- rnubel 2y ago
  It's more manageable to have automation (i.e., Atlantis) do the apply on the open PR and then only merge if it succeeds. You have to rely on the locking feature of Atlantis to avoid conflicts, but it's worked pretty well at scale at Enova (~50 Terraform changes a day).
- deleted 2y ago
  [deleted]
bradleybuda 2y ago
I use this pattern a lot (and I often find myself wishing I used it more). Another way to think about this is "noun-ification" (aka the Command pattern) - instead of invoking a function directly, capture all of its arguments, serialize them to some "store", then provide the ability to rehydrate them and actually run it.
- caseyohara 2y ago
  Interesting you used the term hydrate because I only hear that used when talking about front-end web development (specifically Next and React) and it's a relatively new term. The "Hydration (web development)" Wikipedia article was only created in December 2020. https://en.wikipedia.org/wiki/Hydration_(web_development) https://en.wikipedia.org/wiki/Hydration_(web_development) I feel like serialize/deserialize or marshal/demarshal are more common in broader computer science contexts.
  - stevesimmons 2y ago
    "Hydrate" is much older than 2020. Here's a StackOverflow reference from 2011 [1], for instance. I first heard it used around that time, for loading objects from databases (in a banking context). [1] https://stackoverflow.com/a/4929478 https://stackoverflow.com/a/4929478
- munificent 2y ago
  I came here to say basically the same thing. To me, this reads like an example of the Command pattern, which is one of my favorite, most-used patterns. The idea of taking some chunk of imperative behavior and reifying it into an object that can perform that behavior is very powerful: * It allows you to attach other metadata to the object so that code can ask questions about it: "Do you perform IO?" "Can you be reverted?" It lets surrounding code introspect over operations in useful ways. * It allows you to separate the code that creates the command from the code that performs it (what this blog post is about). This can nicely decouple "strategy"-type code from "execution harness" type code. Both of those are often fairly complex in their own right, so a pattern that lets you build a nice abstraction layer between them can be very helpful. * It allows the command object to support multiple related operations. The most common example, which I deeply love, is implementing undo. A Command object for some UI operation can also support a separate method to undo the same operation. Your undo stack is then just a list of these Command objects.
tomberek 2y ago
This is an excellent pattern. Nix uses it to allow one to create a full transitive build graph, serialized that plan, move it around, and execute it in a distributed and reproducible fashion. There are always temptations to loosen the constraints this imposes, but they nearly always come at a cost of undermining the value of the pattern.
Vadim_samokhin 2y ago
I think postgres really excelled at this pattern. As well as other databases implementing a query planner.
jkaptur 2y ago
> I feel uneasy about design patterns. ... they solve problems that a choice of programming language or paradigm creates. I don't understand the ambivalence. I have to make some choice of language and paradigm, and then I have to solve problems using it. Great article and description of the plan-execute pattern, though!
onetimeuse92304 2y ago
Used something very similar many times in my past without knowing it is formalised as a pattern. For example, one application of this was a long migration project where a large collection of files (some petabytes of data) was to be migrated from an on-prem NAS to cloud filesystem. The files on NAS were managed with additional asset management solution which stored metadata in a PostgreSQL (actual filenames, etc.) The application I wrote was composed of a series of small tools. One tool would contact all of the sources of information and create a file with a series of commands (copy file from location A to location B, create a folder, set metadata on a file, etc.) Other tools could take that large file and run operations on it. Split it into smaller pieces, prioritise specific folders, filter out modified files by date, calculate fingerprints from actual file data for deduplication, etc. These tools would just operate on this common file format without actually doing any operations on files. And finally tool that could be instantiated somewhere and execute a plan. I designed it all this way as it was much more resilient process. I could have many different processes running at the same times and I had a common format (a file with a collection of directives) as a way to communicate between all these different tools.
pshirshov 2y ago
The correct name is "staged generative programming". The best form for plans is DAG w/o control flow. You may run perception->planning->execution in a loop.
ijustlovemath 2y ago
I do this for a code generation tool we use internally! Plans contain all the metadata to actually do a thing, but allow transformation, filtering etc before execution. It also forces you to abstract your code in a cleaner way, and minimize the surface of your plan generation/execution functions, which makes them infinitely more testable. The way I think about when to use a plan is whenever you're doing batches of I/O of some kind, or anything that you might want to make idempotent.
evmar 2y ago
[ninja author] I like how their example is a build system, given Ninja constructs an object literally called "Plan"[1]. However, when I later revisited the design of Ninja I found that removing the separate planning phase better reflected the dynamic nature of how builds end up working out, where you discover information during the build that impacts what work you have planned. See the "Single pass" discussion in my design notes[2], which includes some negatives of the change. If the author reads this, in [2] I made a Rust-based Ninja rewrite that has some similar data structures to those in the blog post, you might find it interesting! [1] https://github.com/ninja-build/ninja/blob/dcefb838534a56b2621baf33c7e0583ff63ee35f/src/build.h#L38 https://github.com/ninja-build/ninja/blob/dcefb838534a56b262... [2] https://neugierig.org/software/blog/2022/03/n2.html https://neugierig.org/software/blog/2022/03/n2.html
- klysm 2y ago
  Indeed plans get hard when the plan has to be dynamic. Pulumi and terraform have similar troubles.
  - roenxi 2y ago
    Cast that thought in terms of state. If all the state can be represented up front then plans are great. But if the understanding of the problem changes half way through a project and previously un-identified state is found during the runtime then a plan-execute pattern will get ugly quickly. It is good to name things, but the challenge to manage isn't in the implementing of a plan-execute architecture. It is in identifying the threshold where there is enough runtime state in the problem domain that a plan-execute architecture can't be sustained.
WhyNotHugo 2y ago
I really like this pattern and have been using it in the rewrite for vdirsyncer. One nice advantage is that an application can show me the whole plan for me to review before executing it. This is basically what terraform did too. This pattern also makes coding and testing much much easier.
CGamesPlay 2y ago
A key motivator of this pattern is identifying invalid configurations early, and in particular preventing execution if a known invalid configuration is requested (e.g. fail-fast). For example, if you delete a resource from a configuration, and it ends up being one that is depended upon by another, a "plan-execute" pattern should identify this at the start and prevent any execution from happening.
krick 2y ago
I don't see how that is a pattern. Rather, it's a new name to an old thing: writing a virtual machine or an JIT-compiler. And no, interpreter is not a "special case of plan-execute pattern": by definition, if data structure can be run, it means it is a program for some virtual machine. And "a plan" is a lousy definition of "something that can be run", i.e. a program. So it's not a "special case", it's synonymous. That also explains why it isn't a good and useful "pattern": inventing new DSLs with custom compilers all the time is not a trivial thing to pull off without shooting yourself in the leg, so if are writing a VM (e.g. an RDBMS query planner) you must be already perfectly aware that you are writing a fucking VM with its own programming language (e.g. SQL). And if you are not, inventing a data structure that encapsulates a whole program is probably a bigger task than the problem you are actually trying to solve.

154 points by surprisetalk 2y ago

There are a wide range of “patterns” that derive from compilers that are the higher teachings behind functional programming and OO patterns. People don’t usually call them patterns. (Scheme isn’t special because it has closures, it is special because you can write functions that write functions.) Reminds me of a thread a few days back when someone was talking about state machines as if they were obscure. At a systems programming level they aren’t obscure at all (how do you write non-blocking network code without async/await? how does regex and lexers work?). Application programmers are usually “parsing” with regex.match() or index() or split() and frequently run into brick walls when it gets complex (say parsing email headers that can span several lines) but state machines make it easy.

Spivak 2y ago
> Scheme isn’t special because it has closures, it is special because you can write functions that write functions. I swear this informational parasite will never die, any language with eval can write functions that write functions. What makes Scheme and other homoiconic languages special is that the code that is currently executing can be manipulated live like any other data structure and it works.
- hollerith 2y ago
  >What makes Scheme and other homoiconic languages special is that the code that is currently executing can be manipulated live like any other data structure I think I know Scheme well, and I don't catch your meaning. Most Scheme implementations are compilers that output a machine-language executable. How can this executable be "manipulated live" in a way that differs from an executable produced by any other language? What am I not seeing?
  - Spivak 2y ago
    At the level where you're executing assembly every language is the same. Which is a way of saying that assembly is homoiconic. And even in C you're more than capable of W&Xing your binary to change behavior at runtime but you're doing it at a level below the language. It's the (not completely unrestricted) self-modifying bit that's cool and unique. You can go into an existing function and change what it does by manipulating or outright replacing its code. Other languages you end up having to evaluate and re-point/redefine the reference and it gets you where you need to go but not nearly as cool. For folks that liked Dijkstra's whole structured programming thing but didn't want to give up the power entirely.
    - PaulHoule 2y ago
      How about monkey patching in languages like Javascript and Python? I'd also say "homoiconic" is a less important concept than people would think. A while back I was writing a lodash-like library for Java because I don't like the Streams API (though in the end I just used the collectors from the Stream API because they were easy to fit into my system) One thing I found frustrating about that was that I didn't want to write 8 different functions for the 8 primitive types (never mind 64 functions for the direct product of the primitive types) and conceived of a code generator that would let you write Java-in-Java with a somewhat Lispy looking API that builds up something like S-expressions with typed static imports, here is some sample code https://github.com/paulhoule/ferocity/blob/main/ferocity0/src/test/java/com/ontology2/ferocity/TestUrFierce.java https://github.com/paulhoule/ferocity/blob/main/ferocity0/sr... The plan was to write a ferocity0 sufficient to write a code generator that could generate stubs for the standard library plus a persistent collection library, then write most of ferocity1 in ferocity0 and maybe write a ferocity2 in ferocity1 and ferocity2 would be the master tool for code generation in Java. Look how nice the stub generator is https://github.com/paulhoule/ferocity/blob/main/ferocity0/src/main/java/com/ontology2/ferocity/WrapperGenerator.java https://github.com/paulhoule/ferocity/blob/main/ferocity0/sr... Now many people would think this combines all the drawbacks of Common Lisp and Java and they might be right. On the other hand, I think if you're aggressive in organizing your code and using code generation you might cut your code volume enough to make up for the bulking that happens when you write Java-in-Java. The stubs to write Java-in-Java are a huge amount of code but they're 99% generated and once you have Java-in-Java you can keep recursing and messing around with quote and eval and do all the stuff you'd do in a homoiconic language except you are having to write code in a DSL.
- PaulHoule 2y ago
  Back in the 1980s I found a book at the public library where somebody had a BASIC program that would read a BASIC program without line numbers and with structured programming loops and rewrite it an ordinary BASIC. It was oriented towards CP/M so I typed the program in and modified it to run on my TRS-80 Coco. It was kinda like https://en.wikipedia.org/wiki/Ratfor https://en.wikipedia.org/wiki/Ratfor Similarly I've written awk programs that write shell scripts (guilty pleasure), Python programs that write AVR-8 assembly, toolkits to write Java ASTs with static imports like @Test public void buildAList() { var stream = callBuild(callAdd(callAdd(callAdd(callBuilder(), Literal.of(7)), Literal.of(11)),Literal.of(15))); var expr = callToList(stream); var l = expr.evaluateRT(); assertEquals(3,l.size()); assertEquals(7, l.get(0)); assertEquals(11, l.get(1)); assertEquals(15, l.get(2)); } and evaluate with a tree-walking interpreter or use as a code generator in maven, do transformations on the ASTs, etc. There are many paths to metaprogramming but the Lisp family leads in putting it on your fingertips. Once you learn those methods you can apply them in the trashiest programming languages, but I think often people miss the point.
  - touisteur 2y ago
    Very common for me to write shell scripts that generate shell scripts to run. Instead of creating gnuplot scripts to loaded, sometimes a good sequence of echo "set terminal png 800,600; plot..." | gnuplot > my.png or a very long imagemagick drawing command or a simple svg generated then converted to png then assembled (thank montage), can do the job. The whole advantage of generating intermediate shell scripts, is to replay one part specifically to debug a small part of a script pipeline.
- andai 2y ago
  >code that is currently executing can be manipulated live like any other data structure and it works. I heard from someone who did this with x86(-64?) assembly in production code, but I don't know the details.
  - deleted 2y ago
    [deleted]
- lispm 2y ago
  > What makes Scheme and other homoiconic languages special is that the code that is currently executing can be manipulated live like any other data structure and it works. That would be true for source level interpreters, but not for compiled implementations.
- kazinator 2y ago
  No, Scheme does not make self-modifying code defined. See this example in R7RS for the set-car! function: (define (f) (list 'not-a-constant-list)) (define (g) '(constant-list)) (set-car! (f) 3) ⟹ unspecified (set-car! (g) 3) ⟹ error It is an error in Scheme to modify a constant list. The implementation is not required to diagnose it (an exception is not required), which basically means that it's undefined behavior. In Common Lisp, it is likewise undefined behavior. Running Lisp code is not necessarily still made of nested lists. That may be the case if it is interpreted, but Lisp code is often compiled. Modifying the original nested lists that comprise the source code will do nothing to the object code, except if those parts are modified that serve as literals, which the compiled code may be sharing with the source code. In short, your understanding is wrong. When Lisp people talking about modifying the running program, that just refers to redefining functions and other entities. Common Lisp has strong support for redefining classes of which there are existing instances. Existing objects can be migrated to new representations in the redefined classes. None of that is related to homoiconicity or related topics.
lispm 2y ago
> Scheme isn’t special because it has closures, it is special because you can write functions that write functions. When Scheme was developed, it was special, because it used lexical binding and provided lexical closures. That was new. Writing functions that write functions would not have been new, since LISP had that already -> including Maclisp, which was used for the first SCHEME implementation. Later Scheme got used for exploring different macro systems.
deleted 2y ago
[deleted]

skrebbel 2y ago

I love this pattern. I agree with the author that design patterns that only exist because, say, your programming languages doesn't have function types, aren’t worth writing a book on. This pattern seems like a specific application of the general idea to favor data structures over logic. If you manage to make a clear datastructure that encapsulates all the different ways in which your data/users/etc might behave, the implementation of the logic is often very straight forward. Conversely, if your data structure is either too generic or too specific, you end up having to code a lot of special cases, exceptions, mutable state and so on in the logic to deal with that.

advael 2y ago

I spent the first bit of this article going "Wait isn't that just what a finite state machine is?" and then was subsequently super satisfied by the next bit where they say that this is indeed a really good way to handle the general case Reducing program logic to a small state machine and business logic that interacts with it in predefined ways is a pattern I learned while doing asynchronous netcode for a game project, but gradually realized was applicable in a ton of places, and framing it as a robust version of a "plan-execute" pattern is really intuitive but also really powerful. Great article

scotty79 2y ago
It's quite sad that generał purpose language designers are not yet at the stage of daring to provide any convenient syntax for high level concepts like state machines or entity component systems.
- rdtsc 2y ago
  At least in Erlang there is a standard state machine built-in module: https://www.erlang.org/doc/system/statem.html https://www.erlang.org/doc/system/statem.html Most of TLS support in the built-in library is implemented using gen_statem, on top of the lower level crypto primitives provided by OpenSSL, same for SSH support. Others have built Raft using it. It's quite nice and I appreciate that it comes as a standard module.
- advael 2y ago
  For both cases I think it's less that there isn't convenient syntax for it, and more that the generality of the syntax doesn't restrict you enough to enforce the pattern. For example, you can do a decent job approximating a monadic state machine system in C++14 with just optional and structs, so long as you follow the rule that all mutable state is kept in a state struct and all business logic must be in the form of functions with the signature optional<state>(const state&). Hell, you could even remove the optional if you don't consider there to be any "failure" cases, just cases that don't mutate the state Then, you can use any container you like to keep track of your valid state transitions and not even have to use polymorphism. But of course C++ gives you many ways in which to trivially break this nice system of guarantees, including mutable lambdas, or even just implicit reliance on fixed memory addresses via explicit allocation and pointers in an external scope. This pattern is very general, so it's hard to capture in a guardrail

gfaure 2y ago

I've also successfully used this in production — another side effect is that you can inspect the exact information that each step is using to compute its own output, if you ensure that the output plan is a pure function of the input plan.

rmckayfleming 2y ago

"I feel uneasy about design patterns. On the one hand, my university class on design patterns revived my interest in programming. On the other hand, I find most patterns in the Gang of Four book to be irrelevant to my daily work; they solve problems that a choice of programming language or paradigm creates." My relationship with design patterns changed when I stopped viewing them as prescriptive and started viewing them as descriptive.

codeflo 2y ago
Descriptive of the kinds of workarounds that a lack of proper abstraction capabilities made necessary in ancient Java.
- layer8 2y ago
  The GoF book uses C++ and was written before the first version of Java existed. Furthermore, the purpose of design patterns is not to introduce anything new, it’s to provide a catalog of repeatedly occurring existing patterns in software development, including the respective situations each pattern is applicable in, and their individual benefits and drawbacks. An important motivation was to name each pattern, so that when one software engineer says “builder”, the other engineers know what it means. Previously developers encountered certain patterns sooner or later, but didn’t know what to call each pattern, or each developer invented their own names, or meant different things by the same name. The set of relevant patterns will change over time to some extent as software development changes, but that is orthogonal to the concept of design patterns. It’s a mistake to think of design patterns as specifically the set of 23 GoF patterns. Though some GoF patterns are unlikely to ever become irrelevant, like Adapter, Decorator, Composite, Proxy, Interpreter, Strategy.
- groby_b 2y ago
  Or any other major commercially used language right now? Look, I'm glad you work in a language where you don't need to describe those things any more, but the vast majority of people still do. And for that group of people, understanding that "Design Patterns" is a dictionary, not a recipe book, is a really important thing.

BoppreH 2y ago

This pattern has several other advantages that are worth mentioning: - You can attach the plan to your Change Management, both for documentation and approval. - You can manually edit the plan. - The plan can be processed by other tools (linting, optimizations). - The "complexity" of the pair planner+executor is often lower than a monolithic algorithm (above a certain size). - Having a side-effect-free planner makes testing so much easier. The big downside, of course, is that you're building a VM. You'll have to design operations, parameters, encoding. And stacktraces will make much less sense. My rule of thumb is to use plan-execute for scripts that combine business logic with destructive operations or long runtime.

JohnMakin 2y ago

This is a similar pattern to Terraform, correct? (Plan/Apply). I like this pattern, however, when implementing it into automation it presents a conundrum. Often, code will be checked into a repository with a "plan" attached to it. Great. Merge after plan is approved, then automation executes the plan. However, lots of providers will only show errors on the "execute" phase (e.g., on apply, "resource already exists" errors, etc.), leading to a ridiculously slow feedback loop of having to open new PR's, go through all approval chains again, merge, execute plan again. Debugging a simple issue like a misnamed variable can take days this way. I have yet to see a graceful work flow here in a production environment and would love to hear about one. (No I will not use terraform cloud)

DougBTX 2y ago
I’d recommend testing the Apply step in a staging environment before rolling out. For releases, the plans don’t need to be committed to version control. Instead they can be generated in CI just before release, then put reviewing the plan on the checklist before confirming the release to prod.
rnubel 2y ago
It's more manageable to have automation (i.e., Atlantis) do the apply on the open PR and then only merge if it succeeds. You have to rely on the locking feature of Atlantis to avoid conflicts, but it's worked pretty well at scale at Enova (~50 Terraform changes a day).
deleted 2y ago
[deleted]

bradleybuda 2y ago

I use this pattern a lot (and I often find myself wishing I used it more). Another way to think about this is "noun-ification" (aka the Command pattern) - instead of invoking a function directly, capture all of its arguments, serialize them to some "store", then provide the ability to rehydrate them and actually run it.

caseyohara 2y ago
Interesting you used the term hydrate because I only hear that used when talking about front-end web development (specifically Next and React) and it's a relatively new term. The "Hydration (web development)" Wikipedia article was only created in December 2020. https://en.wikipedia.org/wiki/Hydration_(web_development) https://en.wikipedia.org/wiki/Hydration_(web_development) I feel like serialize/deserialize or marshal/demarshal are more common in broader computer science contexts.
- stevesimmons 2y ago
  "Hydrate" is much older than 2020. Here's a StackOverflow reference from 2011 [1], for instance. I first heard it used around that time, for loading objects from databases (in a banking context). [1] https://stackoverflow.com/a/4929478 https://stackoverflow.com/a/4929478
munificent 2y ago
I came here to say basically the same thing. To me, this reads like an example of the Command pattern, which is one of my favorite, most-used patterns. The idea of taking some chunk of imperative behavior and reifying it into an object that can perform that behavior is very powerful: * It allows you to attach other metadata to the object so that code can ask questions about it: "Do you perform IO?" "Can you be reverted?" It lets surrounding code introspect over operations in useful ways. * It allows you to separate the code that creates the command from the code that performs it (what this blog post is about). This can nicely decouple "strategy"-type code from "execution harness" type code. Both of those are often fairly complex in their own right, so a pattern that lets you build a nice abstraction layer between them can be very helpful. * It allows the command object to support multiple related operations. The most common example, which I deeply love, is implementing undo. A Command object for some UI operation can also support a separate method to undo the same operation. Your undo stack is then just a list of these Command objects.

tomberek 2y ago

This is an excellent pattern. Nix uses it to allow one to create a full transitive build graph, serialized that plan, move it around, and execute it in a distributed and reproducible fashion. There are always temptations to loosen the constraints this imposes, but they nearly always come at a cost of undermining the value of the pattern.

Vadim_samokhin 2y ago

I think postgres really excelled at this pattern. As well as other databases implementing a query planner.

jkaptur 2y ago

> I feel uneasy about design patterns. ... they solve problems that a choice of programming language or paradigm creates. I don't understand the ambivalence. I have to make some choice of language and paradigm, and then I have to solve problems using it. Great article and description of the plan-execute pattern, though!

onetimeuse92304 2y ago

Used something very similar many times in my past without knowing it is formalised as a pattern. For example, one application of this was a long migration project where a large collection of files (some petabytes of data) was to be migrated from an on-prem NAS to cloud filesystem. The files on NAS were managed with additional asset management solution which stored metadata in a PostgreSQL (actual filenames, etc.) The application I wrote was composed of a series of small tools. One tool would contact all of the sources of information and create a file with a series of commands (copy file from location A to location B, create a folder, set metadata on a file, etc.) Other tools could take that large file and run operations on it. Split it into smaller pieces, prioritise specific folders, filter out modified files by date, calculate fingerprints from actual file data for deduplication, etc. These tools would just operate on this common file format without actually doing any operations on files. And finally tool that could be instantiated somewhere and execute a plan. I designed it all this way as it was much more resilient process. I could have many different processes running at the same times and I had a common format (a file with a collection of directives) as a way to communicate between all these different tools.

pshirshov 2y ago

The correct name is "staged generative programming". The best form for plans is DAG w/o control flow. You may run perception->planning->execution in a loop.

ijustlovemath 2y ago

I do this for a code generation tool we use internally! Plans contain all the metadata to actually do a thing, but allow transformation, filtering etc before execution. It also forces you to abstract your code in a cleaner way, and minimize the surface of your plan generation/execution functions, which makes them infinitely more testable. The way I think about when to use a plan is whenever you're doing batches of I/O of some kind, or anything that you might want to make idempotent.

evmar 2y ago

[ninja author] I like how their example is a build system, given Ninja constructs an object literally called "Plan"[1]. However, when I later revisited the design of Ninja I found that removing the separate planning phase better reflected the dynamic nature of how builds end up working out, where you discover information during the build that impacts what work you have planned. See the "Single pass" discussion in my design notes[2], which includes some negatives of the change. If the author reads this, in [2] I made a Rust-based Ninja rewrite that has some similar data structures to those in the blog post, you might find it interesting! [1] https://github.com/ninja-build/ninja/blob/dcefb838534a56b2621baf33c7e0583ff63ee35f/src/build.h#L38 https://github.com/ninja-build/ninja/blob/dcefb838534a56b262... [2] https://neugierig.org/software/blog/2022/03/n2.html https://neugierig.org/software/blog/2022/03/n2.html

klysm 2y ago
Indeed plans get hard when the plan has to be dynamic. Pulumi and terraform have similar troubles.
- roenxi 2y ago
  Cast that thought in terms of state. If all the state can be represented up front then plans are great. But if the understanding of the problem changes half way through a project and previously un-identified state is found during the runtime then a plan-execute pattern will get ugly quickly. It is good to name things, but the challenge to manage isn't in the implementing of a plan-execute architecture. It is in identifying the threshold where there is enough runtime state in the problem domain that a plan-execute architecture can't be sustained.

WhyNotHugo 2y ago

I really like this pattern and have been using it in the rewrite for vdirsyncer. One nice advantage is that an application can show me the whole plan for me to review before executing it. This is basically what terraform did too. This pattern also makes coding and testing much much easier.

CGamesPlay 2y ago

A key motivator of this pattern is identifying invalid configurations early, and in particular preventing execution if a known invalid configuration is requested (e.g. fail-fast). For example, if you delete a resource from a configuration, and it ends up being one that is depended upon by another, a "plan-execute" pattern should identify this at the start and prevent any execution from happening.

krick 2y ago

I don't see how that is a pattern. Rather, it's a new name to an old thing: writing a virtual machine or an JIT-compiler. And no, interpreter is not a "special case of plan-execute pattern": by definition, if data structure can be run, it means it is a program for some virtual machine. And "a plan" is a lousy definition of "something that can be run", i.e. a program. So it's not a "special case", it's synonymous. That also explains why it isn't a good and useful "pattern": inventing new DSLs with custom compilers all the time is not a trivial thing to pull off without shooting yourself in the leg, so if are writing a VM (e.g. an RDBMS query planner) you must be already perfectly aware that you are writing a fucking VM with its own programming language (e.g. SQL). And if you are not, inventing a data structure that encapsulates a whole program is probably a bigger task than the problem you are actually trying to solve.