4 ms·

It's interesting how every build system, frontend framework, programming language implements its own promise pipeline/delayed execution/observables/event propag

It's interesting how every build system, frontend framework, programming language implements its own promise pipeline/delayed execution/observables/event propagation. But the implementations are rarely extracted out for general purpose usage and rarely have a rich API. I've been thinking a lot about a general purpose "epoll" which be registered on objects that change. I want to be able to register a reaction to a sequence of actions on arbitrary objects with an epoll style API. One of my ideas is GUI thunking. The idea that every interaction with the GUI raises a new type that can be interacted with, to queue up behaviours on the GUI. This is essentially Future<> that are typed and the system reacts to the new type based on what you did and presents a GUI that is as if the operation you queued up was completed. (You can interact with the future because any action on something that isn't done yet, is queued up) It's a bit like terraform plan and apply, but applied to general purpose GUIs. For example, you can click download file, then queue up installation and then using the application, ALL BEFORE it is installed. Because the actual computation is separate from the type information that was queued up. Imagine using AWS Console to set up an entire infrastructure and wire everything together but not actually execute anything until the very end when you click "Run". https://github.com/samsquire/gui-thunks https://github.com/samsquire/gui-thunks I feel we are still early days with regard how to build computer user interfaces that are easy to build, maintain and understand the code for. I used knockout and angularjs 1 and I enjoyed Knockout especially. ko.observables and the map plugin makes creating reactive code very straightforward.

threatofrain 3y ago
> It's interesting how every build system, frontend framework, programming language implements its own promise pipeline/delayed execution/observables/event propagation. Angular relied on RxJS.
- spiralx 3y ago
  And RxJS is itself an implementation of the language-agnostic Reactive framework.
bsaul 3y ago
"It's interesting how every build system, frontend framework, programming language implements its own promise pipeline/delayed execution/observables/event propagation." This rings so true to me. I've recently realized how every single non trivial part of my app is in fact a workflow problem : it could be ideally written as a pipe of asynchronous steps, glued together. It's true both for the frontend part and the backend. I believe that's the point of reactive frameworks, but somehow those frameworks are usually designed around continuous streams of incoming events. Which isn't what i've noticed is the most widespread case. One-shot instanciation of pre-designed workflows would be really ideal.
- samsquire 3y ago
  Thanks for this thoughtful and insightful comment. Many problems can be workflow problems, sometimes even pulling in a rule engine, or require a job queue to do things that can fail. Then you have software such as https://temporal.io/ https://temporal.io/ which is really powerful for resilient workflows. Imagine coordinating the user with a workflow with asynchronous data collection steps. Imagine programming "reaction to user behaviour as a workflow engine". Can coordinate global user behaviour with a resilient workflow script. await user.login(); if (user.showTutorial()) { await user.tutorial(); } await user.checkout(); await user.submitOrder(); You could have seamless weaving of code to be executed on the frontend, or on the server, the workflow is cross-machine and cross job queue.
  - marginalia_nu 3y ago
    This workflow engine of yours seems to expect a very predictable and linear sequence of actions. What happens if the user clicks "log out" after user.tutorial(); It's basically making an already easy case easier.
    - samsquire 3y ago
      I assume you would have conditions for each workflow which are then pattern matched to user behaviour to see if that workflow is relevant. You still have a globally defined happy path of coordination but your overarching application logic isn't spread everywhere but contained in one place. The preconditions for the logout would trigger a different workflow. I am the author of additive-gui, which is based around the idea that you provide all the rules of the GUI and the computer works it out - what applies when. Additive GUIs loosely models dataflow between components and layout. https://github.com/samsquire/additive-guis https://github.com/samsquire/additive-guis It's not ready for anything, it's just a proof of concept. But it doesn't really implement workflows yet. I would rather maintain a codebase that uses this pattern of user workflows, similar to state management than sprinkled logic everywhere. Think of workflow engines as a runtime, not a hardcoded sequence of steps that can be activated or deactivated based on events, like a state machine.
mpweiher 3y ago
> [constraints] the implementations are rarely extracted out for general purpose usage and rarely have a rich API. Yes. One of the goals for Objective-S [1] was that it should be possible to build constraints using the mechanisms of the language (so not hardcoded into the language), but then have them work as if they were built into the language. Part of that was defining how they should look, roughly, and figuring out the architectural structure. I did this in Constraints as Polymorphic Connects [2]. For syntax, I use the symbol |= for a one-way dataflow constraint and =|= for a two-way dataflow constraint. This combines the := that we use for assignment and the | we use for dataflow. Also relates the whole thing to the idea of a "permanent assignment", which I think was introduced in CPL. The structure is simple and general: you need stores that can notify when they are changed, and a copy-element that then copies the data over. At least for one-level constraint. If you want to have multiple levels, you can I was very surprised and happy when I discovered that I had actually figured this out, sort of by accident, when I did Storage Combinators [3]. There is a generic store that does the notifications, which you can compose with any other store. The notifications get sent to a "copier" stream which then copies the changed element. Very easy. And general, as it works for any store. For example, I have been using this to sync up UI with internal state, or two filesystem directories. And when I added a store for SFTP support, syncing to SFTP worked out-of-the-box without any additional work. ("Dropbox in a line of code" is a slogan a colleague came up with, and it's pretty close though of course not 100%) [1] http://objective.st http://objective.st (Site currently being revamped) [2] https://dl.acm.org/doi/10.1145/2889443.2889456?cid=81316491227 https://dl.acm.org/doi/10.1145/2889443.2889456?cid=813164912... / http://www.hpi.uni-potsdam.de/hirschfeld/publications/media/WeiherHirschfeld_2016_ConstraintsAsPolymorphicConnectors_AuthorsVersion.pdf http://www.hpi.uni-potsdam.de/hirschfeld/publications/media/... [3] https://dl.acm.org/doi/10.1145/3359591.3359729 https://dl.acm.org/doi/10.1145/3359591.3359729 / https://www.hpi.uni-potsdam.de/hirschfeld/publications/media/WeiherHirschfeld_2019_StorageCombinators_AcmDL_Preprint.pdf https://www.hpi.uni-potsdam.de/hirschfeld/publications/media...
genuine_smiles 3y ago
> But the implementations are rarely extracted out for general purpose usage and rarely have a rich API. SolidJS pitches its state management as being robust enough for use outside of itself. IMO, most state management tools baked into ui frameworks aren’t robust enough to be worth extracting.
httptoolkit 3y ago
I too loved Knockout's observable model - I've found Mobx a worth successor. As it's entirely independent of any framework it is very practical for general purpose usage. I use it for reactive state with both React and in backend Node code with no problems.
mirekrusin 3y ago
Jane Street/Yaron Minsky/at al came to similar conclusion with their formalized incremental implementation [0] - that this type of computation reappears in different contexts (build systems, ui, optimal data processing/display on large volumes of high frequency data etc). It's very interesting approach indeed. [0] https://blog.janestreet.com/introducing-incremental https://blog.janestreet.com/introducing-incremental
- samsquire 3y ago
  I wonder if there is some wisdom in Rich Hickey's Transducers There is also differential dataflow. I feel all the ideas are related and could be combined. What I want is a rich runtime and API that lets me fork/join, cancel, queue, schedule mutual exclusion (like a lock without a mutex), create dependency trees or graphs. I am also reminded of dataflow programming and Esterel which a kind HN user pointed me towards for synchronous programming of signals.
- cormacrelf 3y ago
  May be of interest: I released a port of Incremental to Rust the other week. It has a bit of a way to go in polish and docs, but the core implementation and API should be very familiar. (To the extent that it uses GADTs for node kinds just like the OCaml!) It has Expert nodes so incremental-map is feasible and already works. I’ve been using it as the state management for some UI with much success. Credit to the authors because it’s a good design. https://github.com/cormacrelf/incremental-rs https://github.com/cormacrelf/incremental-rs
AndrewKemendo 3y ago
> It's interesting how every build system, frontend framework, programming language implements its own promise pipeline/delayed execution/observables/event propagation This is exactly the reason to use state machines/data flows IMO. Every implementation is unique enough that simply following how data flows through the different states and transitions, and where the sinks and funnels are, will tell you everything you need to know about what your system is actually doing at any point in time. The challenge there is, things like that are a shitload of instrumentation and requires a lot of forethought to not just jam everything into a framework that puts boundaries on what you can design and implement. So for 99% of applications, it's not worth the hassle and you're better off with just basic text documentation.
hcarvalhoalves 3y ago
I believe the consequence of this line of thinking is the old idea of writing domain-specific interpreters. In your example, the UI generate a list of commands, and you execute/reduce over at an arbitrary time to get to a new global state, instead of tangling all the UI code with dispatching commands and managing state immediately.
MrJohz 3y ago
I'm not sure generalisation here is ideal - quite often these are specialised implementations for specialised cases, and using them in other situations tends to only work so far. For example, I'm currently working on a spreadsheet-style tool where the reactivity is largely being handled by SolidJS signals. It works fairly well up to a point (and that point is probably good enough for the client's needs), but it's very clear that there are big limitations here, and a more complete solution would bundle its own reactivity system. Things like computing results that spill across several cells just don't map cleanly onto conventional signals, so we instead have lots of ways to manually trigger recomputation, rather than just setting up the perfect pipeline flow. Likewise, figuring out where data loops are happening just isn't really possible. That's not to say that SolidJS is bad for this sort of stuff - it has been great, and it's impressive how well the underlying reactive primitives work even for this project. But I think even when the underlying theory between these tools is pretty similar, the practical tradeoffs that need to be made are very different. And as a result, the different libraries servicing these different use cases will look very different. I suspect this is the reason why these implementations are rarely extracted out more broadly. The sort of system that works well for one situation will rarely work so well for another.
still_grokking 3y ago
This sounds almost like you've just invented Haskell-like `IO`. All operations get "queued up" in IO and only run "at the end of the world". To write your program you `flatMap` over the `IO`: An `IO` contains the computation(s) that will run at some point, but you can map on the result of them right away, and return another `IO` value; than `flatten` the `IO[IO]` data structure (which makes `flatMap`). In Haskell you have extra syntax for `flatMap` called "do notation". In Scala, where there are library solutions for `IO`, you can use "for comprehensions" instead. In both cases the nested `flatMap` calls get sequenced. This way and you can write code almost like a consecutive chain of imperative procedure calls but it all gets "queued up" and the whole program only runs when the `IO` data structure gets evaluated by the runtime ("at the end of the world", as last call in your program). But that's not really related to data flow.

threatofrain 3y ago

> It's interesting how every build system, frontend framework, programming language implements its own promise pipeline/delayed execution/observables/event propagation. Angular relied on RxJS.

spiralx 3y ago
And RxJS is itself an implementation of the language-agnostic Reactive framework.

bsaul 3y ago

"It's interesting how every build system, frontend framework, programming language implements its own promise pipeline/delayed execution/observables/event propagation." This rings so true to me. I've recently realized how every single non trivial part of my app is in fact a workflow problem : it could be ideally written as a pipe of asynchronous steps, glued together. It's true both for the frontend part and the backend. I believe that's the point of reactive frameworks, but somehow those frameworks are usually designed around continuous streams of incoming events. Which isn't what i've noticed is the most widespread case. One-shot instanciation of pre-designed workflows would be really ideal.

samsquire 3y ago
Thanks for this thoughtful and insightful comment. Many problems can be workflow problems, sometimes even pulling in a rule engine, or require a job queue to do things that can fail. Then you have software such as https://temporal.io/ https://temporal.io/ which is really powerful for resilient workflows. Imagine coordinating the user with a workflow with asynchronous data collection steps. Imagine programming "reaction to user behaviour as a workflow engine". Can coordinate global user behaviour with a resilient workflow script. await user.login(); if (user.showTutorial()) { await user.tutorial(); } await user.checkout(); await user.submitOrder(); You could have seamless weaving of code to be executed on the frontend, or on the server, the workflow is cross-machine and cross job queue.
- marginalia_nu 3y ago
  This workflow engine of yours seems to expect a very predictable and linear sequence of actions. What happens if the user clicks "log out" after user.tutorial(); It's basically making an already easy case easier.
  - samsquire 3y ago
    I assume you would have conditions for each workflow which are then pattern matched to user behaviour to see if that workflow is relevant. You still have a globally defined happy path of coordination but your overarching application logic isn't spread everywhere but contained in one place. The preconditions for the logout would trigger a different workflow. I am the author of additive-gui, which is based around the idea that you provide all the rules of the GUI and the computer works it out - what applies when. Additive GUIs loosely models dataflow between components and layout. https://github.com/samsquire/additive-guis https://github.com/samsquire/additive-guis It's not ready for anything, it's just a proof of concept. But it doesn't really implement workflows yet. I would rather maintain a codebase that uses this pattern of user workflows, similar to state management than sprinkled logic everywhere. Think of workflow engines as a runtime, not a hardcoded sequence of steps that can be activated or deactivated based on events, like a state machine.

mpweiher 3y ago

> [constraints] the implementations are rarely extracted out for general purpose usage and rarely have a rich API. Yes. One of the goals for Objective-S [1] was that it should be possible to build constraints using the mechanisms of the language (so not hardcoded into the language), but then have them work as if they were built into the language. Part of that was defining how they should look, roughly, and figuring out the architectural structure. I did this in Constraints as Polymorphic Connects [2]. For syntax, I use the symbol |= for a one-way dataflow constraint and =|= for a two-way dataflow constraint. This combines the := that we use for assignment and the | we use for dataflow. Also relates the whole thing to the idea of a "permanent assignment", which I think was introduced in CPL. The structure is simple and general: you need stores that can notify when they are changed, and a copy-element that then copies the data over. At least for one-level constraint. If you want to have multiple levels, you can I was very surprised and happy when I discovered that I had actually figured this out, sort of by accident, when I did Storage Combinators [3]. There is a generic store that does the notifications, which you can compose with any other store. The notifications get sent to a "copier" stream which then copies the changed element. Very easy. And general, as it works for any store. For example, I have been using this to sync up UI with internal state, or two filesystem directories. And when I added a store for SFTP support, syncing to SFTP worked out-of-the-box without any additional work. ("Dropbox in a line of code" is a slogan a colleague came up with, and it's pretty close though of course not 100%) [1] http://objective.st http://objective.st (Site currently being revamped) [2] https://dl.acm.org/doi/10.1145/2889443.2889456?cid=81316491227 https://dl.acm.org/doi/10.1145/2889443.2889456?cid=813164912... / http://www.hpi.uni-potsdam.de/hirschfeld/publications/media/WeiherHirschfeld_2016_ConstraintsAsPolymorphicConnectors_AuthorsVersion.pdf http://www.hpi.uni-potsdam.de/hirschfeld/publications/media/... [3] https://dl.acm.org/doi/10.1145/3359591.3359729 https://dl.acm.org/doi/10.1145/3359591.3359729 / https://www.hpi.uni-potsdam.de/hirschfeld/publications/media/WeiherHirschfeld_2019_StorageCombinators_AcmDL_Preprint.pdf https://www.hpi.uni-potsdam.de/hirschfeld/publications/media...

genuine_smiles 3y ago

> But the implementations are rarely extracted out for general purpose usage and rarely have a rich API. SolidJS pitches its state management as being robust enough for use outside of itself. IMO, most state management tools baked into ui frameworks aren’t robust enough to be worth extracting.

httptoolkit 3y ago

I too loved Knockout's observable model - I've found Mobx a worth successor. As it's entirely independent of any framework it is very practical for general purpose usage. I use it for reactive state with both React and in backend Node code with no problems.

mirekrusin 3y ago

Jane Street/Yaron Minsky/at al came to similar conclusion with their formalized incremental implementation [0] - that this type of computation reappears in different contexts (build systems, ui, optimal data processing/display on large volumes of high frequency data etc). It's very interesting approach indeed. [0] https://blog.janestreet.com/introducing-incremental https://blog.janestreet.com/introducing-incremental

samsquire 3y ago
I wonder if there is some wisdom in Rich Hickey's Transducers There is also differential dataflow. I feel all the ideas are related and could be combined. What I want is a rich runtime and API that lets me fork/join, cancel, queue, schedule mutual exclusion (like a lock without a mutex), create dependency trees or graphs. I am also reminded of dataflow programming and Esterel which a kind HN user pointed me towards for synchronous programming of signals.
cormacrelf 3y ago
May be of interest: I released a port of Incremental to Rust the other week. It has a bit of a way to go in polish and docs, but the core implementation and API should be very familiar. (To the extent that it uses GADTs for node kinds just like the OCaml!) It has Expert nodes so incremental-map is feasible and already works. I’ve been using it as the state management for some UI with much success. Credit to the authors because it’s a good design. https://github.com/cormacrelf/incremental-rs https://github.com/cormacrelf/incremental-rs

AndrewKemendo 3y ago

> It's interesting how every build system, frontend framework, programming language implements its own promise pipeline/delayed execution/observables/event propagation This is exactly the reason to use state machines/data flows IMO. Every implementation is unique enough that simply following how data flows through the different states and transitions, and where the sinks and funnels are, will tell you everything you need to know about what your system is actually doing at any point in time. The challenge there is, things like that are a shitload of instrumentation and requires a lot of forethought to not just jam everything into a framework that puts boundaries on what you can design and implement. So for 99% of applications, it's not worth the hassle and you're better off with just basic text documentation.

hcarvalhoalves 3y ago

I believe the consequence of this line of thinking is the old idea of writing domain-specific interpreters. In your example, the UI generate a list of commands, and you execute/reduce over at an arbitrary time to get to a new global state, instead of tangling all the UI code with dispatching commands and managing state immediately.

MrJohz 3y ago

I'm not sure generalisation here is ideal - quite often these are specialised implementations for specialised cases, and using them in other situations tends to only work so far. For example, I'm currently working on a spreadsheet-style tool where the reactivity is largely being handled by SolidJS signals. It works fairly well up to a point (and that point is probably good enough for the client's needs), but it's very clear that there are big limitations here, and a more complete solution would bundle its own reactivity system. Things like computing results that spill across several cells just don't map cleanly onto conventional signals, so we instead have lots of ways to manually trigger recomputation, rather than just setting up the perfect pipeline flow. Likewise, figuring out where data loops are happening just isn't really possible. That's not to say that SolidJS is bad for this sort of stuff - it has been great, and it's impressive how well the underlying reactive primitives work even for this project. But I think even when the underlying theory between these tools is pretty similar, the practical tradeoffs that need to be made are very different. And as a result, the different libraries servicing these different use cases will look very different. I suspect this is the reason why these implementations are rarely extracted out more broadly. The sort of system that works well for one situation will rarely work so well for another.

still_grokking 3y ago

This sounds almost like you've just invented Haskell-like `IO`. All operations get "queued up" in IO and only run "at the end of the world". To write your program you `flatMap` over the `IO`: An `IO` contains the computation(s) that will run at some point, but you can map on the result of them right away, and return another `IO` value; than `flatten` the `IO[IO]` data structure (which makes `flatMap`). In Haskell you have extra syntax for `flatMap` called "do notation". In Scala, where there are library solutions for `IO`, you can use "for comprehensions" instead. In both cases the nested `flatMap` calls get sequenced. This way and you can write code almost like a consecutive chain of imperative procedure calls but it all gets "queued up" and the whole program only runs when the `IO` data structure gets evaluated by the runtime ("at the end of the world", as last call in your program). But that's not really related to data flow.