3 ms·

Author of the post here. I would love to hear thoughts on folks experience using the different architectures and more pros and cons to each one. There was surpr

by amath 3y ago

Author of the post here. I would love to hear thoughts on folks experience using the different architectures and more pros and cons to each one. There was surprisingly not a lot of information comparing these different architectures that I could find and I would love to be able to update this post with more details.

skadamat 3y ago
besides Ray, are there other tools or platforms that embrace the agent actor architecture model?
- felixgallo 3y ago
  https://www.erlang.org/ https://www.erlang.org/ https://elixir-lang.org/ https://elixir-lang.org/
  - photonthug 3y ago
    Elixir is amazing, see also stuff like https://github.com/bitwalker/libcluster https://github.com/bitwalker/libcluster . I'm always surprised there's not more excitement generated by the possibility of having language-level features that would normally require all the weight of something like k8s. I don't get the impression that many people are seriously using it for data-engineering work though, which seems like a shame
  - Nezteb 3y ago
    Elixir definitely! If you need to use Go, there is also ergo: https://github.com/ergo-services/ergo https://github.com/ergo-services/ergo Yet another cool tool is lunatic: https://github.com/lunatic-solutions/lunatic https://github.com/lunatic-solutions/lunatic
- Sardtok 3y ago
  Akka uses the actor model. https://doc.akka.io/docs/akka/current/typed/actors.html https://doc.akka.io/docs/akka/current/typed/actors.html IIRC, Akka based their implementation on Erlang (possibly OTP), but it's been a long time since I did anything with either Erlang or Scala, and I never used Akka in Java.
  - stereosky 3y ago
    Akka was indeed inspired by Erlang. For a modern take, there's Apache Pekko (incubating), a fork of Akka 2.6.x https://github.com/apache/incubator-pekko https://github.com/apache/incubator-pekko
- jgraettinger1 3y ago
  Estuary Flow derivations [1], I suspect (I work here)? You write a program which has messages pushed to it, which maintains and updates an internal state, and which publishes output messages. You can build cyclic data-flows [2] - not just DAGs - which seems a key ingredient. You can also write contract-based tests that verify the end-to-end behaviors of your "agents" [3]. Today, you're limited to SQLite or TypeScript. Looking forward, we aim to open it up to any program packaged in a docker container. [1] https://docs.estuary.dev/concepts/derivations/ https://docs.estuary.dev/concepts/derivations/ [2] https://docs.estuary.dev/concepts/derivations/#approving-transfers https://docs.estuary.dev/concepts/derivations/#approving-tra... [3] https://docs.estuary.dev/concepts/tests/ https://docs.estuary.dev/concepts/tests/
- brahbrah 3y ago
  Dask added an actor model after seeing it in Ray. I’ve used dasks in some computationally intensive applications (already had existing dask infrastructure which is why we didn’t go with rays)
Sardtok 3y ago
There's the Seven Concurrency Models in Seven Weeks book. It's a bit old, and not so much about frameworks, but more about programming languages and their available threading models or alternatives to pure threads. It still covers a lot of the same theory, but not distributed, and a bit lower level. https://pragprog.com/titles/pb7con/seven-concurrency-models-in-seven-weeks/ https://pragprog.com/titles/pb7con/seven-concurrency-models-...
mamcx 3y ago
I put here an aside: All this is super-complicated for most use-cases. What I have never used, but is inferred from https://en.wikipedia.org/wiki/List_of_concurrent_and_parallel_programming_languages https://en.wikipedia.org/wiki/List_of_concurrent_and_paralle... and in special the original CSP is how all this lacks total control in build your network patterns. In special, is my dream to have a language that allows to express better what is explained starting here: https://zguide.zeromq.org/docs/chapter2/ https://zguide.zeromq.org/docs/chapter2/ There is not (AFAIK) languages that make nice to properly wire this AND keep the semantic understanding/composability of it. For example, is hard to know if a function if is running in a thread/process, what is their priority, etc. I imagine that a language with orchestration (like Erlang/Elixir) in a explicit way will be far easier, but is hard to figure how do it with good composability
- frumiousirc 3y ago
  For static composability, a "ported graph" model makes a lot of sense. This model has graph nodes representing a unit of processing with its input and output data coming through labeled "ports" and edges connecting the ports. Subgraphs can be defined which then appear as a single node in constructing a larger graph (the composability). Such a model can then be mapped to an implementation (ie, used as configuration) in an abstract way that leaves the ability to select the data transport (edges) and graph execution independently from the graph itself. That is, the same graph may execute in a single multi-thread (MT) process or as a multi-process (MP) distributed application. Most simplistically, ports are designated as "input" or "output" (from point of view of the node). But, these designations can be expanded so that ports can carry designations matching the ZeroMQ socket patterns. The implementation can then leverage the transport agnosticity of ZeroMQ when supporting adaptation to MT vs MP but with the cost that all edges require data serialization instead of pointer passing.

by amath 3y ago

skadamat 3y ago

besides Ray, are there other tools or platforms that embrace the agent actor architecture model?

felixgallo 3y ago
https://www.erlang.org/ https://www.erlang.org/ https://elixir-lang.org/ https://elixir-lang.org/
- photonthug 3y ago
  Elixir is amazing, see also stuff like https://github.com/bitwalker/libcluster https://github.com/bitwalker/libcluster . I'm always surprised there's not more excitement generated by the possibility of having language-level features that would normally require all the weight of something like k8s. I don't get the impression that many people are seriously using it for data-engineering work though, which seems like a shame
- Nezteb 3y ago
  Elixir definitely! If you need to use Go, there is also ergo: https://github.com/ergo-services/ergo https://github.com/ergo-services/ergo Yet another cool tool is lunatic: https://github.com/lunatic-solutions/lunatic https://github.com/lunatic-solutions/lunatic
Sardtok 3y ago
Akka uses the actor model. https://doc.akka.io/docs/akka/current/typed/actors.html https://doc.akka.io/docs/akka/current/typed/actors.html IIRC, Akka based their implementation on Erlang (possibly OTP), but it's been a long time since I did anything with either Erlang or Scala, and I never used Akka in Java.
- stereosky 3y ago
  Akka was indeed inspired by Erlang. For a modern take, there's Apache Pekko (incubating), a fork of Akka 2.6.x https://github.com/apache/incubator-pekko https://github.com/apache/incubator-pekko
jgraettinger1 3y ago
Estuary Flow derivations [1], I suspect (I work here)? You write a program which has messages pushed to it, which maintains and updates an internal state, and which publishes output messages. You can build cyclic data-flows [2] - not just DAGs - which seems a key ingredient. You can also write contract-based tests that verify the end-to-end behaviors of your "agents" [3]. Today, you're limited to SQLite or TypeScript. Looking forward, we aim to open it up to any program packaged in a docker container. [1] https://docs.estuary.dev/concepts/derivations/ https://docs.estuary.dev/concepts/derivations/ [2] https://docs.estuary.dev/concepts/derivations/#approving-transfers https://docs.estuary.dev/concepts/derivations/#approving-tra... [3] https://docs.estuary.dev/concepts/tests/ https://docs.estuary.dev/concepts/tests/
brahbrah 3y ago
Dask added an actor model after seeing it in Ray. I’ve used dasks in some computationally intensive applications (already had existing dask infrastructure which is why we didn’t go with rays)

Sardtok 3y ago

There's the Seven Concurrency Models in Seven Weeks book. It's a bit old, and not so much about frameworks, but more about programming languages and their available threading models or alternatives to pure threads. It still covers a lot of the same theory, but not distributed, and a bit lower level. https://pragprog.com/titles/pb7con/seven-concurrency-models-in-seven-weeks/ https://pragprog.com/titles/pb7con/seven-concurrency-models-...

mamcx 3y ago

I put here an aside: All this is super-complicated for most use-cases. What I have never used, but is inferred from https://en.wikipedia.org/wiki/List_of_concurrent_and_parallel_programming_languages https://en.wikipedia.org/wiki/List_of_concurrent_and_paralle... and in special the original CSP is how all this lacks total control in build your network patterns. In special, is my dream to have a language that allows to express better what is explained starting here: https://zguide.zeromq.org/docs/chapter2/ https://zguide.zeromq.org/docs/chapter2/ There is not (AFAIK) languages that make nice to properly wire this AND keep the semantic understanding/composability of it. For example, is hard to know if a function if is running in a thread/process, what is their priority, etc. I imagine that a language with orchestration (like Erlang/Elixir) in a explicit way will be far easier, but is hard to figure how do it with good composability

frumiousirc 3y ago
For static composability, a "ported graph" model makes a lot of sense. This model has graph nodes representing a unit of processing with its input and output data coming through labeled "ports" and edges connecting the ports. Subgraphs can be defined which then appear as a single node in constructing a larger graph (the composability). Such a model can then be mapped to an implementation (ie, used as configuration) in an abstract way that leaves the ability to select the data transport (edges) and graph execution independently from the graph itself. That is, the same graph may execute in a single multi-thread (MT) process or as a multi-process (MP) distributed application. Most simplistically, ports are designated as "input" or "output" (from point of view of the node). But, these designations can be expanded so that ports can carry designations matching the ZeroMQ socket patterns. The implementation can then leverage the transport agnosticity of ZeroMQ when supporting adaptation to MT vs MP but with the cost that all edges require data serialization instead of pointer passing.