10 ms·

Goblins: A transactional, distributed actor model environment

134 points by snicker7 6y ago

nerdponx 6y ago
Cool! I hope one day Racket will take off as a serious language for app development. I know it's not really the point of Racket, but it has all the right attributes^, all it needs is a bigger library ecosystem. ^: including a package manager & build toolchain, an IDE, nicely-formatted web hosting for docs, runtime is reasonably fast, has plenty of batteries included, has cool stuff like contracts, supports multiple programming paradigms.
- tartoran 6y ago
  That would be great but even if doesn’t get there, Racket is still an amazing ecosystem for exploring new ideas and will continue to love it. But of course any improvements are welcome
- mark_l_watson 6y ago
  I agree. Racketfest Amateur Night happened last Friday and Saturday and many cool projects were show cased.
- giancarlostoro 6y ago
  Kind of weird hearing that when we're posting on a website running atop Racket.
  - TeMPOraL 6y ago
    This website also runs on a single server[0], stores user state in closures[1] and uses tables for layout[2], in violation of just about every single modern web development practice. And it's all the better for it. I wish more sites would follow HN's example. -- [0] - https://news.ycombinator.com/item?id=16076041 https://news.ycombinator.com/item?id=16076041 [1] - Mostly, at least - https://news.ycombinator.com/item?id=23374471 https://news.ycombinator.com/item?id=23374471. It used to be almost for every action, the pg's OG hack, but got optimized away. [2] - Press CTRL+U.
  - deleted 6y ago
    [deleted]
  - yen223 6y ago
    Hacker News isn't Racket, it's Arc
    - vulcan01 6y ago
      Arc is built in / runs on Racket: http://www.arclanguage.org/install http://www.arclanguage.org/install
      - yen223 6y ago
        Oh, my mistake
      - rurban 6y ago
        On the new Chez Racket already? (Sorry, didn't bother to create an arcforum account yet)
Jeff_Brown 6y ago
Looks fun. How novel is it? Can I do these things already in some other language? (Esp. Haskell or Python -- my favorite language and my work language, respectively.)
- paroneayea 6y ago
  If the version of CapTP we have is ported to other languages, then you could do some of it, but probably not all. Haskell might be better up for some, but Python won't be able to do some of the more interesting parts (like the time-travel support) quite as easily since it's a much more mutation-oriented environment. Still, you could get systems talking to each other over the network if a compatible CapTP was implemented.
- zenhack 6y ago
  The networked actor-model bit & CapTP goes back to E originally[1]. The other contemporary real-world protocol based on this design is capnproto rpc[2], which has implementations in several languages including both Haskell (of which I am the author) and Python. The ergonomics of the Haskell implementation could use some work IMO, and I've got some ongoing refactoring work on a branch. But it does work, and folks are using it. [1]: http://erights.org/ http://erights.org/ [2]: https://capnproto.org/ https://capnproto.org/
paroneayea 6y ago
Oh hey neat, some coverage. Actually I'm right on the verge of the next big release (v0.8) which should make doing the networked programming significantly easier. But here's some more background: I'm co-author/co-editor of the ActivityPub specification, which might give you some idea that I have some experience with trying to build networked systems. Goblins is part of the Spritely project, or even more accurately, the foundation for it: https://spritelyproject.org/ https://spritelyproject.org/ Spritely has some pretty wild ambitions... I won't go into them in detail, the video near the top of the above site explains better. But achieving those in a feasible timeframe means we need some better tooling. Goblins is the foundation of that; it's implemented as a Racket library, but can/will be ported elsewhere. The docs linked above explain some of the core ideas, but honestly I think the video linked from this section of the site explains it better: https://spritelyproject.org/#goblins https://spritelyproject.org/#goblins In general, people tend to realize that something interesting is happening when they see a cool shiny demo. So here are two pages that might give you an idea with some cool shiny demos: - Demonstrating "time travel" in a space shooter game (no special code added for this other than exposing it in the GUI): https://dustycloud.org/blog/goblins-time-travel-micropreview/ https://dustycloud.org/blog/goblins-time-travel-micropreview... - "goblin-chat", which doesn't look quite as shiny but is interesting more that the code you see above is "end to end encrypted", but is written in a mere 250 lines of code for the whole protocol http://dustycloud.org/blog/spritely-goblins-v0.7-released/ http://dustycloud.org/blog/spritely-goblins-v0.7-released/ What? How can the last one be only 250 lines of code? (And a mere 300 more for the GUI!) Well, that's because we're implementing a peer-to-peer distributed object programming protocol called CapTP (which includes wild things like distributed garbage collection over the network (for acyclic references), "promise pipelining", and is object capability secure). The next release will be advancing that work significantly. The Agoric organization is also implementing CapTP, but their stuff is in Javascript instead of Racket; we plan to have our CapTPs to converge. Thus it shouldn't really matter whether you write your code in Javascript or Racket, your code should be able to talk to each other. Anyway, new release coming out soon. Hope that answers some things since the docs page might not convey what's fully interesting about it (and indeed, neither does this text, but the videos mentioned above do a better job).
- carterschonwald 6y ago
tartoran 6y ago
What are the best use cases for distributed actor model solutions? Im just slightly familiar with it and it’s on my learning list. Anybody here have any experience with it? How about with Goblins?
- unoti 6y ago
  The use case for actors is any time you want to distribute work across multiple nodes. In general, actors are a great concurrency model that simplifies a lot of the complexity of distributed programming. For more info on "why", see: https://doc.akka.io/docs/akka/current/typed/guide/actors-intro.html https://doc.akka.io/docs/akka/current/typed/guide/actors-int... https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/Orleans-MSR-TR-2014-41.pdf https://www.microsoft.com/en-us/research/wp-content/uploads/...
  - tartoran 6y ago
    Thanks, the first link answers some of my questions. >Use of actors allows us to: > Enforce encapsulation without resorting to locks. > Use the model of cooperative entities reacting to signals, changing state, and sending signals to each other to drive the whole application forward. > Stop worrying about an executing mechanism which is a mismatch to our world view. Did anyone here use Goblins for any project and would like to share their experience?
    - snicker7 6y ago
      The author posted some interesting applications using Goblins (a video game and chat app). There is some interest in porting Goblins to Guile Scheme to integrate with GNU Shepherd (a service orchestrator) ... somehow.
  - blandflakes 6y ago
    I don't want to distract from this comment, because I think it's spot on, but I think another interesting use for actors is as one of the few sane concurrency models for mutation. Pony does this - objects are actors by default. Things can act concurrently without fear external to an object, but all mutations within a given object are serialized. Seems like a nice set of defaults.
sriram_malhar 6y ago
Can someone familiar with the framework answer a few questions? Do transactions span distributed actors? Perhaps I missed it, but I see no recovery log. It implements distributed garbage collection. This is a non-starter for any environment where the network cannot be taken for granted. RMI/DCOM/CORBA had no shortage of problems due to distributed GC alone.
- evgen 6y ago
  If you want to understand the tech more you should start with the E language it is based on at erights.org. In the case of distributed GC the system understands that the network can fail and plans for this: http://www.erights.org/history/original-e/dgc/index.html http://www.erights.org/history/original-e/dgc/index.html (caveat: I know original E and have no direct knowledge of whether or not this bit has been implemented in Goblins)
  - sriram_malhar 6y ago
    I am quite familiar with E. Or used to be during my PhD. As far I recall, neither E's implementation nor spec cover distributed transactions, fault-tolerance or GC issues.
    - odipar 6y ago
      self-plug: you may want to checkout Manikin: it is very similar to Goblins, but with a clear transactional model. https://odipar.github.io/manikin https://odipar.github.io/manikin
    - paroneayea 6y ago
      Hi! So, first of all, I think I need to make the description of Goblins clearer. It says "A transactional, distributed actor model environment", but "transactional" and "distributed" are two separate components. This follows the "vat turn" communicating event loop model of E, where every message handling in an event loop is done in a transaction. It isn't meant to say distributed transactions. You can build distributed transactions a couple of ways though: as an abstraction on top of the distributed object system provided, or as a consensus model where a quorum of nodes agree on the order of messages for a deterministic vat/event loop. By the way, the "open source version of E" does do distributed garbage collection, but it can't collect cycles across a network barrier. However, the original version of E did collect cycles across the network too: http://erights.org/history/original-e/dgc/ http://erights.org/history/original-e/dgc/ However, you needed hooks into the garbage collector for it to work, since you had to do some introspection of how things were rooted iirc. Mark S. Miller told me that originally they thought this could be fixed because Sun had promised to make Java open source, but they took a damn long time to do it. In the middle period, apparently the E folks said "please! just take this patch... you can have it even!" but Sun didn't merge it. But Mark also told me that he learned not to push for this, because the cycles-across the network stuff turned out to not be common enough to bother, and you could build distributed acyclic gc using just weakrefs (and weak maps) and gc finalizer hooks, which are much more common than the rooting-introspection stuff. Not sure if I got all that right, but there you go. Anyway, I'll fix the tagline for Goblins for next release to make it clearer. Thanks for the feedback.
TeMPOraL 6y ago
I've just discovered someone built "a completely peer to peer, end to end encrypted chat system" working over Tor[0][1] using Goblins. Not sure how significant this is, but I found it interesting. -- [0] - https://gitlab.com/spritely/goblin-chat https://gitlab.com/spritely/goblin-chat [1] - https://octodon.social/@cwebber/105992747855007110 https://octodon.social/@cwebber/105992747855007110
- rapnie 6y ago
  > someone It is from the researcher / developer of Goblins himself, Christopher Lemmer Webber. Very interesting indeed.
deleted 6y ago
[deleted]

134 points by snicker7 6y ago

Cool! I hope one day Racket will take off as a serious language for app development. I know it's not really the point of Racket, but it has all the right attributes^, all it needs is a bigger library ecosystem. ^: including a package manager & build toolchain, an IDE, nicely-formatted web hosting for docs, runtime is reasonably fast, has plenty of batteries included, has cool stuff like contracts, supports multiple programming paradigms.

tartoran 6y ago
That would be great but even if doesn’t get there, Racket is still an amazing ecosystem for exploring new ideas and will continue to love it. But of course any improvements are welcome
mark_l_watson 6y ago
I agree. Racketfest Amateur Night happened last Friday and Saturday and many cool projects were show cased.
giancarlostoro 6y ago
Kind of weird hearing that when we're posting on a website running atop Racket.
- TeMPOraL 6y ago
  This website also runs on a single server[0], stores user state in closures[1] and uses tables for layout[2], in violation of just about every single modern web development practice. And it's all the better for it. I wish more sites would follow HN's example. -- [0] - https://news.ycombinator.com/item?id=16076041 https://news.ycombinator.com/item?id=16076041 [1] - Mostly, at least - https://news.ycombinator.com/item?id=23374471 https://news.ycombinator.com/item?id=23374471. It used to be almost for every action, the pg's OG hack, but got optimized away. [2] - Press CTRL+U.
- deleted 6y ago
  [deleted]
- yen223 6y ago
  Hacker News isn't Racket, it's Arc
  - vulcan01 6y ago
    Arc is built in / runs on Racket: http://www.arclanguage.org/install http://www.arclanguage.org/install
    - yen223 6y ago
      Oh, my mistake
    - rurban 6y ago
      On the new Chez Racket already? (Sorry, didn't bother to create an arcforum account yet)

Jeff_Brown 6y ago

Looks fun. How novel is it? Can I do these things already in some other language? (Esp. Haskell or Python -- my favorite language and my work language, respectively.)

paroneayea 6y ago
If the version of CapTP we have is ported to other languages, then you could do some of it, but probably not all. Haskell might be better up for some, but Python won't be able to do some of the more interesting parts (like the time-travel support) quite as easily since it's a much more mutation-oriented environment. Still, you could get systems talking to each other over the network if a compatible CapTP was implemented.
zenhack 6y ago
The networked actor-model bit & CapTP goes back to E originally[1]. The other contemporary real-world protocol based on this design is capnproto rpc[2], which has implementations in several languages including both Haskell (of which I am the author) and Python. The ergonomics of the Haskell implementation could use some work IMO, and I've got some ongoing refactoring work on a branch. But it does work, and folks are using it. [1]: http://erights.org/ http://erights.org/ [2]: https://capnproto.org/ https://capnproto.org/

paroneayea 6y ago

Oh hey neat, some coverage. Actually I'm right on the verge of the next big release (v0.8) which should make doing the networked programming significantly easier. But here's some more background: I'm co-author/co-editor of the ActivityPub specification, which might give you some idea that I have some experience with trying to build networked systems. Goblins is part of the Spritely project, or even more accurately, the foundation for it: https://spritelyproject.org/ https://spritelyproject.org/ Spritely has some pretty wild ambitions... I won't go into them in detail, the video near the top of the above site explains better. But achieving those in a feasible timeframe means we need some better tooling. Goblins is the foundation of that; it's implemented as a Racket library, but can/will be ported elsewhere. The docs linked above explain some of the core ideas, but honestly I think the video linked from this section of the site explains it better: https://spritelyproject.org/#goblins https://spritelyproject.org/#goblins In general, people tend to realize that something interesting is happening when they see a cool shiny demo. So here are two pages that might give you an idea with some cool shiny demos: - Demonstrating "time travel" in a space shooter game (no special code added for this other than exposing it in the GUI): https://dustycloud.org/blog/goblins-time-travel-micropreview/ https://dustycloud.org/blog/goblins-time-travel-micropreview... - "goblin-chat", which doesn't look quite as shiny but is interesting more that the code you see above is "end to end encrypted", but is written in a mere 250 lines of code for the whole protocol http://dustycloud.org/blog/spritely-goblins-v0.7-released/ http://dustycloud.org/blog/spritely-goblins-v0.7-released/ What? How can the last one be only 250 lines of code? (And a mere 300 more for the GUI!) Well, that's because we're implementing a peer-to-peer distributed object programming protocol called CapTP (which includes wild things like distributed garbage collection over the network (for acyclic references), "promise pipelining", and is object capability secure). The next release will be advancing that work significantly. The Agoric organization is also implementing CapTP, but their stuff is in Javascript instead of Racket; we plan to have our CapTPs to converge. Thus it shouldn't really matter whether you write your code in Javascript or Racket, your code should be able to talk to each other. Anyway, new release coming out soon. Hope that answers some things since the docs page might not convey what's fully interesting about it (and indeed, neither does this text, but the videos mentioned above do a better job).

carterschonwald 6y ago

tartoran 6y ago

What are the best use cases for distributed actor model solutions? Im just slightly familiar with it and it’s on my learning list. Anybody here have any experience with it? How about with Goblins?

unoti 6y ago
The use case for actors is any time you want to distribute work across multiple nodes. In general, actors are a great concurrency model that simplifies a lot of the complexity of distributed programming. For more info on "why", see: https://doc.akka.io/docs/akka/current/typed/guide/actors-intro.html https://doc.akka.io/docs/akka/current/typed/guide/actors-int... https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/Orleans-MSR-TR-2014-41.pdf https://www.microsoft.com/en-us/research/wp-content/uploads/...
- tartoran 6y ago
  Thanks, the first link answers some of my questions. >Use of actors allows us to: > Enforce encapsulation without resorting to locks. > Use the model of cooperative entities reacting to signals, changing state, and sending signals to each other to drive the whole application forward. > Stop worrying about an executing mechanism which is a mismatch to our world view. Did anyone here use Goblins for any project and would like to share their experience?
  - snicker7 6y ago
    The author posted some interesting applications using Goblins (a video game and chat app). There is some interest in porting Goblins to Guile Scheme to integrate with GNU Shepherd (a service orchestrator) ... somehow.
- blandflakes 6y ago
  I don't want to distract from this comment, because I think it's spot on, but I think another interesting use for actors is as one of the few sane concurrency models for mutation. Pony does this - objects are actors by default. Things can act concurrently without fear external to an object, but all mutations within a given object are serialized. Seems like a nice set of defaults.

sriram_malhar 6y ago

Can someone familiar with the framework answer a few questions? Do transactions span distributed actors? Perhaps I missed it, but I see no recovery log. It implements distributed garbage collection. This is a non-starter for any environment where the network cannot be taken for granted. RMI/DCOM/CORBA had no shortage of problems due to distributed GC alone.

evgen 6y ago
If you want to understand the tech more you should start with the E language it is based on at erights.org. In the case of distributed GC the system understands that the network can fail and plans for this: http://www.erights.org/history/original-e/dgc/index.html http://www.erights.org/history/original-e/dgc/index.html (caveat: I know original E and have no direct knowledge of whether or not this bit has been implemented in Goblins)
- sriram_malhar 6y ago
  I am quite familiar with E. Or used to be during my PhD. As far I recall, neither E's implementation nor spec cover distributed transactions, fault-tolerance or GC issues.
  - odipar 6y ago
    self-plug: you may want to checkout Manikin: it is very similar to Goblins, but with a clear transactional model. https://odipar.github.io/manikin https://odipar.github.io/manikin
  - paroneayea 6y ago
    Hi! So, first of all, I think I need to make the description of Goblins clearer. It says "A transactional, distributed actor model environment", but "transactional" and "distributed" are two separate components. This follows the "vat turn" communicating event loop model of E, where every message handling in an event loop is done in a transaction. It isn't meant to say distributed transactions. You can build distributed transactions a couple of ways though: as an abstraction on top of the distributed object system provided, or as a consensus model where a quorum of nodes agree on the order of messages for a deterministic vat/event loop. By the way, the "open source version of E" does do distributed garbage collection, but it can't collect cycles across a network barrier. However, the original version of E did collect cycles across the network too: http://erights.org/history/original-e/dgc/ http://erights.org/history/original-e/dgc/ However, you needed hooks into the garbage collector for it to work, since you had to do some introspection of how things were rooted iirc. Mark S. Miller told me that originally they thought this could be fixed because Sun had promised to make Java open source, but they took a damn long time to do it. In the middle period, apparently the E folks said "please! just take this patch... you can have it even!" but Sun didn't merge it. But Mark also told me that he learned not to push for this, because the cycles-across the network stuff turned out to not be common enough to bother, and you could build distributed acyclic gc using just weakrefs (and weak maps) and gc finalizer hooks, which are much more common than the rooting-introspection stuff. Not sure if I got all that right, but there you go. Anyway, I'll fix the tagline for Goblins for next release to make it clearer. Thanks for the feedback.

TeMPOraL 6y ago

I've just discovered someone built "a completely peer to peer, end to end encrypted chat system" working over Tor[0][1] using Goblins. Not sure how significant this is, but I found it interesting. -- [0] - https://gitlab.com/spritely/goblin-chat https://gitlab.com/spritely/goblin-chat [1] - https://octodon.social/@cwebber/105992747855007110 https://octodon.social/@cwebber/105992747855007110

rapnie 6y ago
> someone It is from the researcher / developer of Goblins himself, Christopher Lemmer Webber. Very interesting indeed.

deleted 6y ago

[deleted]