8 ms·

Rama on Clojure's terms, and the magic of continuation-passing style

106 points by nathanmarz 2y ago

robertlagrant 2y ago
This article is written for people who know Clojure, or at least the examples are. It might be nice to see the examples written in a non-LISP as well. E.g. (?<- (ops/explode [1 2 3 4] :> *v) (println "Val:" *v)) Is (I believe) the equivalent of Python's for element in [1, 2, 3, 4]: print(element)
- nathanmarz 2y ago
  That Python code has the same effect, but the equivalent Python in CPS would be: def explode(args, cont): for e in args: cont(e) (explode [1, 2, 3, 4], print)
  - gpderetta 2y ago
    That's not in CPS form as cont returns into explode. Even if we treat the for loop in explode as an implementation detail, cont should take the next continuation as a parameter.
    - cryptonector 2y ago
      From TFA: > For every invoke, Rama determines if it’s executing a deframaop or a deframafn . If it’s a deframafn , then it invokes it just like how functions are invoked in Clojure by unrolling the stack frame with the return value. [...] So I think GP's Python example matches quite well with what Rama does in the Rama equivalent from TFA. Functions in Rama do return like normal functions in Clojure. It's Rama "ops" that call the continuation, but "ops" clearly _can_ return back to the caller, as that's how the whole generator aspect of Rama works -- it couldn't be any other way. So `println` is a function and it returns to its caller. This is very different from a Scheme that compiles to CPS such that every return is a continuation call. The point is that in such a Scheme you end up with all activation frames being on the heap and thus needing garbage collection. This is why delimited continuations were invented: Scheme-style call/cc continuations are just too expensive! But there is another optimization one can do: some returns can just unwind, while others can call a continuation. This is the optimization that Rama is going for near as I can see. It's a lot like Icon's `return`, which unwinds (in the Icon-to-C compiler anyways) and Icon's `suspend`, which calls the current continuation (again, in the Icon-to-C compiler case). This way you can return when you're not generating results -- when you're returning the last value, or when pruning (in Icon that's failure, which also unwinds, like `return`).
cryptonector 2y ago
The whole thing about distinguishing functions that yield once from all other functions is pretty clever. Reminds me of how in Icon a `return` is a C-like return that unwinds the stack while `suspend` is more like calling a continuation with the yielded value. In the old Icon-to-C compiler that's exactly how `return` and `suspend` worked. But Icon does not analyze function ("procedure") bodies to see if they only return, so the `return`-unwinds optimization is a run-time one. Whereas Rama does a compile-time optimization, which makes a lot of sense.
cryptonector 2y ago
Just hours (a day?) earlier there was this very interesting -and IMO relevant here- HN post: Can logic programming be liberated from predicates and backtracking? [pdf] (uni-kiel.de) https://news.ycombinator.com/item?id=41816545 That deals with backtracking, which is often implemented with continuations, as in TFA.
fire_lake 2y ago
What is the best introductory post to Rama right now? I would like to skip the marketing and understand how it compares to a wiring together Kafka, Spark, MySQL, etc.
- deleted 2y ago
  [deleted]
- bbor 2y ago
  AFAICT it's not actually published yet? It's a bit of a confusing situation because you can "download Rama" but it's not "full", whatever that means. See https://redplanetlabs.com/learn-rama https://redplanetlabs.com/learn-rama E: this download has "the full Rama API for use in simulated clusters within a single process", but also "Rama is currently available in a private beta." That's a highly unusual way to release what appears to be a Java library at the end of the day, but hopefully that's because it's unusually awesome! Looking forward to actual info some time in the future. I wonder if the "private beta" costs money...
  - nathanmarz 2y ago
    What we've released publicly on our public Maven repository is a different build of Rama which can only be used for testing/experimentation in a single process. So it can't be used to run full clusters. It's API-equivalent to the full Rama build. This will change when we move out of private beta, when Rama will be free to use for production for small-scale applications.
    - grounder 2y ago
      Do you have an estimate of when you'll be out of private beta and available? Can you share any more about pricing or what you consider to be a "small-scale application". Thanks!
      - nathanmarz 2y ago
        We're aiming to be out of private beta early next year. "Small-scale" basically means the kind of scale a single Postgres node + application server can handle.
beders 2y ago
> A Rama operation does not return a value to its caller. It emits values to its continuation. This is a critical distinction, as part of what makes Rama operations more general than functions is how they can emit multiple times, not emit at all, or emit asynchronously.
- thom 2y ago
  Which is obviously very similar to how transducers already work in Clojure, but they still lack some of concurrency options of reducers. Getting all this on a smart, distributed runtime seems very promising.
  - mschaef 2y ago
    > Getting all this on a smart, distributed runtime seems very promising. Hopefully it is. This CPS article is the first of the Rama blog posts where it seemed like there might be something there. The earlier posts - "I built Twitter scale Twitter in 10Kloc" - were never really all that convincing. The thing they claimed to have built was too ambitious a claim.
    - thom 2y ago
      Oh I think there’s a lot of good stuff baked in there. The big idea downstream is that you have incrementally calculated, indexed data structures to query all the results of this fancy CPS logic. It’s all slightly esoteric even coming from a Clojure background but it ticks every box I want from a modern data platform, short of speaking SQL.
  - dig1 2y ago
    There is a library called tesser [1] (by Jepsen/Riemann author) that behaves like parallel transducers with more Clojure "native" syntax. With transducers, you have to use "comp," and with tesser, you use "->>" as you'd use with lazy functions. Sadly, tesser is not advertised as it should; I find it much more flexible than transducers. E.g. you could parallelize tesser code over Spark/Hadoop cluster. [1] https://github.com/aphyr/tesser https://github.com/aphyr/tesser
moomin 2y ago
I feel like CPS is one of those tar pits smart developers fall into. It’s just a fundamentally unfriendly API, like mutexes. We saw this with node as well: eventually the language designers just sighed and added promises. You’re better off with an asynchronous result stream, which is equivalent in power but much easier to reason about. C#’s got IAsyncEnumerable, I know that Rust is working on designing something similar. Even then, it can be hard to analyse the behaviour of multiple levels of asynchronous streams and passing pieces of information from the top level to the bottom level like a tag is a pain in the neck.
- neonsunset 2y ago
  I found myself liking the F# way of consuming/producing IAsyncEnumerable's with taskseq. It's very terse and looks nice: https://github.com/fsprojects/FSharp.Control.TaskSeq?tab=readme-ov-file#examples https://github.com/fsprojects/FSharp.Control.TaskSeq?tab=rea...
  - moomin 2y ago
    Apparently Microsoft hold a patent on “yield!”, which makes it all the more frustrating that they haven’t included it in C#.
    - neonsunset 2y ago
      In F#, yield! is a computation expression, C#'s yield within methods that return IAsyncEnumerable<T> works more or less the same way.
- cryptonector 2y ago
  IMO TFA makes much of CPS because that's how they chose to implement generators, but the main thing about the language is that it has generators. Now having generators is nothing new, but I don't want to take too much away from TFA, as there are some interesting things there. I'll limit myself to pointing out that the Icon programming language had generators and pervasive backtracking using CPS in the Icon-to-C compiler, and that other languages with generators and pervasive backtracking have been implemented with CPS as well as with bytecode VMs that don't have any explicit (internally) continuations. Examples include Prolog, Icon, and jq, to name just three, and now of course, Rama.
thunkingdeep 2y ago
I’ve always thought that CPS is a good barometer for finding out whether a developer is talented or whether they THINK they’re talented enough to design and/or implement these kinds of compiler components. This kind of thing but definitely CPS in particular is so much trickier to nail down than it initially seems if you’ve written a compiler before. Up there in difficulty with automatic parallelization and loop transforming. I tried to write a very small POC lisp once with an idea to have all vectors of known sizes get map’d in parallel and I never could nail it down. Kudos to all involved. Clojure is such a mind bending tool. God only knows what it takes these people to maintain the guts of it all.
kamma4434 2y ago
I don’t want to be a party pooper, but from my very cursory look at the page, I don’t think this will go much farther than a very small community. I feel like you’re adding a lot of complexity compared to normal backhand/frontend/websetvices Scenario that everybody already understand.
- diggan 2y ago
  With that mindset, should we just stop trying to improve anything regarding backend/frontend/webservices since "everybody already understand it"?
  - jgalt212 2y ago
    You're arguing change (a broadly defined term) is not necessarily bad, but the OP is arguing adding complexity (a type of change) is bad.
  - kimi 2y ago
    I second the OP - I'm not sure where the big prize is. I have a feeling that whomever wrote the article thinks there is a 10x (or 100x) improvement to be made, but I was not able to see it. I find the syntax very clunky, and I have been programming professional Clojure for at least 10 years. It reminds me of clojure.async - wonderful idea, but if you use the wrong sigil at the wrong place, you are dead in the water. Been there, done that - thanks but no thanks. OTOH I know who Nathan is, so I'm sure there is a gem hidden somewhere. But the article did not convince me that I should go the Rama way for my next webapp. I doubt the average JS programmer will be convinced. Maybe someone else will find the gem, polish it, and everybody will be using a derivative in 5 years.
    - bbor 2y ago
      TBF "this Clojure library has clunky syntax that makes it brittle" is a far more sophisticated and valid critique than "it's not built on Node so no one will use it" ;)
    - stingraycharles 2y ago
deleted 2y ago
[deleted]
waffletower 2y ago
Could I politely suggest more Clojure-like naming? `deframaop -> defop`. You can always `(require 'com.rpl.rama :as rama)` and invoke with `(rama/defop ...)` for the desired level of clarity and improved readability.
- nathanmarz 2y ago
  I named it like this so there would be consistency between deframaop and deframafn. Shortening deframafn like you suggest would be "deffn" or "deffunction", which would be very confusing. And I'd rather have deframaop + deframafn than defop + deframafn.
  - knubie 2y ago
    Why not rama/defop and rama/defn?
    - nathanmarz 2y ago
      That would make it so you can't do "use" on com.rpl.rama. Since Rama is a full language, doing a "use" on the namespace is generally preferred as otherwise you would have to write "rama/" everywhere, which is irritating. I also don't like overloading "defn" with something that's completely different. Also, a deframafn is more than a Clojure defn since it can emit to other output streams.
  - arunix 2y ago
    Why is it called Rama? (there may be an FAQ, but I couldn't find it)
    - nathanmarz 2y ago
      It's named after the Arthur C. Clarke book.
      - cryptonector 2y ago
        But do things in Rama the language come in threes?

106 points by nathanmarz 2y ago

This article is written for people who know Clojure, or at least the examples are. It might be nice to see the examples written in a non-LISP as well. E.g. (?<- (ops/explode [1 2 3 4] :> *v) (println "Val:" *v)) Is (I believe) the equivalent of Python's for element in [1, 2, 3, 4]: print(element)

nathanmarz 2y ago
That Python code has the same effect, but the equivalent Python in CPS would be: def explode(args, cont): for e in args: cont(e) (explode [1, 2, 3, 4], print)
- gpderetta 2y ago
  That's not in CPS form as cont returns into explode. Even if we treat the for loop in explode as an implementation detail, cont should take the next continuation as a parameter.
  - cryptonector 2y ago
    From TFA: > For every invoke, Rama determines if it’s executing a deframaop or a deframafn . If it’s a deframafn , then it invokes it just like how functions are invoked in Clojure by unrolling the stack frame with the return value. [...] So I think GP's Python example matches quite well with what Rama does in the Rama equivalent from TFA. Functions in Rama do return like normal functions in Clojure. It's Rama "ops" that call the continuation, but "ops" clearly _can_ return back to the caller, as that's how the whole generator aspect of Rama works -- it couldn't be any other way. So `println` is a function and it returns to its caller. This is very different from a Scheme that compiles to CPS such that every return is a continuation call. The point is that in such a Scheme you end up with all activation frames being on the heap and thus needing garbage collection. This is why delimited continuations were invented: Scheme-style call/cc continuations are just too expensive! But there is another optimization one can do: some returns can just unwind, while others can call a continuation. This is the optimization that Rama is going for near as I can see. It's a lot like Icon's `return`, which unwinds (in the Icon-to-C compiler anyways) and Icon's `suspend`, which calls the current continuation (again, in the Icon-to-C compiler case). This way you can return when you're not generating results -- when you're returning the last value, or when pruning (in Icon that's failure, which also unwinds, like `return`).

cryptonector 2y ago

The whole thing about distinguishing functions that yield once from all other functions is pretty clever. Reminds me of how in Icon a `return` is a C-like return that unwinds the stack while `suspend` is more like calling a continuation with the yielded value. In the old Icon-to-C compiler that's exactly how `return` and `suspend` worked. But Icon does not analyze function ("procedure") bodies to see if they only return, so the `return`-unwinds optimization is a run-time one. Whereas Rama does a compile-time optimization, which makes a lot of sense.

cryptonector 2y ago

Just hours (a day?) earlier there was this very interesting -and IMO relevant here- HN post: Can logic programming be liberated from predicates and backtracking? [pdf] (uni-kiel.de) https://news.ycombinator.com/item?id=41816545 That deals with backtracking, which is often implemented with continuations, as in TFA.

fire_lake 2y ago

What is the best introductory post to Rama right now? I would like to skip the marketing and understand how it compares to a wiring together Kafka, Spark, MySQL, etc.

deleted 2y ago
[deleted]
bbor 2y ago
AFAICT it's not actually published yet? It's a bit of a confusing situation because you can "download Rama" but it's not "full", whatever that means. See https://redplanetlabs.com/learn-rama https://redplanetlabs.com/learn-rama E: this download has "the full Rama API for use in simulated clusters within a single process", but also "Rama is currently available in a private beta." That's a highly unusual way to release what appears to be a Java library at the end of the day, but hopefully that's because it's unusually awesome! Looking forward to actual info some time in the future. I wonder if the "private beta" costs money...
- nathanmarz 2y ago
  What we've released publicly on our public Maven repository is a different build of Rama which can only be used for testing/experimentation in a single process. So it can't be used to run full clusters. It's API-equivalent to the full Rama build. This will change when we move out of private beta, when Rama will be free to use for production for small-scale applications.
  - grounder 2y ago
    Do you have an estimate of when you'll be out of private beta and available? Can you share any more about pricing or what you consider to be a "small-scale application". Thanks!
    - nathanmarz 2y ago
      We're aiming to be out of private beta early next year. "Small-scale" basically means the kind of scale a single Postgres node + application server can handle.

beders 2y ago

> A Rama operation does not return a value to its caller. It emits values to its continuation. This is a critical distinction, as part of what makes Rama operations more general than functions is how they can emit multiple times, not emit at all, or emit asynchronously.

thom 2y ago
Which is obviously very similar to how transducers already work in Clojure, but they still lack some of concurrency options of reducers. Getting all this on a smart, distributed runtime seems very promising.
- mschaef 2y ago
  > Getting all this on a smart, distributed runtime seems very promising. Hopefully it is. This CPS article is the first of the Rama blog posts where it seemed like there might be something there. The earlier posts - "I built Twitter scale Twitter in 10Kloc" - were never really all that convincing. The thing they claimed to have built was too ambitious a claim.
  - thom 2y ago
    Oh I think there’s a lot of good stuff baked in there. The big idea downstream is that you have incrementally calculated, indexed data structures to query all the results of this fancy CPS logic. It’s all slightly esoteric even coming from a Clojure background but it ticks every box I want from a modern data platform, short of speaking SQL.
- dig1 2y ago
  There is a library called tesser [1] (by Jepsen/Riemann author) that behaves like parallel transducers with more Clojure "native" syntax. With transducers, you have to use "comp," and with tesser, you use "->>" as you'd use with lazy functions. Sadly, tesser is not advertised as it should; I find it much more flexible than transducers. E.g. you could parallelize tesser code over Spark/Hadoop cluster. [1] https://github.com/aphyr/tesser https://github.com/aphyr/tesser

moomin 2y ago

I feel like CPS is one of those tar pits smart developers fall into. It’s just a fundamentally unfriendly API, like mutexes. We saw this with node as well: eventually the language designers just sighed and added promises. You’re better off with an asynchronous result stream, which is equivalent in power but much easier to reason about. C#’s got IAsyncEnumerable, I know that Rust is working on designing something similar. Even then, it can be hard to analyse the behaviour of multiple levels of asynchronous streams and passing pieces of information from the top level to the bottom level like a tag is a pain in the neck.

neonsunset 2y ago
I found myself liking the F# way of consuming/producing IAsyncEnumerable's with taskseq. It's very terse and looks nice: https://github.com/fsprojects/FSharp.Control.TaskSeq?tab=readme-ov-file#examples https://github.com/fsprojects/FSharp.Control.TaskSeq?tab=rea...
- moomin 2y ago
  Apparently Microsoft hold a patent on “yield!”, which makes it all the more frustrating that they haven’t included it in C#.
  - neonsunset 2y ago
    In F#, yield! is a computation expression, C#'s yield within methods that return IAsyncEnumerable<T> works more or less the same way.
cryptonector 2y ago
IMO TFA makes much of CPS because that's how they chose to implement generators, but the main thing about the language is that it has generators. Now having generators is nothing new, but I don't want to take too much away from TFA, as there are some interesting things there. I'll limit myself to pointing out that the Icon programming language had generators and pervasive backtracking using CPS in the Icon-to-C compiler, and that other languages with generators and pervasive backtracking have been implemented with CPS as well as with bytecode VMs that don't have any explicit (internally) continuations. Examples include Prolog, Icon, and jq, to name just three, and now of course, Rama.

thunkingdeep 2y ago

I’ve always thought that CPS is a good barometer for finding out whether a developer is talented or whether they THINK they’re talented enough to design and/or implement these kinds of compiler components. This kind of thing but definitely CPS in particular is so much trickier to nail down than it initially seems if you’ve written a compiler before. Up there in difficulty with automatic parallelization and loop transforming. I tried to write a very small POC lisp once with an idea to have all vectors of known sizes get map’d in parallel and I never could nail it down. Kudos to all involved. Clojure is such a mind bending tool. God only knows what it takes these people to maintain the guts of it all.

kamma4434 2y ago

I don’t want to be a party pooper, but from my very cursory look at the page, I don’t think this will go much farther than a very small community. I feel like you’re adding a lot of complexity compared to normal backhand/frontend/websetvices Scenario that everybody already understand.

diggan 2y ago
With that mindset, should we just stop trying to improve anything regarding backend/frontend/webservices since "everybody already understand it"?
- jgalt212 2y ago
  You're arguing change (a broadly defined term) is not necessarily bad, but the OP is arguing adding complexity (a type of change) is bad.
- kimi 2y ago
  I second the OP - I'm not sure where the big prize is. I have a feeling that whomever wrote the article thinks there is a 10x (or 100x) improvement to be made, but I was not able to see it. I find the syntax very clunky, and I have been programming professional Clojure for at least 10 years. It reminds me of clojure.async - wonderful idea, but if you use the wrong sigil at the wrong place, you are dead in the water. Been there, done that - thanks but no thanks. OTOH I know who Nathan is, so I'm sure there is a gem hidden somewhere. But the article did not convince me that I should go the Rama way for my next webapp. I doubt the average JS programmer will be convinced. Maybe someone else will find the gem, polish it, and everybody will be using a derivative in 5 years.
  - bbor 2y ago
    TBF "this Clojure library has clunky syntax that makes it brittle" is a far more sophisticated and valid critique than "it's not built on Node so no one will use it" ;)
  - stingraycharles 2y ago

deleted 2y ago

[deleted]

waffletower 2y ago

Could I politely suggest more Clojure-like naming? `deframaop -> defop`. You can always `(require 'com.rpl.rama :as rama)` and invoke with `(rama/defop ...)` for the desired level of clarity and improved readability.

nathanmarz 2y ago
I named it like this so there would be consistency between deframaop and deframafn. Shortening deframafn like you suggest would be "deffn" or "deffunction", which would be very confusing. And I'd rather have deframaop + deframafn than defop + deframafn.
- knubie 2y ago
  Why not rama/defop and rama/defn?
  - nathanmarz 2y ago
    That would make it so you can't do "use" on com.rpl.rama. Since Rama is a full language, doing a "use" on the namespace is generally preferred as otherwise you would have to write "rama/" everywhere, which is irritating. I also don't like overloading "defn" with something that's completely different. Also, a deframafn is more than a Clojure defn since it can emit to other output streams.
- arunix 2y ago
  Why is it called Rama? (there may be an FAQ, but I couldn't find it)
  - nathanmarz 2y ago
    It's named after the Arthur C. Clarke book.
    - cryptonector 2y ago
      But do things in Rama the language come in threes?