4 ms·

I second the OP - I'm not sure where the big prize is. I have a feeling that whomever wrote the article thinks there is a 10x (or 100x) improvement to be made,

by kimi 2y ago

I second the OP - I'm not sure where the big prize is. I have a feeling that whomever wrote the article thinks there is a 10x (or 100x) improvement to be made, but I was not able to see it. I find the syntax very clunky, and I have been programming professional Clojure for at least 10 years. It reminds me of clojure.async - wonderful idea, but if you use the wrong sigil at the wrong place, you are dead in the water. Been there, done that - thanks but no thanks. OTOH I know who Nathan is, so I'm sure there is a gem hidden somewhere. But the article did not convince me that I should go the Rama way for my next webapp. I doubt the average JS programmer will be convinced. Maybe someone else will find the gem, polish it, and everybody will be using a derivative in 5 years.

bbor 2y ago
TBF "this Clojure library has clunky syntax that makes it brittle" is a far more sophisticated and valid critique than "it's not built on Node so no one will use it" ;)
stingraycharles 2y ago
I would have expected better from HN that to shoot down smart people tinkering with potentially elegant solutions to complex problems. It’s something we should embrace. Having said that, as a long term Clojure developer myself, I’m also not a big fan of this approach myself (I try to avoid libraries that use a lot of macros, and instead prefer a more “data driven” approach, which is also why I’m not a fan of spec), but I’m not one to judge.
- oldpersonintx 2y ago
  [dead]
eduction 2y ago
> It reminds me of clojure.async - wonderful idea, but if you use the wrong sigil at the wrong place, you are dead in the water. Isn’t that how any programming works? If you call the wrong function, pass the wrong var, typo a hash key etc etc the whole thing can blow up. Not sure how it’s a knock on core.async that you have to use the right macro or function in the right place. Are there async libraries that let you typo the name of their core components? (And yes some of the macros are named like “<!”, is that naming the issue?)
- ValentinA23 2y ago
  No it is different because libraries such as core.async or Rama rely on inversion of control [1]: the framework is in charge of the control flow and code fed to the framework will be executed by some kind of black box. To achieve this, these frameworks build their own machinery on top of existing core facilities (normal functions, call stacks, etc) to implement similar concepts (rama ops for instance) one level above. The real issues arise when something goes wrong. If you're lucky you'll get an exception but it won't tell you anything about the process you described at the framework level using the abstractions it offers (like core.async channels). The exception will just tell you how the framework's "executor" failed at running some particular abstraction. You'll be able to follow the flow of the executor but not the flow of the process it executes. In other words the exception is describing what is happening one level of abstraction too low. If you're not lucky, the code you wrote will get stuck somewhere, but issuing a ^C from your REPL will have no effect because the problematic code runs in another thread or in another machine. The forced halting happens at the wrong level of abstraction too. These are serious obstacles because your only recourse is to bisect your code by commenting out portions of it just to identify where the problem arises. I personally have resorted to writing my own half-baked core.async debugger, implementing instrumentation of core.async primitives gradually, as I need them. Having said that, I don't think this is a fatal flaw of inversion of control, and in fact looking at the problem closely I don't think the root issue is that they come with their own black box execution systems. Those are not black boxes, as shown by the stack traces these frameworks produce which give a clear picture of their internals, they are grey boxes leaking info about one execution level into another level. And this happens because these frameworks (talking about core.async specifically, maybe this isn't the case with Rama) do not but should come with their own exception system to handle errors and forced interruption. Lacking these facilities they fallback on spitting a trace about the executor instead of the executed process. What does implementing a new exception system entails ? Case 1, your IoC framework does not modify the shape of execution, it' still a call-tree and there is a unique call-path leading to the error point, but it changes how execution happens, for instance it dislocates the code by running it on different machines/threads. Then the goal is to aggregate those sparse code points that constitute the call-path at the framework's abstraction level. You'll deal with "synthetic exceptions" that still have the shape of a classical exception with a stack of function calls, except that these calls are in succession only from the framework semantics; at a lower-level, they are not. Case 2, the framework also changes the shape of execution, you're not dealing with a mere call-tree anymore, you're using a dataflow, a DAG. There is not a single call-path up to the error point anymore, but potentially many. You need to replace the stack in your exception type by a graph-shaped trace in addition to handling sparse code point aggregation as in case 1. Aggregation to put in succession stack trace elements that are distant one abstraction level lower and to hide parts of the code that are not relevant at this level. And new exception types to account for different execution shapes. In addition to these two requirement, you need to find a way to stitch different exception types together to bridge the gap between the executor process and the executed process as well as between the executed process and callbacks/continuations/predicates the user may provide using the native language execution semantics. [1] https://en.wikipedia.org/wiki/Inversion_of_control https://en.wikipedia.org/wiki/Inversion_of_control
nathanmarz 2y ago
Well, this article is to help people understand just Rama's dataflow API, as opposed to an introduction to Rama for backend development. Rama does have a learning curve. If you think its API is "clunky", then you just haven't invested any time in learning and tinkering with it. Here are two examples of how elegant it is: This one does atomic bank transfers with cross-partition transactions, as well as keeping track of everyone's activity: https://github.com/redplanetlabs/rama-demo-gallery/blob/master/src/main/clj/rama/gallery/bank_transfer_module.clj https://github.com/redplanetlabs/rama-demo-gallery/blob/mast... This one does scalable time-series analytics, aggregating across multiple granularities and minimizing reads at query time by intelligently choosing buckets across multiple granularities: https://github.com/redplanetlabs/rama-demo-gallery/blob/master/src/main/clj/rama/gallery/time_series_module.clj https://github.com/redplanetlabs/rama-demo-gallery/blob/mast... There are equivalent Java examples in that repository as well.
- goostavos 2y ago
  This question is probably obvious if I knew what a microbatch or topology or depot was, but as a Rama outsider, is there a good high level mental model for what makes the cross-partition transactions work? From the comments that mention queuing and transaction order, is serialized isolation a good way to imagine what's going on behind the scenes or is that way off base?
  - nathanmarz 2y ago
    A depot is a distributed log of events that you append to as a user. In this case, there's one depot for appending "deposits" (an increase to one user's account) and another depot for appending "transfers" (an attempt to move funds from one account to another). A microbatch topology is a coordinated computation across the entire cluster. It reads a fixed amount of data from each partition of each depot and processes it all in batch. Changes don't become visible until all computation is finished across all partitions. Additionally, a microbatch topology always starts computation with the PStates (the indexed views that are like databases) at the state of the last microbatch. This means a microbatch topology has exactly-once semantics – it may need to reprocess if there's a failure (like a node dying), but since it always starts from the same state the results are as if there were no failures at all. Finally, all events on a partition execute in sequence. So when the code checks if the user has the required amount of funds for the transfer, there's no possibility of a concurrent deduction that would create a race condition that would invalidate the check. So in this code, it first checks if the user has the required amount of funds. If so, it deducts that amount. This is safe because it's synchronous with the check. The code then changes to the partition storing the funds for the target user and adds that amount to their account. If they're receiving multiple transfers, those will be added one at a time because only one event runs at a time on a partition. To summarize: - Colocated computation and storage eliminates race conditions - Microbatch topologies have exactly-once semantics due to starting computation at the exact same state every time regardless of failures or how much it progressed on the last attempt The docs have more detail on how this works: https://redplanetlabs.com/docs/~/microbatch.html#_operation_and_fault_tolerance https://redplanetlabs.com/docs/~/microbatch.html#_operation_...

by kimi 2y ago

bbor 2y ago

TBF "this Clojure library has clunky syntax that makes it brittle" is a far more sophisticated and valid critique than "it's not built on Node so no one will use it" ;)

stingraycharles 2y ago

I would have expected better from HN that to shoot down smart people tinkering with potentially elegant solutions to complex problems. It’s something we should embrace. Having said that, as a long term Clojure developer myself, I’m also not a big fan of this approach myself (I try to avoid libraries that use a lot of macros, and instead prefer a more “data driven” approach, which is also why I’m not a fan of spec), but I’m not one to judge.

oldpersonintx 2y ago
[dead]

eduction 2y ago

> It reminds me of clojure.async - wonderful idea, but if you use the wrong sigil at the wrong place, you are dead in the water. Isn’t that how any programming works? If you call the wrong function, pass the wrong var, typo a hash key etc etc the whole thing can blow up. Not sure how it’s a knock on core.async that you have to use the right macro or function in the right place. Are there async libraries that let you typo the name of their core components? (And yes some of the macros are named like “<!”, is that naming the issue?)

ValentinA23 2y ago
No it is different because libraries such as core.async or Rama rely on inversion of control [1]: the framework is in charge of the control flow and code fed to the framework will be executed by some kind of black box. To achieve this, these frameworks build their own machinery on top of existing core facilities (normal functions, call stacks, etc) to implement similar concepts (rama ops for instance) one level above. The real issues arise when something goes wrong. If you're lucky you'll get an exception but it won't tell you anything about the process you described at the framework level using the abstractions it offers (like core.async channels). The exception will just tell you how the framework's "executor" failed at running some particular abstraction. You'll be able to follow the flow of the executor but not the flow of the process it executes. In other words the exception is describing what is happening one level of abstraction too low. If you're not lucky, the code you wrote will get stuck somewhere, but issuing a ^C from your REPL will have no effect because the problematic code runs in another thread or in another machine. The forced halting happens at the wrong level of abstraction too. These are serious obstacles because your only recourse is to bisect your code by commenting out portions of it just to identify where the problem arises. I personally have resorted to writing my own half-baked core.async debugger, implementing instrumentation of core.async primitives gradually, as I need them. Having said that, I don't think this is a fatal flaw of inversion of control, and in fact looking at the problem closely I don't think the root issue is that they come with their own black box execution systems. Those are not black boxes, as shown by the stack traces these frameworks produce which give a clear picture of their internals, they are grey boxes leaking info about one execution level into another level. And this happens because these frameworks (talking about core.async specifically, maybe this isn't the case with Rama) do not but should come with their own exception system to handle errors and forced interruption. Lacking these facilities they fallback on spitting a trace about the executor instead of the executed process. What does implementing a new exception system entails ? Case 1, your IoC framework does not modify the shape of execution, it' still a call-tree and there is a unique call-path leading to the error point, but it changes how execution happens, for instance it dislocates the code by running it on different machines/threads. Then the goal is to aggregate those sparse code points that constitute the call-path at the framework's abstraction level. You'll deal with "synthetic exceptions" that still have the shape of a classical exception with a stack of function calls, except that these calls are in succession only from the framework semantics; at a lower-level, they are not. Case 2, the framework also changes the shape of execution, you're not dealing with a mere call-tree anymore, you're using a dataflow, a DAG. There is not a single call-path up to the error point anymore, but potentially many. You need to replace the stack in your exception type by a graph-shaped trace in addition to handling sparse code point aggregation as in case 1. Aggregation to put in succession stack trace elements that are distant one abstraction level lower and to hide parts of the code that are not relevant at this level. And new exception types to account for different execution shapes. In addition to these two requirement, you need to find a way to stitch different exception types together to bridge the gap between the executor process and the executed process as well as between the executed process and callbacks/continuations/predicates the user may provide using the native language execution semantics. [1] https://en.wikipedia.org/wiki/Inversion_of_control https://en.wikipedia.org/wiki/Inversion_of_control

nathanmarz 2y ago

Well, this article is to help people understand just Rama's dataflow API, as opposed to an introduction to Rama for backend development. Rama does have a learning curve. If you think its API is "clunky", then you just haven't invested any time in learning and tinkering with it. Here are two examples of how elegant it is: This one does atomic bank transfers with cross-partition transactions, as well as keeping track of everyone's activity: https://github.com/redplanetlabs/rama-demo-gallery/blob/master/src/main/clj/rama/gallery/bank_transfer_module.clj https://github.com/redplanetlabs/rama-demo-gallery/blob/mast... This one does scalable time-series analytics, aggregating across multiple granularities and minimizing reads at query time by intelligently choosing buckets across multiple granularities: https://github.com/redplanetlabs/rama-demo-gallery/blob/master/src/main/clj/rama/gallery/time_series_module.clj https://github.com/redplanetlabs/rama-demo-gallery/blob/mast... There are equivalent Java examples in that repository as well.

goostavos 2y ago
This question is probably obvious if I knew what a microbatch or topology or depot was, but as a Rama outsider, is there a good high level mental model for what makes the cross-partition transactions work? From the comments that mention queuing and transaction order, is serialized isolation a good way to imagine what's going on behind the scenes or is that way off base?
- nathanmarz 2y ago
  A depot is a distributed log of events that you append to as a user. In this case, there's one depot for appending "deposits" (an increase to one user's account) and another depot for appending "transfers" (an attempt to move funds from one account to another). A microbatch topology is a coordinated computation across the entire cluster. It reads a fixed amount of data from each partition of each depot and processes it all in batch. Changes don't become visible until all computation is finished across all partitions. Additionally, a microbatch topology always starts computation with the PStates (the indexed views that are like databases) at the state of the last microbatch. This means a microbatch topology has exactly-once semantics – it may need to reprocess if there's a failure (like a node dying), but since it always starts from the same state the results are as if there were no failures at all. Finally, all events on a partition execute in sequence. So when the code checks if the user has the required amount of funds for the transfer, there's no possibility of a concurrent deduction that would create a race condition that would invalidate the check. So in this code, it first checks if the user has the required amount of funds. If so, it deducts that amount. This is safe because it's synchronous with the check. The code then changes to the partition storing the funds for the target user and adds that amount to their account. If they're receiving multiple transfers, those will be added one at a time because only one event runs at a time on a partition. To summarize: - Colocated computation and storage eliminates race conditions - Microbatch topologies have exactly-once semantics due to starting computation at the exact same state every time regardless of failures or how much it progressed on the last attempt The docs have more detail on how this works: https://redplanetlabs.com/docs/~/microbatch.html#_operation_and_fault_tolerance https://redplanetlabs.com/docs/~/microbatch.html#_operation_...