4 ms·

This is intriguing, and solves real world problems that I have come into writing Clojure. On the other hand, it is complicated, taking up to four arguments. It

This is intriguing, and solves real world problems that I have come into writing Clojure. On the other hand, it is complicated, taking up to four arguments. It reminds me of Common Lisp's `loop` macro, able to handle complex and interesting situations, but has its own chapter in "Common Lisp, the Language" because it is so complicated. Is it really helping solve the accidental complexity problem at that point? Perhaps. I use the loop macro all the time, so it does bear thinking about. The OP also speaks of decreasing indentation as a goal instead of decreasing cyclomatic complexity. It is true that one tracks with the other, and I'll be the first to say that I too am an 80 character wide masochist, but optimizing code structure around decreasing indentation is dubious at best.

justinhj 4y ago
I don’t really find loop complicated because the basic cases are straightforward. The advanced cases are discovered complexity when you deliberately wade into the depths to solve some weird problem.
zdragnar 4y ago
> but optimizing code structure around decreasing indentation is dubious at best But when you get improved performance and decreased indentation, everyone wins :)
crdrost 4y ago
In a language that I don't quite have the cojones to write, the ideal API for reduce comes from Haskell’s Data.Foldable, where it is called foldMap. If you don't know Haskell the idea is simple, it is that the essence of MapReduce is for you to have an aggregating data type A with two operations, empty(): A combine(A, A): A such that combine(combine(x,y), z) = combine(x, combine(y, z)) combine(empty(), x) = x combine(x, empty()) = x The requirement that `combine(x, y) = combine(y, x)` is not strictly necessary if the MapReduce implementation maintains shards and shard indices and is careful to only merge neighboring shards, the first property is then sufficient. To make this into MapReduce, you need your underlying data type X and an injection, (x: X) → A. This is the mapping part of MapReduce. So for example “First10” is such an aggregation; given the first 10 (or fewer) in shard 1 and the first 10 (or fewer) in shard 2, you can form the first 10 (or fewer) in shards 1+2. The “map” function can use the empty value to provide “filtering,” so you get “the first 10 which satisfy this logical condition” directly. Sum is another such aggregation, Count is a specialization of it, Mean is technically just a pair of Sum and Count. Oh, did I forget to mention? Given two aggregations you can easily produce an aggregation that does both. “give me the sum of this field and the first 10 records summed and the count of terms” ... One (parallelizable) pass for all three, because the aggregations compose. So a macro API is possible where this is instead something like, (I don't know Clojure) (loopM [x [1 2 3 4 5 6 7]] (if (< x 5) [(sum x) (count) (sum (* x x)) (first 3 x)])) #=> [15, 5, 55, [1, 2, 3]]
geokon 4y ago
You can argue if it decreases complexity or not - I think it's hard to judge until you use it regularly bc right now at first sight, for me at least, it has a strong unfamiliarity to it. It looks kinda long and complicated and doesn't play well with the threading macro (in comments this is discussed) If it's a neat trick macro you use once every two months then you may never get used to it But the fact that it yields a clear performance gain is a more objective "win". It's not just stylistic at this point and I'm inclinded to give it a shot

justinhj 4y ago

I don’t really find loop complicated because the basic cases are straightforward. The advanced cases are discovered complexity when you deliberately wade into the depths to solve some weird problem.

zdragnar 4y ago

> but optimizing code structure around decreasing indentation is dubious at best But when you get improved performance and decreased indentation, everyone wins :)

crdrost 4y ago

In a language that I don't quite have the cojones to write, the ideal API for reduce comes from Haskell’s Data.Foldable, where it is called foldMap. If you don't know Haskell the idea is simple, it is that the essence of MapReduce is for you to have an aggregating data type A with two operations, empty(): A combine(A, A): A such that combine(combine(x,y), z) = combine(x, combine(y, z)) combine(empty(), x) = x combine(x, empty()) = x The requirement that `combine(x, y) = combine(y, x)` is not strictly necessary if the MapReduce implementation maintains shards and shard indices and is careful to only merge neighboring shards, the first property is then sufficient. To make this into MapReduce, you need your underlying data type X and an injection, (x: X) → A. This is the mapping part of MapReduce. So for example “First10” is such an aggregation; given the first 10 (or fewer) in shard 1 and the first 10 (or fewer) in shard 2, you can form the first 10 (or fewer) in shards 1+2. The “map” function can use the empty value to provide “filtering,” so you get “the first 10 which satisfy this logical condition” directly. Sum is another such aggregation, Count is a specialization of it, Mean is technically just a pair of Sum and Count. Oh, did I forget to mention? Given two aggregations you can easily produce an aggregation that does both. “give me the sum of this field and the first 10 records summed and the count of terms” ... One (parallelizable) pass for all three, because the aggregations compose. So a macro API is possible where this is instead something like, (I don't know Clojure) (loopM [x [1 2 3 4 5 6 7]] (if (< x 5) [(sum x) (count) (sum (* x x)) (first 3 x)])) #=> [15, 5, 55, [1, 2, 3]]

geokon 4y ago

You can argue if it decreases complexity or not - I think it's hard to judge until you use it regularly bc right now at first sight, for me at least, it has a strong unfamiliarity to it. It looks kinda long and complicated and doesn't play well with the threading macro (in comments this is discussed) If it's a neat trick macro you use once every two months then you may never get used to it But the fact that it yields a clear performance gain is a more objective "win". It's not just stylistic at this point and I'm inclinded to give it a shot