4 ms·

WL doesn't yet have a laziness/streaming/reducing framework, but the prototype we're working on uses 'operator forms' like Select, Map, GroupBy and so on in the

WL doesn't yet have a laziness/streaming/reducing framework, but the prototype we're working on uses 'operator forms' like Select, Map, GroupBy and so on in the way you describe. I don't think the exact details are the same, because our operators don't actually evaluate to transformers (they remain totally symbolic). Rather, the conversion of composed operators to an actual reducer pipeline happens lazily 'at the right time', which I think will make optimization a bit easier to express.

jamii 12y ago
So if I understand correctly, you would arrive at a symbolic expression like (->> (map f) (map g) x) and directly rewrite it to (->> (map (f . g)) x), rather than having to manipulate the resulting data-structures?
- deleted 12y ago
  [deleted]
- taliesinb 12y ago
  Yeah, it's pretty easy to turn Map[#^2&] /* Select[PrimeQ] /* Select[OddQ] into Select[OddQ[#] && PrimeQ[#]&] /* Map[#^2&] and so on via rules that look like: Select[s1_] /* Select[s2_] :> Select[s1[#] && s2[#]&] Map[f_ ? SideEffectFreeQ] /* s_Select :> s /* Map[f] Map[Extract[p_Integer | p_Key]] :> Extract[{All, p}] I'm already doing a bunch of that for Dataset. But in reality you need to move to a DSL for complex enough rules, and then Clojure and WL will be on the same footing (Clojure even a bit stronger, maybe, WL doesn't really have a proper macro system). Does Clojure core do or allow for any of that kind of optimization already?
  - malisper 12y ago
    I don't know about Clojure, but Common Lisp provides something called Compiler Macros[0]. They basically allow the programmer to define two versions of a procedure. One that is a macro, and one that is an actual procedure. The macro will expand only at compile time (it can also specify to call the procedure instead of expanding), and the procedure will be called only at run time. I suggest you look at [0] for some examples of how it is possible to optimize something as simple as squaring a number. [0] http://clhs.lisp.se/Body/m_define.htm http://clhs.lisp.se/Body/m_define.htm

jamii 12y ago

So if I understand correctly, you would arrive at a symbolic expression like (->> (map f) (map g) x) and directly rewrite it to (->> (map (f . g)) x), rather than having to manipulate the resulting data-structures?

deleted 12y ago
[deleted]
taliesinb 12y ago
Yeah, it's pretty easy to turn Map[#^2&] /* Select[PrimeQ] /* Select[OddQ] into Select[OddQ[#] && PrimeQ[#]&] /* Map[#^2&] and so on via rules that look like: Select[s1_] /* Select[s2_] :> Select[s1[#] && s2[#]&] Map[f_ ? SideEffectFreeQ] /* s_Select :> s /* Map[f] Map[Extract[p_Integer | p_Key]] :> Extract[{All, p}] I'm already doing a bunch of that for Dataset. But in reality you need to move to a DSL for complex enough rules, and then Clojure and WL will be on the same footing (Clojure even a bit stronger, maybe, WL doesn't really have a proper macro system). Does Clojure core do or allow for any of that kind of optimization already?
- malisper 12y ago
  I don't know about Clojure, but Common Lisp provides something called Compiler Macros[0]. They basically allow the programmer to define two versions of a procedure. One that is a macro, and one that is an actual procedure. The macro will expand only at compile time (it can also specify to call the procedure instead of expanding), and the procedure will be called only at run time. I suggest you look at [0] for some examples of how it is possible to optimize something as simple as squaring a number. [0] http://clhs.lisp.se/Body/m_define.htm http://clhs.lisp.se/Body/m_define.htm