3 ms·

Seems like it's going to be a tough sell to get people to want to write (tc/select-rows ds #(> (% "year") 2008)) instead of filter(ds, year > 2008)

by zmmmmm 7mo ago

Seems like it's going to be a tough sell to get people to want to write (tc/select-rows ds #(> (% "year") 2008)) instead of filter(ds, year > 2008) They seem to ignore the existance of Spark, so even if you specifically want to use JVM it feels clearer and simpler: ds.filter(r => r.year > 2008)

condwanaland 7mo ago
Couldn't agree more. R and dplyrs ability to pass column names as unquoted objects actually reduces cognitive load for new people so much (pure anecdata, nothing to back this up except lots of teaching people). And that's on top of the vastly simpler syntax compared to what's being shown here
- iLemming 7mo ago
  > vastly simpler syntax I've been programming for decades. I've used dozens of different, at times enormously esoteric languages. At one point I built ERPs in a language where operators were abbreviated Russian terms. After just a few years of using Lisp dialects I am absolutely convinced - there's no simpler and more readable syntax than of Lisp's. Anyone who doesn't see that, in my eyes just not made the distinction between familiarity and simplicity. They're measuring how quickly their eyes can parse something they've already seen a thousand times, and calling that readability. But readability isn't recognition speed - it's the cognitive distance between the code and the computation it describes. And on that measure, Lisp is essentially lossless. There's no syntactic residue. No ceremony the language demands for its own sake. What you write is the structure of the thing, all the way down. "You get used to it. I don't even see the code. All I see is blonde, brunette, redhead..." I don't look at Matrix feeling puzzled anymore. I see the truth. People who bounce off the parentheses are reacting to something real: it doesn't look like what they already know. But that's not the language failing them. That's just the last bit of the old syntax dying. Give it a few months of structural editing and a proper REPL workflow, and you won't see parentheses anymore - you'll see shape. You'll see depth. And going back to anything else will feel like someone handed you a map drawn in crayon and called it a feature.
  - upghost 7mo ago
    > familiarity vs simplicity Love this, I've never heard it put that way before.
    - iLemming 7mo ago
      Rich Hickey did in "Simple made easy" talk.
  - zmmmmm 7mo ago
    Even though I'm firmly in the familiarity camp, I love this description of what it's like to be fully lisp-pilled. I definitely want to try it out one day.
    - iLemming 7mo ago
      One doesn't need to dance around weird circles to give it a try. VSCode's Calva extension has great "Quickstart guide". Or you can just install Clojure, run it - it drops you into a REPL. Once you ready to get serious, grok some basic structural editing commands (grabbing an expression and moving it somewhere else would suffice) and REPL-driven workflow (Lispers typically don't type into REPL, they eval things directly where the source is)
geokon 7mo ago
In my experience the advantage comes when you have a few more lines of code The Clojure pipelining makes code much more readable. Granted dplyr has them too, but tidyverse pipes always felt like a hack on top of R (though my experience is dated here). While in Clojure I always feel like I'm playing with the fundamental language data-types/protocols. I can extend things in any way I want
aphyr 7mo ago
You're right, that is longer! I get why though; `filter` is a clojure.core function name people don't necessarily feel comfortable shadowing, and the Clojure and Spark versions make it clear what's a symbol in local scope versus a field in the dataset. I don't think it'd be hard to make a little wrapper for this sort of thing though! Here's an example which turns any symbols not in local scope into field lookups on an implicit row variable. (require '[clojure.walk :refer [postwalk]]) (defmacro filter [ds & anaphoric-pred] (let [row-name (gensym 'row) pred (postwalk (fn [form] (if (and (symbol? form) (nil? (resolve form))) `(get ~row-name ~(str form)) form)) anaphoric-pred)] `(tc/select-rows ds (fn [~row-name] ~@pred)))) Now you can write (filter ds (> year 2008)) And it'll expand to the ts form: (pprint (macroexpand '(filter ds (> year 2008)))) => (tc/select-rows ds (fn [row2411] (> (get row2411 "year") 2008)))
hrrld 7mo ago
In practice we use `ds/filter-column` and `ds/filter` much more than `select-rows`. The sell isn't about typing a few more or a few less characters, it's about doing data science functionally.
iLemming 7mo ago
[dead]

by zmmmmm 7mo ago

condwanaland 7mo ago

Couldn't agree more. R and dplyrs ability to pass column names as unquoted objects actually reduces cognitive load for new people so much (pure anecdata, nothing to back this up except lots of teaching people). And that's on top of the vastly simpler syntax compared to what's being shown here

iLemming 7mo ago
> vastly simpler syntax I've been programming for decades. I've used dozens of different, at times enormously esoteric languages. At one point I built ERPs in a language where operators were abbreviated Russian terms. After just a few years of using Lisp dialects I am absolutely convinced - there's no simpler and more readable syntax than of Lisp's. Anyone who doesn't see that, in my eyes just not made the distinction between familiarity and simplicity. They're measuring how quickly their eyes can parse something they've already seen a thousand times, and calling that readability. But readability isn't recognition speed - it's the cognitive distance between the code and the computation it describes. And on that measure, Lisp is essentially lossless. There's no syntactic residue. No ceremony the language demands for its own sake. What you write is the structure of the thing, all the way down. "You get used to it. I don't even see the code. All I see is blonde, brunette, redhead..." I don't look at Matrix feeling puzzled anymore. I see the truth. People who bounce off the parentheses are reacting to something real: it doesn't look like what they already know. But that's not the language failing them. That's just the last bit of the old syntax dying. Give it a few months of structural editing and a proper REPL workflow, and you won't see parentheses anymore - you'll see shape. You'll see depth. And going back to anything else will feel like someone handed you a map drawn in crayon and called it a feature.
- upghost 7mo ago
  > familiarity vs simplicity Love this, I've never heard it put that way before.
  - iLemming 7mo ago
    Rich Hickey did in "Simple made easy" talk.
- zmmmmm 7mo ago
  Even though I'm firmly in the familiarity camp, I love this description of what it's like to be fully lisp-pilled. I definitely want to try it out one day.
  - iLemming 7mo ago
    One doesn't need to dance around weird circles to give it a try. VSCode's Calva extension has great "Quickstart guide". Or you can just install Clojure, run it - it drops you into a REPL. Once you ready to get serious, grok some basic structural editing commands (grabbing an expression and moving it somewhere else would suffice) and REPL-driven workflow (Lispers typically don't type into REPL, they eval things directly where the source is)

geokon 7mo ago

In my experience the advantage comes when you have a few more lines of code The Clojure pipelining makes code much more readable. Granted dplyr has them too, but tidyverse pipes always felt like a hack on top of R (though my experience is dated here). While in Clojure I always feel like I'm playing with the fundamental language data-types/protocols. I can extend things in any way I want

aphyr 7mo ago

You're right, that is longer! I get why though; `filter` is a clojure.core function name people don't necessarily feel comfortable shadowing, and the Clojure and Spark versions make it clear what's a symbol in local scope versus a field in the dataset. I don't think it'd be hard to make a little wrapper for this sort of thing though! Here's an example which turns any symbols not in local scope into field lookups on an implicit row variable. (require '[clojure.walk :refer [postwalk]]) (defmacro filter [ds & anaphoric-pred] (let [row-name (gensym 'row) pred (postwalk (fn [form] (if (and (symbol? form) (nil? (resolve form))) `(get ~row-name ~(str form)) form)) anaphoric-pred)] `(tc/select-rows ds (fn [~row-name] ~@pred)))) Now you can write (filter ds (> year 2008)) And it'll expand to the ts form: (pprint (macroexpand '(filter ds (> year 2008)))) => (tc/select-rows ds (fn [row2411] (> (get row2411 "year") 2008)))

hrrld 7mo ago

In practice we use `ds/filter-column` and `ds/filter` much more than `select-rows`. The sell isn't about typing a few more or a few less characters, it's about doing data science functionally.

iLemming 7mo ago

[dead]