12 ms·

Specter – Clojure API for immutable programming (2017)

152 points by amgreg 6y ago

boxed 6y ago
I invented instar (https://github.com/boxed/Instar https://github.com/boxed/Instar) which is quite similar to specter (imo a little bit nicer) because I also thought the built in API was weirdly clunky in these respects.
- didibus 6y ago
  What's the performance vis-a-vis Specter?
  - boxed 6y ago
    I have never run a benchmark and the specter guy really cares about it so instar is probably way worse :) I was more trying to show that the API could be better and hoping this would put pressure on core devs to do it properly.
- ledgerdev 6y ago
  Very nice syntax, looks great.
- SomeHacker44 6y ago
  I have used both Spectre and Instar. I prefer Instar for straight forward simple transforms, but Spectre can do more complex ones. Spectre has a more difficult API to wrap your head around and that is its biggest downside, IMO. You really need to dedicate time to learn it rather than try to pull it out on demand.
john-shaffer 6y ago
Specter is fantastic. It makes transformations on deeply nested data structures easy. Want to transform all the "datetimes" from JSON data into actual datetimes (or vice versa)? It takes about five minutes if you have to look everything up. Works with ClojureScript too. Some production code: (defn transform-tagged-values [x] (sp/transform (sp/walker tr/tagged-value?) tagged-value->cljs x)) Give it a predicate for which values to transform, and a function to do the transformation. Couldn't be easier.
- narwally 6y ago
  It really does remove multiple layers of complexity that usually stand between your mental model of what your transformation function should do and the code that actually implements it. With Specter if I can clearly articulate the transformation I want to do on a data structure, then I can almost always translate that one-to-one into the code that does the transformation. When I started using it in one of my existing projects I was able to completely eliminate a few needlessly long and complex functions that were littered with comments and calls to helper functions. In their place are functions that are about 5 lines of code that barely even need a docstring because it's readily apparent what they do just by looking at the source.
slifin 6y ago
There's a lot of alternatives in this space Meander was the last one I can remember: https://www.youtube.com/watch?v=9fhnJpCgtUw https://www.youtube.com/watch?v=9fhnJpCgtUw
- deleted 6y ago
  [deleted]
tekacs 6y ago
Specter is tremendously useful and rarely lets on a case where it can’t perform a transformation. For ensuring that you’re getting the most efficient form of a transformation [0] it’s incredible —- and it is somewhat data-oriented, allowing easy saving and reuse of navigators. Meander [1] is also wonderful. Its unification-based approach makes for nice expressions of intent. Its strategy-based approach to rewriting [2] is more flexible than most common walker libraries, too. [0]: https://github.com/redplanetlabs/specter/wiki/Specter's-inline-caching-implementation https://github.com/redplanetlabs/specter/wiki/Specter's-inli... [1]: https://github.com/noprompt/meander https://github.com/noprompt/meander [2]: https://github.com/noprompt/meander/blob/epsilon/doc/strategies.md https://github.com/noprompt/meander/blob/epsilon/doc/strateg...
- involans 6y ago
  So specter is lens and meander is syb/generics. Nice
  - devin 6y ago
    I know the author of meander, and I kind of doubt they'd describe it this way. I will ask them, though.
jdc 6y ago
Previous discussion of Specter: https://news.ycombinator.com/item?id=10295060 https://news.ycombinator.com/item?id=10295060 also cf Glom for Python: https://sedimental.org/glom_restructured_data.html https://sedimental.org/glom_restructured_data.html Hashdig in Ruby's standard library: https://ruby-doc.org/core-2.7.1/Hash.html#method-i-dig https://ruby-doc.org/core-2.7.1/Hash.html#method-i-dig
sova 6y ago
Like SQL for nested maps in Clojureland. Neat. The DSL [ALL :a even?] does feel a tiny bit un-clojurey to me as well, maybe making it look like [:all :a even?] would make it clearer.
- escherize 6y ago
  Well, not sure about that exactly. Consider a map like: {:all {:a 2} :foo {:a 4}} Maybe namespaced keys would fit that usecase
- boxed 6y ago
  In instar it's [* :a even?] which I think is prettier... being the author of instar :)
- tekacs 6y ago
  Well there’s a reason it’s like that — ALL is actually a value, not just a sentinel or keyword. It’s found at com.rpl.specter/ALL. Keywords don’t form a DSL in Specter, they just do keyword navigation into maps, the same way ‘0’ navigates into the first element of a list. This touches upon a really nice element of Specter, which is its pattern for reuse. Here’s a very real but deliberately simple example of using it: (def chat-threads {[“sova” “tekacs”] [{:text [“hey” “you there?”]} {:text [“line 1” “line 2”]}]}) (def ALL-TEXT [sr/ALL :text sr/ALL]) (defn text-for [chat] (sr/select [chat ALL-TEXT])) (def ALL-CHAT-TEXT [sr/MAP-VALS ALL-TEXT]) (text-for [“sova” “tekacs”]) ;=> [...] Here the navigator into a list, then the :text field and then the list is bottled simply by putting several navigators in a vector. Then, it can be used alongside other navigators in a first class fashion — [chat ALL-TEXT] is just a nested vector. :)
setzer22 6y ago
Specter is always on my clojure toolbox. Not only it makes transforming arbitrarily nested data painless, it also is usually more efficient than the average hand-rolled code! A great example of why macros in Lisp are powerful, and how they empower you as a user even if you never write one.
deleted 6y ago
[deleted]
dwohnitmok 6y ago
I hope that more and more lens implementations should catch on in languages with immutable data structures. They really are the key to make immutable data structures as easy to update and work with as mutable data structures.
dimovich 6y ago
Specter is really handy for manipulating data structures, and have been happily using it. I wonder, is it possible to select some subset of a map? Say you have these paths: (def paths [[:some :path] [:another :deeper :path] [:yet :another :one] [:yet :another :one :deeper]]) And select from a map these paths. You could do it with get-in, but maybe Specter can do it faster? I couldn't find a way to do it with Specter...
kazinator 6y ago
TXR Lisp: This is the TXR Lisp interactive listener of TXR 243. Quit with :quit or Ctrl-D on an empty line. Ctrl-X ? for cheatsheet. Give me all the .c files: 1> (glob "*.c") ("abcd.c" "args.c" "arith.c" "buf.c" "cadr.c" "chksum.c" "combi.c" "crazyffi.c" "debug.c" "dict.c" "eval.c" "ffi.c" "filter.c" "ftw.c" "gc.c" "glob.c" "halfsip.c" "hash.c" "highest-bit-main.c" "highest-bit.c" "highest-test.c" "isqrt.c" "itypes.c" "lex.yy.c" "lib.c" "lisplib.c" "match.c" "parser.c" "protsym.c" "query.c" "rand.c" "regex.c" "signal.c" "socket.c" "stream.c" "struct.c" "strudel.c" "sysif.c" "syslog.c" "termios.c" "test.c" "tree.c" "txr.c" "unwind.c" "utf8.c" "vecho.c" "vm.c" "y.tab.c") OK, which ones are over 100000 bytes: 2> (keep-if [chain stat .size (op < 100000)] (glob "*.c")) ("arith.c" "eval.c" "ffi.c" "lex.yy.c" "lib.c" "match.c" "stream.c" "y.tab.c")
- jonahbenton 6y ago
  Clojure is already good at that- at conditional pipe / filter / aggregate semantics, bc it is a lisp. Specter is for a different problem. When you have a data structure that has some semantics and is deeply nested (and you aren't discovering the structure, like getting size from a node via a function call) Specter has a declarative set of operations to navigate and extract on that data structure, extremely efficiently.
  - kazinator 6y ago
    1> (defstruct node () item next) #<struct-type node> 2> (new node next (new node next (new node item 42))) #S(node item nil next #S(node item nil next #S(node item 42 next nil))) 3> (call .next.next.item *2) 42
    - reitzensteinm 6y ago
      I don't usually have a problem with you posting TXR comparisons in every lisp thread, but if you're not going to take the time to understand what the library does, this is just spam.
      - deleted 6y ago
cutler 6y ago
So it turns out that Clojure is the ultimate data-wrangling tool yet there's no chance of earning a living with it. So much for innovation. Technology is a strange industry where conservative Java bondage and a 2-bit browser scripting extension with no stdlib reign supreme.
- wellpast 6y ago
  Clojure’s just a Java library, friend.
  - cutler 6y ago
    I wonder how many fanfold pages of Java, complete with getters, setters, toString and hashCode, one of Nathan's examples expands into?
    - deleted 6y ago
      [deleted]
- hota_mazi 6y ago
  Clojure is wholly inadequate to meeting challenges of the industry today. Not just because it's dynamically typed (which means tooling for it is close to nonexistent and it has terrible performance), but also because its ecosystem is a ghost town. It takes more than just a language to create a productive development environment in 2020, especially one which is so flawed by designed to the point of not even being statically typed.
  - wellpast 6y ago
    Your Google skills must be short. Clojure community is on fire with open-source and it’s IDE and build support is copious and stellar. Also it’s not statically typed on purpose; the reasons for this have been described & analyzed at length. I’d recommend you get out more.
  - deleted 6y ago
    [deleted]
mrbonner 6y ago
Darn! This is sleek! I have been learning Clojure off and on for almost a year now. I gotta say that it has changed my mind to a "Aha, I got lisp now" moment. Working with it is a joy, especially it integrates so well with the JVM and all available Java libraries out there. I can't believe I have missed Specter during my learning journey. How could I not find out about this soon enough. I mostly deal with static typed languages and seeing the primary use of map and keywords as main data structure scares me. Rich Hicky said not to put too much emphasis in what in the map but the data/keywords you need, kind of like you do not care what in the UPS truck as long as it could deliver your package, right? But, navigating maps and nested data structures around is the pain. Specter definitely easies that pain a lot now that I know it! This is why I am grateful for hackernews. Once in awhile, browsing the site and discover something like this makes me rethink the ways I do stuff. It also reminds me that I could be missing a lots of similar goodies on my way learning Clojure. I am prepared to have me mind blown again!!!

152 points by amgreg 6y ago

I invented instar (https://github.com/boxed/Instar https://github.com/boxed/Instar) which is quite similar to specter (imo a little bit nicer) because I also thought the built in API was weirdly clunky in these respects.

didibus 6y ago
What's the performance vis-a-vis Specter?
- boxed 6y ago
  I have never run a benchmark and the specter guy really cares about it so instar is probably way worse :) I was more trying to show that the API could be better and hoping this would put pressure on core devs to do it properly.
ledgerdev 6y ago
Very nice syntax, looks great.
SomeHacker44 6y ago
I have used both Spectre and Instar. I prefer Instar for straight forward simple transforms, but Spectre can do more complex ones. Spectre has a more difficult API to wrap your head around and that is its biggest downside, IMO. You really need to dedicate time to learn it rather than try to pull it out on demand.

john-shaffer 6y ago

Specter is fantastic. It makes transformations on deeply nested data structures easy. Want to transform all the "datetimes" from JSON data into actual datetimes (or vice versa)? It takes about five minutes if you have to look everything up. Works with ClojureScript too. Some production code: (defn transform-tagged-values [x] (sp/transform (sp/walker tr/tagged-value?) tagged-value->cljs x)) Give it a predicate for which values to transform, and a function to do the transformation. Couldn't be easier.

narwally 6y ago
It really does remove multiple layers of complexity that usually stand between your mental model of what your transformation function should do and the code that actually implements it. With Specter if I can clearly articulate the transformation I want to do on a data structure, then I can almost always translate that one-to-one into the code that does the transformation. When I started using it in one of my existing projects I was able to completely eliminate a few needlessly long and complex functions that were littered with comments and calls to helper functions. In their place are functions that are about 5 lines of code that barely even need a docstring because it's readily apparent what they do just by looking at the source.

slifin 6y ago

There's a lot of alternatives in this space Meander was the last one I can remember: https://www.youtube.com/watch?v=9fhnJpCgtUw https://www.youtube.com/watch?v=9fhnJpCgtUw

deleted 6y ago
[deleted]

tekacs 6y ago

Specter is tremendously useful and rarely lets on a case where it can’t perform a transformation. For ensuring that you’re getting the most efficient form of a transformation [0] it’s incredible —- and it is somewhat data-oriented, allowing easy saving and reuse of navigators. Meander [1] is also wonderful. Its unification-based approach makes for nice expressions of intent. Its strategy-based approach to rewriting [2] is more flexible than most common walker libraries, too. [0]: https://github.com/redplanetlabs/specter/wiki/Specter's-inline-caching-implementation https://github.com/redplanetlabs/specter/wiki/Specter's-inli... [1]: https://github.com/noprompt/meander https://github.com/noprompt/meander [2]: https://github.com/noprompt/meander/blob/epsilon/doc/strategies.md https://github.com/noprompt/meander/blob/epsilon/doc/strateg...

involans 6y ago
So specter is lens and meander is syb/generics. Nice
- devin 6y ago
  I know the author of meander, and I kind of doubt they'd describe it this way. I will ask them, though.

jdc 6y ago

Previous discussion of Specter: https://news.ycombinator.com/item?id=10295060 https://news.ycombinator.com/item?id=10295060 also cf Glom for Python: https://sedimental.org/glom_restructured_data.html https://sedimental.org/glom_restructured_data.html Hashdig in Ruby's standard library: https://ruby-doc.org/core-2.7.1/Hash.html#method-i-dig https://ruby-doc.org/core-2.7.1/Hash.html#method-i-dig

sova 6y ago

Like SQL for nested maps in Clojureland. Neat. The DSL [ALL :a even?] does feel a tiny bit un-clojurey to me as well, maybe making it look like [:all :a even?] would make it clearer.

escherize 6y ago
Well, not sure about that exactly. Consider a map like: {:all {:a 2} :foo {:a 4}} Maybe namespaced keys would fit that usecase
boxed 6y ago
In instar it's [* :a even?] which I think is prettier... being the author of instar :)
tekacs 6y ago
Well there’s a reason it’s like that — ALL is actually a value, not just a sentinel or keyword. It’s found at com.rpl.specter/ALL. Keywords don’t form a DSL in Specter, they just do keyword navigation into maps, the same way ‘0’ navigates into the first element of a list. This touches upon a really nice element of Specter, which is its pattern for reuse. Here’s a very real but deliberately simple example of using it: (def chat-threads {[“sova” “tekacs”] [{:text [“hey” “you there?”]} {:text [“line 1” “line 2”]}]}) (def ALL-TEXT [sr/ALL :text sr/ALL]) (defn text-for [chat] (sr/select [chat ALL-TEXT])) (def ALL-CHAT-TEXT [sr/MAP-VALS ALL-TEXT]) (text-for [“sova” “tekacs”]) ;=> [...] Here the navigator into a list, then the :text field and then the list is bottled simply by putting several navigators in a vector. Then, it can be used alongside other navigators in a first class fashion — [chat ALL-TEXT] is just a nested vector. :)

setzer22 6y ago

Specter is always on my clojure toolbox. Not only it makes transforming arbitrarily nested data painless, it also is usually more efficient than the average hand-rolled code! A great example of why macros in Lisp are powerful, and how they empower you as a user even if you never write one.

deleted 6y ago

[deleted]

dwohnitmok 6y ago

I hope that more and more lens implementations should catch on in languages with immutable data structures. They really are the key to make immutable data structures as easy to update and work with as mutable data structures.

dimovich 6y ago

Specter is really handy for manipulating data structures, and have been happily using it. I wonder, is it possible to select some subset of a map? Say you have these paths: (def paths [[:some :path] [:another :deeper :path] [:yet :another :one] [:yet :another :one :deeper]]) And select from a map these paths. You could do it with get-in, but maybe Specter can do it faster? I couldn't find a way to do it with Specter...

kazinator 6y ago

TXR Lisp: This is the TXR Lisp interactive listener of TXR 243. Quit with :quit or Ctrl-D on an empty line. Ctrl-X ? for cheatsheet. Give me all the .c files: 1> (glob "*.c") ("abcd.c" "args.c" "arith.c" "buf.c" "cadr.c" "chksum.c" "combi.c" "crazyffi.c" "debug.c" "dict.c" "eval.c" "ffi.c" "filter.c" "ftw.c" "gc.c" "glob.c" "halfsip.c" "hash.c" "highest-bit-main.c" "highest-bit.c" "highest-test.c" "isqrt.c" "itypes.c" "lex.yy.c" "lib.c" "lisplib.c" "match.c" "parser.c" "protsym.c" "query.c" "rand.c" "regex.c" "signal.c" "socket.c" "stream.c" "struct.c" "strudel.c" "sysif.c" "syslog.c" "termios.c" "test.c" "tree.c" "txr.c" "unwind.c" "utf8.c" "vecho.c" "vm.c" "y.tab.c") OK, which ones are over 100000 bytes: 2> (keep-if [chain stat .size (op < 100000)] (glob "*.c")) ("arith.c" "eval.c" "ffi.c" "lex.yy.c" "lib.c" "match.c" "stream.c" "y.tab.c")

jonahbenton 6y ago
Clojure is already good at that- at conditional pipe / filter / aggregate semantics, bc it is a lisp. Specter is for a different problem. When you have a data structure that has some semantics and is deeply nested (and you aren't discovering the structure, like getting size from a node via a function call) Specter has a declarative set of operations to navigate and extract on that data structure, extremely efficiently.
- kazinator 6y ago
  1> (defstruct node () item next) #<struct-type node> 2> (new node next (new node next (new node item 42))) #S(node item nil next #S(node item nil next #S(node item 42 next nil))) 3> (call .next.next.item *2) 42
  - reitzensteinm 6y ago
    I don't usually have a problem with you posting TXR comparisons in every lisp thread, but if you're not going to take the time to understand what the library does, this is just spam.
    - deleted 6y ago

cutler 6y ago

So it turns out that Clojure is the ultimate data-wrangling tool yet there's no chance of earning a living with it. So much for innovation. Technology is a strange industry where conservative Java bondage and a 2-bit browser scripting extension with no stdlib reign supreme.

wellpast 6y ago
Clojure’s just a Java library, friend.
- cutler 6y ago
  I wonder how many fanfold pages of Java, complete with getters, setters, toString and hashCode, one of Nathan's examples expands into?
  - deleted 6y ago
    [deleted]
hota_mazi 6y ago
Clojure is wholly inadequate to meeting challenges of the industry today. Not just because it's dynamically typed (which means tooling for it is close to nonexistent and it has terrible performance), but also because its ecosystem is a ghost town. It takes more than just a language to create a productive development environment in 2020, especially one which is so flawed by designed to the point of not even being statically typed.
- wellpast 6y ago
  Your Google skills must be short. Clojure community is on fire with open-source and it’s IDE and build support is copious and stellar. Also it’s not statically typed on purpose; the reasons for this have been described & analyzed at length. I’d recommend you get out more.
- deleted 6y ago
  [deleted]

mrbonner 6y ago

Darn! This is sleek! I have been learning Clojure off and on for almost a year now. I gotta say that it has changed my mind to a "Aha, I got lisp now" moment. Working with it is a joy, especially it integrates so well with the JVM and all available Java libraries out there. I can't believe I have missed Specter during my learning journey. How could I not find out about this soon enough. I mostly deal with static typed languages and seeing the primary use of map and keywords as main data structure scares me. Rich Hicky said not to put too much emphasis in what in the map but the data/keywords you need, kind of like you do not care what in the UPS truck as long as it could deliver your package, right? But, navigating maps and nested data structures around is the pain. Specter definitely easies that pain a lot now that I know it! This is why I am grateful for hackernews. Once in awhile, browsing the site and discover something like this makes me rethink the ways I do stuff. It also reminds me that I could be missing a lots of similar goodies on my way learning Clojure. I am prepared to have me mind blown again!!!