19 ms·

Clojure from a Schemer's perspective (2021)

168 points by lispybanana 4y ago

lionsdan 4y ago
Previous discussion https://news.ycombinator.com/item?id=26356367 https://news.ycombinator.com/item?id=26356367
kccqzy 4y ago
> A quick look at the code implies that various data structures are used under the hood for what looks like one data structure in the language. That's a lot of complexity! I'm not sure that's a tradeoff I'd be happy to make. It makes it harder to reason about performance. I don't think that's true. A persistent array map is a single data structure. It's just a persistent data structure optimized for non-destructive (functional) updates. On a high-level it's not too different from finger trees which may be more famous. When I used to write Clojure, I didn't feel any need to reason about its internals for performance reasons or any other reasons. It simply feels natural.
- lgessler 4y ago
  I think the author's referring to the multiple implementations of persistent maps (i.e. APersistentMap). Clojure's generated docs[1] say there are four implementations (PersistentArrayMap, PersistentHashMap, PersistentStructMap, PersistentTreeMap), and this Stack Overflow answer[2] explains that the latter two are rarely used, and the former two make up the bulk of actual APersistentMap instances, with the language's core functions for persistent maps changing implementation used under the hood as it likes in order to do what it thinks is best for performance (basically, using PersistentArrayMap if the number of keys is small, and using PersistentHashMap otherwise). You can see this, for example, in PersistentArrayMap's implementation of assoc[3], which replaces itself with a PersistentHashMap if the number of keys has grown over the hardcoded threshold of 16. So like you say, your experience as an "end user" makes it seem like it's just one data structure at least in interface, though in implementation it's actually doing some performance optimization stuff which, I think I'm willing to grudgingly grant to the author, does make performance harder to reason about in principle. Though in my (very unserious) experience, I think I've enjoyed languages that take a possibly incomplete stab at getting performance right over ones that draw a line leaving it entirely out of scope and declare victory. I'll equivocate a little and share that I have been bitten by these internals before, though the fault was mine: one time, I was unwittingly relying on an implementation detail of PersistentArrayMaps, which is that they preserve insertion order. PersistentHashMaps don't, and my code was mysteriously breaking when number of keys grew beyond 16 until several hours later I learned much of what I've just shared above. [1]: https://www.javadoc.io/static/org.clojure/clojure/1.10.1/clojure/lang/APersistentMap.html https://www.javadoc.io/static/org.clojure/clojure/1.10.1/clo... [2]: https://stackoverflow.com/questions/16516176/what-is-the-difference-between-clojures-apersistentmap-implementations https://stackoverflow.com/questions/16516176/what-is-the-dif... [3]: https://github.com/clojure/clojure/blob/5ffe3833508495ca7c635d47ad7a1c8b820eab76/src/jvm/clojure/lang/PersistentArrayMap.java#L259L260 https://github.com/clojure/clojure/blob/5ffe3833508495ca7c63...
  - eru 4y ago
    CPython also switches out what internal implementation it uses to represent your dicts based on a few heuristics.
  - chii 4y ago
    > does make performance harder to reason about in principle. the thing is, you don't reason about performance. you measure it. Once you've obtained measurements, you can make decisions based on the outcome of said measurements. If the Runtime (and not just talking about clojure here) makes a decision on performance, and it's suboptimal, your measurements _should_ reveal it - for example, by increasing your input size, the performance graph should show a clear inflexion of where this happens. The real question is whether you _can_ force the Runtime to do something at the programmer's behest, rather than a hidden decision that you cannot control or change. I believe in clojure, you can change the backing data structure, but i'm not sure if you could do it after the fact?
    - rawoke083600 4y ago
      >the thing is, you don't reason about performance. you measure it. You that "fun person" at parties right ? :)
      - CrazyPyroLinux 4y ago
        The scientific approach would be to invite him to a party and measure it. :)
    - germandiago 4y ago
      > the thing is, you don't reason about performance. you measure it. This is only half-true. You do reason about it and later profile. For example, I want to know if my assinment operator in C++ will run in constant time.
- rippeltippel 4y ago
  Isn't it similar to what happens in other languages? In example, in C++ std::maps are implemented as RB trees, are::vectors instantiate 50% (sometimes 100%) more memory than requested, etc. All of that happens under the bonnet in order to fulfill the performance guarantees of the language. IIRC, one of the most widely quoted memes in Clojure space goes along the lines of "simple != easy". Or better (not= :simple :easy)
  - pjmlp 4y ago
    Traditionally most C++ implementations use RB trees, the standard doesn't require it to be so, all data structures requirements are described via big O notation.
    - zabzonk 4y ago
      except that if you don't use a rb tree, you will have a hard time meeting the performance requirements
      - pjmlp 4y ago
        Kind of, if tomorrow someone comes up with funky zombie trees with the same O() characteristics, or even better, they can ship them on their C++ standard library. And then some clever C++ developers that have code that depends on rb tree's behaviours, would cry that their cheese was moved and their code doesn't work as before, even though nothing prevented funky zombie trees to be shipped instead.
- pluijzer 4y ago
  Lisp programmers know the value of everything and the cost of nothing.
- hxegon 4y ago
  The actual details of how the underlying datastructures work isn't that hard to grasp (see: https://youtu.be/wASCH_gPnDw?t=1817 https://youtu.be/wASCH_gPnDw?t=1817), but in practice you don't even need to go that far. At a high level there's really just a few things you need to keep in mind, the biggest being the big O guarantees of the core DS interfaces, which are listed here https://clojure.org/reference/data_structures https://clojure.org/reference/data_structures. Those guarantees hold across the different underlying implementations, and there are plenty of things you can tune to get a 2x, 10x, 50x improvement in performance before resorting to things that depend on which specific DS is being used under the hood Edit: Fixed timestamp for video link
- deleted 4y ago
  [deleted]
pmoriarty 4y ago
One of the main reasons I don't even want to touch Clojure is because I don't want to be trying to figure out Java stacktraces when things go wrong.
- fuzzythinker 4y ago
  Clojure doesn't need JVM. You can run it with GraalVM either directly [1] or via Babashka [2]. Both are great options for get into Clojure. [1] https://github.com/oracle/graal https://github.com/oracle/graal [2] https://github.com/babashka/babashka https://github.com/babashka/babashka
  - selimco 4y ago
    Technically, graal is also a JVM. Do you mean native image by any chance?
    - capableweb 4y ago
      Technically, I think it's more correct to say GraalVM includes/has a JVM, it runs a couple of languages, not just JVM based ones.
- kaba0 4y ago
  What’s bad about Java stacktraces?
  - vaylian 4y ago
    The problem is that the stacktrace shows you all the gory JVM details that are irrelevant to your Clojure code. Even when your code is 100% Clojure, the Java classes, that are the foundation for Clojure, are shining through when you get the stacktrace. It's been a while since I last used Clojure, so I don't know if the situation has improved in the meantime.
    - thom 4y ago
      For what it’s worth, it’s very easy to filter stack traces in most Clojure tooling. You don’t have to look at anything but your code, or just Clojure code etc. Obviously you’re still looking at Java exceptions for the most part, but I’ve never found being on top of the JVM as icky as some others obviously do.
rawoke083600 4y ago
Recently I tried to jump into scheme (I'm okish in clojure) man what a mission to install scheme on linux. I first tried todo gerbil scheme, then Ubuntu seems to have the wrong version of gambit (gerbil-scheme runs optop of gambit-scheme for those that don't know). That took a few hours of compiling, reading, trying diff versions always ended in some deadlock-issue ?? Then I tried Guile,was a little better and easier, but soon got stumped when I wanted to access mysql-db yes there are packages but those packages I couldn't install for the life of me ! In the end chicken-scheme was surprisingly easy and complete solution for a hit-the-ground-running scheme. Now I'm prob a noob as it comes to scheme, but not to fiddle and massage packages and programs on linux. Trying to learn scheme but spending a few hours just to get it installed was what killed it for me. Clojure is certainly not perfect, but you could get up and running much faster in my opinion. Oh and it uses square brackets for parameters :)
- pjmlp 4y ago
  Next time just install Chez Scheme or Raket, they work perfectly fine straight out Ubuntu repos.
- medo-bear 4y ago
  besides having the best mascot, just curious why you would start with gebril? esp for a newbie. surprised you didnt just pick racket not an expert but i believe you should use guix for anything guile related > Oh and it uses square brackets for parameters :) for many lispers this is a negative because they see as a big positive the manimal (no) syntax value proposition of lisp. besides if you use emacs you can easily (if you are a lisper) make it do something like having parameters highlighted without introducing additional syntax in the underlying language
- bakul 4y ago
  Gambit scheme also just works. That's what I use on FreeBSD, MacOS (m1 and x86-64), Ubuntu and Raspbian. Had a version running under plan9 as well but that was a long time ago.
- vkazanov 4y ago
  One (non?) problem with Scheme is that it is has a long history of being a language implementation research device. This is why it is relatively easy to implement, and this is why there are so many implementations. At least it didn't go the way beautiful Standard ML went!
weatherlight 4y ago
Clojure is a nice well thought out language, I wish it wasn't on the JVM.
- snikeris 4y ago
  Curious when you formed your opinion on the JVM. It's been under active development for 28 years.
  - Steltek 4y ago
    The JVM ecosystem pioneered a lot of what I see as awful anti-patterns in language development: - Language-exclusive build chain: Not that Make is the be-all and end-all of build tools but it worked with everything without prejudice. To understand the basics of a build system, you only needed to know one tool. All attempts at fixing Make's flaws fail to understand this fundamental recipe for success. Now when you switch languages, you need to relearn that layer on top of the language itself. And the tools all do the same thing but with their own syntax, commands, and eccentricities. - Language-exclusive packages and repositories. Once upon a time, you learned your OS's package manager and you were done. Now you need to learn a new package system for every language out there. Again with their own syntax, commands, versioning, and eccentricities. - Needing language-aware parsers to determine dependencies. You can't depend on spatial locality or compiler-output dependency trees to know whether to rebuild, reinstall, etc. You had to parse out the org.whatever's and resolve that back to a jar or class, which was clumsy and mostly wrong. - The OS doesn't know how to run your program. What's so hard about building your executable so it can be natively run by the OS? Everyone has to write the same `#!/bin/sh exec java -jar ...` wrapper script. Java programs worked best by assuming it was going to be a pariah hanging out in /opt and never actually integrate with anything. - Runtime versioning and installation was painful. I disliked having to suddenly manage environment variables just to run something. It continues today with python2 vs python3, GOPATH, etc. On top of that, JRE's weren't a simple "curl|bash" and no one was allowed to repackage the JRE to make it more user friendly.
    - ThatGeoGuy 4y ago

168 points by lispybanana 4y ago

Previous discussion https://news.ycombinator.com/item?id=26356367 https://news.ycombinator.com/item?id=26356367

> A quick look at the code implies that various data structures are used under the hood for what looks like one data structure in the language. That's a lot of complexity! I'm not sure that's a tradeoff I'd be happy to make. It makes it harder to reason about performance. I don't think that's true. A persistent array map is a single data structure. It's just a persistent data structure optimized for non-destructive (functional) updates. On a high-level it's not too different from finger trees which may be more famous. When I used to write Clojure, I didn't feel any need to reason about its internals for performance reasons or any other reasons. It simply feels natural.

lgessler 4y ago
I think the author's referring to the multiple implementations of persistent maps (i.e. APersistentMap). Clojure's generated docs[1] say there are four implementations (PersistentArrayMap, PersistentHashMap, PersistentStructMap, PersistentTreeMap), and this Stack Overflow answer[2] explains that the latter two are rarely used, and the former two make up the bulk of actual APersistentMap instances, with the language's core functions for persistent maps changing implementation used under the hood as it likes in order to do what it thinks is best for performance (basically, using PersistentArrayMap if the number of keys is small, and using PersistentHashMap otherwise). You can see this, for example, in PersistentArrayMap's implementation of assoc[3], which replaces itself with a PersistentHashMap if the number of keys has grown over the hardcoded threshold of 16. So like you say, your experience as an "end user" makes it seem like it's just one data structure at least in interface, though in implementation it's actually doing some performance optimization stuff which, I think I'm willing to grudgingly grant to the author, does make performance harder to reason about in principle. Though in my (very unserious) experience, I think I've enjoyed languages that take a possibly incomplete stab at getting performance right over ones that draw a line leaving it entirely out of scope and declare victory. I'll equivocate a little and share that I have been bitten by these internals before, though the fault was mine: one time, I was unwittingly relying on an implementation detail of PersistentArrayMaps, which is that they preserve insertion order. PersistentHashMaps don't, and my code was mysteriously breaking when number of keys grew beyond 16 until several hours later I learned much of what I've just shared above. [1]: https://www.javadoc.io/static/org.clojure/clojure/1.10.1/clojure/lang/APersistentMap.html https://www.javadoc.io/static/org.clojure/clojure/1.10.1/clo... [2]: https://stackoverflow.com/questions/16516176/what-is-the-difference-between-clojures-apersistentmap-implementations https://stackoverflow.com/questions/16516176/what-is-the-dif... [3]: https://github.com/clojure/clojure/blob/5ffe3833508495ca7c635d47ad7a1c8b820eab76/src/jvm/clojure/lang/PersistentArrayMap.java#L259L260 https://github.com/clojure/clojure/blob/5ffe3833508495ca7c63...
- eru 4y ago
  CPython also switches out what internal implementation it uses to represent your dicts based on a few heuristics.
- chii 4y ago
  > does make performance harder to reason about in principle. the thing is, you don't reason about performance. you measure it. Once you've obtained measurements, you can make decisions based on the outcome of said measurements. If the Runtime (and not just talking about clojure here) makes a decision on performance, and it's suboptimal, your measurements _should_ reveal it - for example, by increasing your input size, the performance graph should show a clear inflexion of where this happens. The real question is whether you _can_ force the Runtime to do something at the programmer's behest, rather than a hidden decision that you cannot control or change. I believe in clojure, you can change the backing data structure, but i'm not sure if you could do it after the fact?
  - rawoke083600 4y ago
    >the thing is, you don't reason about performance. you measure it. You that "fun person" at parties right ? :)
    - CrazyPyroLinux 4y ago
      The scientific approach would be to invite him to a party and measure it. :)
  - germandiago 4y ago
    > the thing is, you don't reason about performance. you measure it. This is only half-true. You do reason about it and later profile. For example, I want to know if my assinment operator in C++ will run in constant time.
rippeltippel 4y ago
Isn't it similar to what happens in other languages? In example, in C++ std::maps are implemented as RB trees, are::vectors instantiate 50% (sometimes 100%) more memory than requested, etc. All of that happens under the bonnet in order to fulfill the performance guarantees of the language. IIRC, one of the most widely quoted memes in Clojure space goes along the lines of "simple != easy". Or better (not= :simple :easy)
- pjmlp 4y ago
  Traditionally most C++ implementations use RB trees, the standard doesn't require it to be so, all data structures requirements are described via big O notation.
  - zabzonk 4y ago
    except that if you don't use a rb tree, you will have a hard time meeting the performance requirements
    - pjmlp 4y ago
      Kind of, if tomorrow someone comes up with funky zombie trees with the same O() characteristics, or even better, they can ship them on their C++ standard library. And then some clever C++ developers that have code that depends on rb tree's behaviours, would cry that their cheese was moved and their code doesn't work as before, even though nothing prevented funky zombie trees to be shipped instead.
pluijzer 4y ago
Lisp programmers know the value of everything and the cost of nothing.
hxegon 4y ago
The actual details of how the underlying datastructures work isn't that hard to grasp (see: https://youtu.be/wASCH_gPnDw?t=1817 https://youtu.be/wASCH_gPnDw?t=1817), but in practice you don't even need to go that far. At a high level there's really just a few things you need to keep in mind, the biggest being the big O guarantees of the core DS interfaces, which are listed here https://clojure.org/reference/data_structures https://clojure.org/reference/data_structures. Those guarantees hold across the different underlying implementations, and there are plenty of things you can tune to get a 2x, 10x, 50x improvement in performance before resorting to things that depend on which specific DS is being used under the hood Edit: Fixed timestamp for video link
deleted 4y ago
[deleted]

pmoriarty 4y ago

One of the main reasons I don't even want to touch Clojure is because I don't want to be trying to figure out Java stacktraces when things go wrong.

fuzzythinker 4y ago
Clojure doesn't need JVM. You can run it with GraalVM either directly [1] or via Babashka [2]. Both are great options for get into Clojure. [1] https://github.com/oracle/graal https://github.com/oracle/graal [2] https://github.com/babashka/babashka https://github.com/babashka/babashka
- selimco 4y ago
  Technically, graal is also a JVM. Do you mean native image by any chance?
  - capableweb 4y ago
    Technically, I think it's more correct to say GraalVM includes/has a JVM, it runs a couple of languages, not just JVM based ones.
kaba0 4y ago
What’s bad about Java stacktraces?
- vaylian 4y ago
  The problem is that the stacktrace shows you all the gory JVM details that are irrelevant to your Clojure code. Even when your code is 100% Clojure, the Java classes, that are the foundation for Clojure, are shining through when you get the stacktrace. It's been a while since I last used Clojure, so I don't know if the situation has improved in the meantime.
  - thom 4y ago
    For what it’s worth, it’s very easy to filter stack traces in most Clojure tooling. You don’t have to look at anything but your code, or just Clojure code etc. Obviously you’re still looking at Java exceptions for the most part, but I’ve never found being on top of the JVM as icky as some others obviously do.

rawoke083600 4y ago

Recently I tried to jump into scheme (I'm okish in clojure) man what a mission to install scheme on linux. I first tried todo gerbil scheme, then Ubuntu seems to have the wrong version of gambit (gerbil-scheme runs optop of gambit-scheme for those that don't know). That took a few hours of compiling, reading, trying diff versions always ended in some deadlock-issue ?? Then I tried Guile,was a little better and easier, but soon got stumped when I wanted to access mysql-db yes there are packages but those packages I couldn't install for the life of me ! In the end chicken-scheme was surprisingly easy and complete solution for a hit-the-ground-running scheme. Now I'm prob a noob as it comes to scheme, but not to fiddle and massage packages and programs on linux. Trying to learn scheme but spending a few hours just to get it installed was what killed it for me. Clojure is certainly not perfect, but you could get up and running much faster in my opinion. Oh and it uses square brackets for parameters :)

pjmlp 4y ago
Next time just install Chez Scheme or Raket, they work perfectly fine straight out Ubuntu repos.
medo-bear 4y ago
besides having the best mascot, just curious why you would start with gebril? esp for a newbie. surprised you didnt just pick racket not an expert but i believe you should use guix for anything guile related > Oh and it uses square brackets for parameters :) for many lispers this is a negative because they see as a big positive the manimal (no) syntax value proposition of lisp. besides if you use emacs you can easily (if you are a lisper) make it do something like having parameters highlighted without introducing additional syntax in the underlying language
bakul 4y ago
Gambit scheme also just works. That's what I use on FreeBSD, MacOS (m1 and x86-64), Ubuntu and Raspbian. Had a version running under plan9 as well but that was a long time ago.
vkazanov 4y ago
One (non?) problem with Scheme is that it is has a long history of being a language implementation research device. This is why it is relatively easy to implement, and this is why there are so many implementations. At least it didn't go the way beautiful Standard ML went!

weatherlight 4y ago

Clojure is a nice well thought out language, I wish it wasn't on the JVM.

snikeris 4y ago
Curious when you formed your opinion on the JVM. It's been under active development for 28 years.
- Steltek 4y ago
  The JVM ecosystem pioneered a lot of what I see as awful anti-patterns in language development: - Language-exclusive build chain: Not that Make is the be-all and end-all of build tools but it worked with everything without prejudice. To understand the basics of a build system, you only needed to know one tool. All attempts at fixing Make's flaws fail to understand this fundamental recipe for success. Now when you switch languages, you need to relearn that layer on top of the language itself. And the tools all do the same thing but with their own syntax, commands, and eccentricities. - Language-exclusive packages and repositories. Once upon a time, you learned your OS's package manager and you were done. Now you need to learn a new package system for every language out there. Again with their own syntax, commands, versioning, and eccentricities. - Needing language-aware parsers to determine dependencies. You can't depend on spatial locality or compiler-output dependency trees to know whether to rebuild, reinstall, etc. You had to parse out the org.whatever's and resolve that back to a jar or class, which was clumsy and mostly wrong. - The OS doesn't know how to run your program. What's so hard about building your executable so it can be natively run by the OS? Everyone has to write the same `#!/bin/sh exec java -jar ...` wrapper script. Java programs worked best by assuming it was going to be a pariah hanging out in /opt and never actually integrate with anything. - Runtime versioning and installation was painful. I disliked having to suddenly manage environment variables just to run something. It continues today with python2 vs python3, GOPATH, etc. On top of that, JRE's weren't a simple "curl|bash" and no one was allowed to repackage the JRE to make it more user friendly.
  - ThatGeoGuy 4y ago