14 ms·

Metabase: Why we picked Clojure

179 points by tlrobinson 11y ago

moomin 11y ago
I had this conversation on twitter already: typed object mappers seem like a questionable choice for the problem space. Since it seems like all of their work in Scala used them, I'm not surprised they hated it. (Disclaimer: I have no particular liking for Scala.)
biokoda 11y ago
Considering their criteria Erlang/Elixir seems like a huge omission from the contender list.
- devinus 11y ago
  > Wide variety of mature database drivers That requirement was probably not met when they started.
  - learc83 11y ago
    Does Erlang lack a variety of mature database drivers?
    - illicium 11y ago
      Compared to something like JDBC et al, Erlang DB drivers are pretty half-baked
    - dsp1234 11y ago
      I'm not an Erlang dev, but a quick look at the Erlang docs shows: "If you need to access a relational database such as sqlserver, mysql, postgres, oracle, cybase etc. from your erlang application using the Erlang ODBC interface is a good way to go about it."[0] "It is built on top of Microsofts ODBC interface and therefore requires that you have an ODBC driver to the database that you want to connect to."[0] "But currently it is only regularly tested for sqlserver and postgres."[1] "But currently it is only tested for Solaris, Windows 2000, Windows XP and NT"[2] [0] - http://www.erlang.org/doc/apps/odbc/databases.html http://www.erlang.org/doc/apps/odbc/databases.html [1] - http://www.erlang.org/doc/man/odbc.html http://www.erlang.org/doc/man/odbc.html [2] - http://www.erlang.org/doc/apps/odbc/getting_started.html http://www.erlang.org/doc/apps/odbc/getting_started.html
kisstheblade 11y ago
" However, we had a strong aversion to using Java itself, so the real short list was Scala or Clojure." Sadly they couldn't articulate any actual problems with Java... I find it weird that so many developers focus on the "writability" of the language. I mean stuff that java gets criticized for, like verbosity etc. Which I don't even find so bad especially since the newer jdk versions. But in my opinion the _readability_ of the result is the most important factor. I rather read copy/pasted code than some smart clojure onliner which nobody dares to touch or really understands. You only write the program once, but you have to read it over and over again when maintaining it and developing new features, and java's maybe a little verbose syntax really helps here, and also the fact that there really is only one way to do a thing (unlike eg. perl). The tooling, documentation, frameworks, libraries and runtimes are all top notch with java. All other languages seem to have problems in one (or more) of these areas.
- vosper 11y ago
  The small amount of development I've done with Clojure convinced me that you just can't ignore Java - you're going to end up spending time reading Java docs, and debugging Java stacktraces. If you really dislike Java I don't think choosing Clojure is a great option, either (I don't know about Scala).
  - tlrobinson 11y ago
    (Metabase dev here) > you're going to end up spending time reading Java docs That's actually a benefit IMHO. If no one has created a nice Clojure library or wrapper for something you want to do it's at least possible to use the massive number of existing Java libraries directly from Clojure.
- kisstheblade 11y ago
  Ouch, got downvoted in under a minute. Guess it's not cool to say anything positive about java! :) I still would have liked it if the authors of the article had said something about why they had a "strong aversion" for java.
vosper 11y ago
> Python’s Mysql and Postgres database drivers required compilation. This made both developer machine setup as well as deployment more complicated than it really needed to be. Am I missing something - this was really a such a problem that it was a factor in switching languages?
- willvarfar 11y ago
  It matches my own experience too. Libraries that need to compile c are a deployment nightmare. Installing Oracle client is even worse though ;)
  - kilink 11y ago
    Isn't that what Python Wheels are for though? http://pythonwheels.com/ http://pythonwheels.com/
    - willvarfar 11y ago
      (pythonwheels doesn't do mysql nor postgres nor sqlalchmey nor ... most DBs in fact?)
- eterm 11y ago
  I've started interpreting a lot of these "Why we use X" posts as less 'bragging' and more these companies trying to justify their own choices. In part because it perhaps makes them feel less isolated in their choices, and in part also because by hoping that others follow them they'll find it easier to recruit, something that is easily underestimated especially outside of major hubs such as NY, SF, or London. Perhaps I am being uncharitable. I note with interest they say, "However, we had a strong aversion to using Java itself", but don't explain why they had that aversion, or whether it was based on anything but the fact that Java isn't trendy.
  - salsakran 11y ago
    Off the cuff, I'd estimate that between the team members at the time, we'd shipped perhaps 100k+ LOC of java into production systems at past jobs. It didn't seem interesting to rehash why we didn't want Java's verbosity in a small team.
mpdehaan2 11y ago
Kind of shocked that building database libraries for python (homebrew install gcc? pip install foo) was "hard" for them. I find the Python ORM options (sqlalchemy/Django/etc) lot better evolved than what is available for clojure and couldn't image that being a roadblock. Korma is basically a query builder and lacks a lot of things I was used to, and I had to engineer a declarative layer for defining models in the ways I wanted. Pain in setting up developer machines could be simplified with an ansible playbook pretty easily. OTOH, one of the worst things I hate is trying to figure out how to automatically get around Sun's paywalls and fiddle with Java classpaths :) I do feel some level of want to avoid vagrant in the development workflow, in which cases I'm usually fond of just logging into a real VM. I do think it's potentially good for tests. IDK, I always want to solve the idea of "fragile deploys" with immutable systems in prod. Uberjar was cool, I'll give it that -- but the startup time of clojure was often not worth the wait. My biggest complaints with clojure is documentation quality, inconsistency in return types (or just willingness to return nil in error scenarios), and tracebacks that are very difficult to sort through. Clojure also presents some problems with the idiomatic nature of trying to roll up everything into one statement, whereas adding debug and debugging require unrolling it, than packing statements up again to be socially acceptable idiomatic clojure. Testing in clojure also strikes me as painful, as it tends to punish you with "let" causing an indentation every time you need to assign a variable, something that is useful in tests that want to check results a lot along the way. My gut feeling is clojure was good for a little light scripting on top of a big Java codebase (kind of like you might use Lua in a game engine) but would not be something I'd pick for a primary language. Primarily using Java libs would also solve some of the ecosystem problems.
- bpicolo 11y ago
  sqlalchemy is my favorite library for pretty much any language. It's fantastic and a huge part of the python draw for me. The only downside is it doesn't play well with new async python tools, but python is totally fine at not being async
- salsakran 11y ago
  It's not hat "homebrew install gcc? pip install foo" is hard for us, it's that we didn't want to support 1000s of people trying to set it up on their machines. Every possible source of error we eliminated meant we'd need to spend less time on twitter debugging installation problems. It's useful to remember that our target install base isn't python developers who might want to hack away on a python project, but rather analysts, end users, DBAs and all kinds of people who've never heard of homebrew and who might not even be comfortable with using a terminal in Mac OS X let alone compiling things.
dustingetz 11y ago
"So, in summary, we chose Clojure for JVM threads + Database drivers, the ability to ship an uberjar and the ease of expressing tree manipulations which were the core piece of complexity in our backend." for people who may not know clojure here is an example of tree manipulations, it's really powerful especially in UI programming which is all about manipulating recursive structures: https://gist.github.com/dustingetz/c11ae4edb9e0d14787d2 https://gist.github.com/dustingetz/c11ae4edb9e0d14787d2 Scala has answers to this too (I would love for someone to reply with the best scala way) but for whatever subjective reasons the clojure way makes me happier for ui work. As to scala pushing towards typed object mappers, here is one example of generating untyped queries with Anorm [1], I'd be surprised if this couldn't be made to work for their query language, but that said the scala ecosystem has a lot of, shall we say "diversity of opinion" about the Proper Way To Do Things, so I am not surprised that newcomers to the language had a bad time. [1] (2013) http://www.dustingetz.com/2013/03/26/orm.html http://www.dustingetz.com/2013/03/26/orm.html
tunesmith 11y ago
Can someone expand on what they mean by lightweight threads for Clojure? My understanding is that lightweight threads (green threads) is what java/jvm explicitly moved away from way back when; it's user-mode threads bound to one processor, so if the processor gets busy, then all of the lightweight threads for that processor get blocked. Meanwhile, Java/jvm uses kernel threads and context-switching so you can do real multi-threading. I'm aware of how Quasar purports to be the best of both worlds, and it looks like Pulsar is a Clojure API for Quasar, maybe that's what they're using. It doesn't appear Scala has something comparative, but I have a hazy sense that Scala people believe that Akka and monads are better for easy asynchronous programming.
- patrickthebold 11y ago
  Clojure's core.async has "go routines", (this is more or less the same as in go), basically you can put and take messages on a channel. These operations will block the green thread, so you end up writing synchronous code. The idea is that synchronous, blocking code is easier to understand then async code. Since these are lightweight threads it will scale better than using real threads. I'm pretty sure it's fine to have hundreds of thousands of go routines. This abstraction sits on top of java's real threads, so you can have several real threads doing the work and they can switch between the go routines once the go routine hits one of the blocking operations. Or something like that... read this: http://clojure.com/blog/2013/06/28/clojure-core-async-channels.html http://clojure.com/blog/2013/06/28/clojure-core-async-channe...
- astine 11y ago
  It's likely that they're refering to core.async which is a Clojure library that provides IoC threads via the `go` construct.
- _halgari 11y ago
  Core.async uses code rewriting to turn blocks of code into state machines. From there puts and takes from channels become state transitions. It works almost exactly like async/await in C# or Yield based generators in other languages. The cool bit is that core.async does this as a library, not a compiler level feature.
RyanZAG 11y ago
> .. However, we had a strong aversion to using Java itself .. Honestly, why? Seems odd given Java's great track record for exactly this problem and huge library and developer share. The code itself is forced to be static and explicit which makes for easy onboarding for new developers. It's a bit strange to just dismiss Java in the context of the rest of the article. You're kind of staking your company on decisions like these, and making them because of some unquantifiable 'strong aversion' seems like a poor choice. That said, Clojure or other lisp base definitely seems like the correct choice for a query language / dsl. That's the shining point of lisps after all.
- zem 11y ago
  personally, i find java pretty painful to use; it's definitely capable of doing whatever job you have in mind, but when i use it i have the constant feeling i'm using a suboptimal tool. and then it leads to reams of boilerplate all of which has to be maintained, so it's not just a development-time cost; there's the depressing feeling of having to deal with it every time you want to read the code. clojure would be worth it just for the joy of using a tool that fits well into your hand.
  - spacemanmatt 11y ago
    My experience is Java costs me around 10x the number of lines of code (and I mean code, not imports) to get the same things done. The brief syntax and higher-order programming eliminate so much from my screen that, in hindsight, served as little more than visual framing for sparse bits of code.
  - RyanZAG 11y ago
    It's kind of like building a model with clay or with sticks. The clay model is going to look nicer with rounded curves and no wasted points. The stick model will have bits sticking out. However when you get 30 people working together to make a big model, the clay model is going to start looking very strange indeed. Different people will have different finger pressure and shape the clay differently. Some people will make more right angled parts with the clay which will clash with the parts of the clay that are carefully smoothed out. Someone might spend a week fixing a perfect parabola for a curve at the top of the model just for someone else to bend it off to the side with an ugly twist to fit with the other side of the model. 30 people on the stick model? Not really a problem. All the sticks look the same. Anybody who needs to attach one side of the stick model to something else just needs to grab a stick and tie it on. You won't see a perfect stick parabola, but then you also won't be stuck with a strangely twisted piece of clay. You can always create a style guide for using the clay - always keep between a 30 and 60 degree angle on joins. Apply one finger of pressure when doing joins. And so on. Sometimes this works with very diligent reviews that keep everything in place, but as soon as a crisis comes up or the model needs to pivot urgently? Everyone rushes to touch up the model in the quickest way possible and your model starts to get a bit thinner on one side than the other, etc. So there's nothing wrong with choosing a great clay model that fits well into your hand and you can shape effortlessly - but you need to remember that sometimes the 'painfulness' of using a model made out of sticks can be a big blessing when you're trying to massage a bit of clay that someone else pressed a bit too firmly on. EDIT: Also, when you've been making and using models for awhile, the stick models get an added benefit: you can browse through the stick models really fast based on only their outline and be able to understand very quickly how they work. Pretty clear why: the rigid sticks form a very distinct outline (think interfaces and forced directory structure in Java). Being a stick model expert therefore has very good job mobility and high demand. Each clay model is generally more unique and requires some study to understand even if it does look nicer.
daxfohl 11y ago
I'd be interested to understand how Korma solves their problem. It's macro-based so I wouldn't think it would be particularly good at dynamically restructuring queries based on runtime variables.
- thom 11y ago
  Yeah, clojureql was much better for this sort of thing but is to all intents and purposes abandoned.
stcredzero 11y ago
Now, choosing a programming language for a project is usually decided on a mix of what a team is most comfortable with, beer-fueled debates on dynamic vs static typing, benchmarks of marginal relevance and that elusive quest for Dev-Hipster points. How about this for an interview strategy: Have these discussions. Hire the people that realize most of the above mentioned things are meaningless. (Clarification: Only the "comfortable" part isn't meaningless. But there too, it's only as meaningful as the placebo effect is significant.)
- ionforce 11y ago
  How is "dynamic vs static typing" meaningless?
  - stcredzero 11y ago
    For most applications, either can work. They both have to be managed correctly.
    - ionforce 11y ago
      What are "most applications"? How do they have to be managed?
      - stcredzero 11y ago
        http://bfy.tw/36aT http://bfy.tw/36aT
- throwaway999888 11y ago
  I don't think an interview strategy of the form "has the same values and opinions as stcredzero" is going to work for everyone.
  - stcredzero 11y ago
    Of course not. Some people really like their illusions. But seriously: Having been part of a community that thought their language made them "super-programmers" I can say this: It's possible that on some level, the language does give a significant tooling advantage. However, this is often going to be completely swamped by various soft/human factors. A baroque quartet consisting of very mature musicians who work supremely well together is still going to produce far better music than a "super-group" of ego-driven martinets who secretly despise each other. It wouldn't matter how technologically advanced their instruments were over the 1st group, even if those instruments played themselves. It's the group that truly gels, has healthy interpersonal dynamics, has relevant experience, and healthy practice habits that is going to be the better musical group. Resolving the analogy left as an exercise.
programminggeek 11y ago
You know what else is easy to work with and deploy? PHP.
- lectrick 11y ago
  I wouldn't call a language that encourages writing spaghetti code and creating dependency hell to be "easy to work with", much less one which doesn't even pass its own test suite, which implies that even if you do everything 100% correctly, the spec itself is faulty, so you WILL have non-deterministic "3AM phone call, up till 3PM fixing" type bugs... No thanks
mark_l_watson 11y ago
I read this article yesterday because I am having a language choice issue, between Ruby and Clojure. I am in my (almost) mid 60s, and except for being on retainer for advice I don't really accept consulting work anymore. I have a long list of projects that I want to do, mostly open source, and settling on one language for a few years would make things easier. Clojure and Ruby both support DSL development very well which is important to me and both play very well with Java (via JRuby). I am starting to ask myself though if I wouldn't just be better off with Java 8, even without good support for writing DSLs. I have mostly stopped using Haskell and Scala. When I was much younger I totally enjoyed learning and using many programming languages but as I get older I feel more like just getting stuff done, especially since I dedicate much less time now to technical pursuits. So in a sense I am facing roughly the same type of decision as the Metabase people.
- nkassis 11y ago
  I'd personally look to find what language fills the gap that the current set of languages I know don't fill. Clojure is my filler for lisp language and language with good macros support. I use Ruby in the slot that many have Python in but with added coolness of metaprogramming not found in most scripting languages. And to add to that a bit, I found that recently it's been getting harder finding new languages with enough new patterns and paradigms to keep me interested in learning and mastering them.
- bribri 11y ago
  Out of curiosity, why not Haskell or Scala if you already know them?
- daxfohl 11y ago
  I follow your blog. I'm a month from 40 and agree. After several years working mostly in Clojure, Scala, and F#, I am finding myself most productive in traditional languages even though they're less interesting. And am finding projects more interesting than implementations, which is probably the way it should be. I'm finding now that OOP is actually a pretty good model for most things. Loops and mutation reflect reality: you do a task N times and write down each result; you don't take the first N results from an infinite seq of tasks. IoC is useful. In addition having a much larger set of things to copy/paste from github/stackoverflow is nice. That said, most (all?) of my paid work is pretty banal, nothing that's really begging for functional style. And for whatever reason I've never had much success with REPLs. Getting them updated correctly with the code and dependency changes, typing in what I want to test, then running it always seems to take me longer than just executing the app and testing changes directly.
iagooar 11y ago
It's great to see such an "exotic" programming language like Clojure getting some serious traction. And by exotic I mean a language that stands out from the imperative or object-oriented crowd. I like how Clojure challenges established paradigms with a surprisingly mature ecosystem. Being a hosted language on the JVM has its pros and cons, but the cleverness and pragmatism of Rich Hickey (creator of Clojure) really stands out. He decided to go with an industry strong, battle-tested environment, effectively standing on the shoulders of giants, while providing a lot of fresh ideas. Also, the fact of being a Lisp can be seen from different angles: for some it's lots of weirdly nested parentheses, for others it has a beautifully simple, yet extremely powerful and expressive syntax. But don't let those aspects fool you. Once you scratch a little bit the surface, you will see the extraordinary power of its persistent data structures. For those who don't know what persistent data structures are: in Clojure, oversimplifying, there is no mutation to data structures. Every time you "add" or "remove" a key from a hash map, you are actually creating a copy of it. And in order to make this efficient, you need special data structures with "superpowers", that are not actually creating a whole copy of themselves, but rather modify atomically only a part (hint: trees). And having those data structures leads to the next big feature of Clojure: concurrency. Not only does Clojure provide a mechanism to dispatch thread-like constructs (core.async), but the whole language is designed around supporting concurrency. I also want to add that what really surprised me about Clojure is its thriving community and smart people using it. I am pretty sure that if you want to hire great, highly motivated talent, going Clojure will help you with that.
- troym 11y ago
  > Once you scratch a little bit the surface, you will see the extraordinary power of its persistent data structures. I've heard this repeatedly, yet in the examples I've read (for web servers and for web ui work with cljs), they're using atoms with `swap!` to provide "current state". Could you provide an example of using persistent data structures in a web app? Or am I mis-understanding?
  - danneu 11y ago
    You're compare-and-swapping out persistent values with an atom (it holds a succession of immutable values over time). It's not an example of people shirking persistent data structures for mutation.
giancarlostoro 11y ago
One of the things that I love about Clojure is Leiningen. It just takes away the headaches of setting up projects in Clojure. I think the only other tool I'm aware of for another language that is quite as useful is probably NuGet. I've never had issues using lein, just drop it into my ~/bin/ folder (I'm using openSUSE so that folder is in the system path by default), open up a terminal and type in lein and it works. I don't even need clojure installed, just the JVM, it will download clojure for you as needed! Props to those working on lein.
- chii 11y ago
  And yet maven does the same thing but gets no love from anyone (including me!)
  - giancarlostoro 11y ago
    Not sure if Maven can be compared to lein, never personally used it, not knowingly at least. I've heard great things about Gradle though.
    - vorg 11y ago
      If Gradle's so great, then why does it use a JVM language that wasn't considered by Metabase. > The shortlist really came down to “something on the JVM” [...] However, we had a strong aversion to using Java itself, so the real short list was Scala or Clojure Gradle requires the configuration files to be written in Groovy, and is itself written in Java and lots of Groovy.
      - giancarlostoro 11y ago
        I do know Android Studio likes to rock Gradle. Clojure has Lein though, why would you use Maven then? :)
- chii 11y ago
  And yet maven does the same thing but gets no love from anyone (including me!)
johansch 11y ago
Has anyone here ever heard of Metabase before? Their github page indicates there are 3 (three) contributors. Wouldn't it be safe to assume this post is primarily a marketing venture?
- escherize 11y ago
  I've used Metabase before, it's a super easy way for a non-technical to do sql queries.
- escherize 11y ago
  I've used Metabase before, it's a super easy way for a non-technical to do sql queries.
staticelf 11y ago
This is probably going to be a very unpopular opinion here, but I think Clojure is a horrible language to work with. I don't have the experience with non imperitive languages which makes learning clojure a really steep curve. Also the error you can recieve with clojure are many times a big mystery, I've never seen as bad error reporting as I've seen with Clojure. Other issues I have with it is that Clojure-fans tends to want to write EVERYTHING in clojure or lisp style code. That includes stuff like Javascript, HTML and similar stuff. Things that just makes it even more complicated. I have really tried to learn it several times but got so frustrated I quit. We have a small thing at work which is written in Clojure and no one knows how to fix issues with it since no one has any experience with Clojure. Clojure does not fit my brain.
- timroy 11y ago
  I work only with Clojure and ClojureScript (including a simple DSL which lets me write HTML in Clojure-syntax). Error reporting in Clojure is bad - no two ways about it. Once I got used to scanning the JVM stack trace, it wasn't so bad. But it could be better. I really prefer writing ClojureScript to JS: using one language, with the same data structures, is a seamless experience. I like writing HTML in Clojure data structures, as it makes manipulating the data much easier. But your mileage may vary here! Clojure won't be my last language. :-)
- chrislloyd 11y ago
  I think your criticisms are definitely valid. The learning curve is steep (intentionally so) but definitely worth it. Your programs will become more reliable and you'll get more done as a consequence. Please push through! The error messages are terrible, and there are a lot of ongoing conversations about them at the moment in the Clojure community[1]. The Clojure[script] toolchain does tend to be all-in but there's work happening now to break that up — stay tuned. StackOverflow and the Clojure Slack channel also full of people who know how to fix issues :) Stick with it, when you'll have an "ah-ha!" moment soon enough! [1]: This talk is a good backstory as to why the errors are so bad, and useful no matter what language you use https://www.youtube.com/watch?v=FihU5JxmnBg https://www.youtube.com/watch?v=FihU5JxmnBg
- jdeisenberg 11y ago
melipone 11y ago
Lisp has been around a while, Java has been around a while. Clojure is the perfect combination and I predict it will be around a while.
melipone 11y ago
Lisp has been around a while. Java has been around a while. Clojure is the perfect combination and I predict it will be around a while. It's exhilarating to program in Clojure. That's the only word I can find to describe what I feel to program in that language.
lectrick 11y ago
Why wasn't Erlang/Elixir even considered? It checks off all their initial set of requirements

179 points by tlrobinson 11y ago

I had this conversation on twitter already: typed object mappers seem like a questionable choice for the problem space. Since it seems like all of their work in Scala used them, I'm not surprised they hated it. (Disclaimer: I have no particular liking for Scala.)

biokoda 11y ago

Considering their criteria Erlang/Elixir seems like a huge omission from the contender list.

devinus 11y ago
> Wide variety of mature database drivers That requirement was probably not met when they started.
- learc83 11y ago
  Does Erlang lack a variety of mature database drivers?
  - illicium 11y ago
    Compared to something like JDBC et al, Erlang DB drivers are pretty half-baked
  - dsp1234 11y ago
    I'm not an Erlang dev, but a quick look at the Erlang docs shows: "If you need to access a relational database such as sqlserver, mysql, postgres, oracle, cybase etc. from your erlang application using the Erlang ODBC interface is a good way to go about it."[0] "It is built on top of Microsofts ODBC interface and therefore requires that you have an ODBC driver to the database that you want to connect to."[0] "But currently it is only regularly tested for sqlserver and postgres."[1] "But currently it is only tested for Solaris, Windows 2000, Windows XP and NT"[2] [0] - http://www.erlang.org/doc/apps/odbc/databases.html http://www.erlang.org/doc/apps/odbc/databases.html [1] - http://www.erlang.org/doc/man/odbc.html http://www.erlang.org/doc/man/odbc.html [2] - http://www.erlang.org/doc/apps/odbc/getting_started.html http://www.erlang.org/doc/apps/odbc/getting_started.html

kisstheblade 11y ago

" However, we had a strong aversion to using Java itself, so the real short list was Scala or Clojure." Sadly they couldn't articulate any actual problems with Java... I find it weird that so many developers focus on the "writability" of the language. I mean stuff that java gets criticized for, like verbosity etc. Which I don't even find so bad especially since the newer jdk versions. But in my opinion the _readability_ of the result is the most important factor. I rather read copy/pasted code than some smart clojure onliner which nobody dares to touch or really understands. You only write the program once, but you have to read it over and over again when maintaining it and developing new features, and java's maybe a little verbose syntax really helps here, and also the fact that there really is only one way to do a thing (unlike eg. perl). The tooling, documentation, frameworks, libraries and runtimes are all top notch with java. All other languages seem to have problems in one (or more) of these areas.

vosper 11y ago
The small amount of development I've done with Clojure convinced me that you just can't ignore Java - you're going to end up spending time reading Java docs, and debugging Java stacktraces. If you really dislike Java I don't think choosing Clojure is a great option, either (I don't know about Scala).
- tlrobinson 11y ago
  (Metabase dev here) > you're going to end up spending time reading Java docs That's actually a benefit IMHO. If no one has created a nice Clojure library or wrapper for something you want to do it's at least possible to use the massive number of existing Java libraries directly from Clojure.
kisstheblade 11y ago
Ouch, got downvoted in under a minute. Guess it's not cool to say anything positive about java! :) I still would have liked it if the authors of the article had said something about why they had a "strong aversion" for java.

vosper 11y ago

> Python’s Mysql and Postgres database drivers required compilation. This made both developer machine setup as well as deployment more complicated than it really needed to be. Am I missing something - this was really a such a problem that it was a factor in switching languages?

willvarfar 11y ago
It matches my own experience too. Libraries that need to compile c are a deployment nightmare. Installing Oracle client is even worse though ;)
- kilink 11y ago
  Isn't that what Python Wheels are for though? http://pythonwheels.com/ http://pythonwheels.com/
  - willvarfar 11y ago
    (pythonwheels doesn't do mysql nor postgres nor sqlalchmey nor ... most DBs in fact?)
eterm 11y ago
I've started interpreting a lot of these "Why we use X" posts as less 'bragging' and more these companies trying to justify their own choices. In part because it perhaps makes them feel less isolated in their choices, and in part also because by hoping that others follow them they'll find it easier to recruit, something that is easily underestimated especially outside of major hubs such as NY, SF, or London. Perhaps I am being uncharitable. I note with interest they say, "However, we had a strong aversion to using Java itself", but don't explain why they had that aversion, or whether it was based on anything but the fact that Java isn't trendy.
- salsakran 11y ago
  Off the cuff, I'd estimate that between the team members at the time, we'd shipped perhaps 100k+ LOC of java into production systems at past jobs. It didn't seem interesting to rehash why we didn't want Java's verbosity in a small team.

mpdehaan2 11y ago

Kind of shocked that building database libraries for python (homebrew install gcc? pip install foo) was "hard" for them. I find the Python ORM options (sqlalchemy/Django/etc) lot better evolved than what is available for clojure and couldn't image that being a roadblock. Korma is basically a query builder and lacks a lot of things I was used to, and I had to engineer a declarative layer for defining models in the ways I wanted. Pain in setting up developer machines could be simplified with an ansible playbook pretty easily. OTOH, one of the worst things I hate is trying to figure out how to automatically get around Sun's paywalls and fiddle with Java classpaths :) I do feel some level of want to avoid vagrant in the development workflow, in which cases I'm usually fond of just logging into a real VM. I do think it's potentially good for tests. IDK, I always want to solve the idea of "fragile deploys" with immutable systems in prod. Uberjar was cool, I'll give it that -- but the startup time of clojure was often not worth the wait. My biggest complaints with clojure is documentation quality, inconsistency in return types (or just willingness to return nil in error scenarios), and tracebacks that are very difficult to sort through. Clojure also presents some problems with the idiomatic nature of trying to roll up everything into one statement, whereas adding debug and debugging require unrolling it, than packing statements up again to be socially acceptable idiomatic clojure. Testing in clojure also strikes me as painful, as it tends to punish you with "let" causing an indentation every time you need to assign a variable, something that is useful in tests that want to check results a lot along the way. My gut feeling is clojure was good for a little light scripting on top of a big Java codebase (kind of like you might use Lua in a game engine) but would not be something I'd pick for a primary language. Primarily using Java libs would also solve some of the ecosystem problems.

bpicolo 11y ago
sqlalchemy is my favorite library for pretty much any language. It's fantastic and a huge part of the python draw for me. The only downside is it doesn't play well with new async python tools, but python is totally fine at not being async
salsakran 11y ago
It's not hat "homebrew install gcc? pip install foo" is hard for us, it's that we didn't want to support 1000s of people trying to set it up on their machines. Every possible source of error we eliminated meant we'd need to spend less time on twitter debugging installation problems. It's useful to remember that our target install base isn't python developers who might want to hack away on a python project, but rather analysts, end users, DBAs and all kinds of people who've never heard of homebrew and who might not even be comfortable with using a terminal in Mac OS X let alone compiling things.

dustingetz 11y ago

"So, in summary, we chose Clojure for JVM threads + Database drivers, the ability to ship an uberjar and the ease of expressing tree manipulations which were the core piece of complexity in our backend." for people who may not know clojure here is an example of tree manipulations, it's really powerful especially in UI programming which is all about manipulating recursive structures: https://gist.github.com/dustingetz/c11ae4edb9e0d14787d2 https://gist.github.com/dustingetz/c11ae4edb9e0d14787d2 Scala has answers to this too (I would love for someone to reply with the best scala way) but for whatever subjective reasons the clojure way makes me happier for ui work. As to scala pushing towards typed object mappers, here is one example of generating untyped queries with Anorm [1], I'd be surprised if this couldn't be made to work for their query language, but that said the scala ecosystem has a lot of, shall we say "diversity of opinion" about the Proper Way To Do Things, so I am not surprised that newcomers to the language had a bad time. [1] (2013) http://www.dustingetz.com/2013/03/26/orm.html http://www.dustingetz.com/2013/03/26/orm.html

tunesmith 11y ago

Can someone expand on what they mean by lightweight threads for Clojure? My understanding is that lightweight threads (green threads) is what java/jvm explicitly moved away from way back when; it's user-mode threads bound to one processor, so if the processor gets busy, then all of the lightweight threads for that processor get blocked. Meanwhile, Java/jvm uses kernel threads and context-switching so you can do real multi-threading. I'm aware of how Quasar purports to be the best of both worlds, and it looks like Pulsar is a Clojure API for Quasar, maybe that's what they're using. It doesn't appear Scala has something comparative, but I have a hazy sense that Scala people believe that Akka and monads are better for easy asynchronous programming.

patrickthebold 11y ago
Clojure's core.async has "go routines", (this is more or less the same as in go), basically you can put and take messages on a channel. These operations will block the green thread, so you end up writing synchronous code. The idea is that synchronous, blocking code is easier to understand then async code. Since these are lightweight threads it will scale better than using real threads. I'm pretty sure it's fine to have hundreds of thousands of go routines. This abstraction sits on top of java's real threads, so you can have several real threads doing the work and they can switch between the go routines once the go routine hits one of the blocking operations. Or something like that... read this: http://clojure.com/blog/2013/06/28/clojure-core-async-channels.html http://clojure.com/blog/2013/06/28/clojure-core-async-channe...
astine 11y ago
It's likely that they're refering to core.async which is a Clojure library that provides IoC threads via the `go` construct.
_halgari 11y ago
Core.async uses code rewriting to turn blocks of code into state machines. From there puts and takes from channels become state transitions. It works almost exactly like async/await in C# or Yield based generators in other languages. The cool bit is that core.async does this as a library, not a compiler level feature.

RyanZAG 11y ago

> .. However, we had a strong aversion to using Java itself .. Honestly, why? Seems odd given Java's great track record for exactly this problem and huge library and developer share. The code itself is forced to be static and explicit which makes for easy onboarding for new developers. It's a bit strange to just dismiss Java in the context of the rest of the article. You're kind of staking your company on decisions like these, and making them because of some unquantifiable 'strong aversion' seems like a poor choice. That said, Clojure or other lisp base definitely seems like the correct choice for a query language / dsl. That's the shining point of lisps after all.

zem 11y ago
personally, i find java pretty painful to use; it's definitely capable of doing whatever job you have in mind, but when i use it i have the constant feeling i'm using a suboptimal tool. and then it leads to reams of boilerplate all of which has to be maintained, so it's not just a development-time cost; there's the depressing feeling of having to deal with it every time you want to read the code. clojure would be worth it just for the joy of using a tool that fits well into your hand.
- spacemanmatt 11y ago
  My experience is Java costs me around 10x the number of lines of code (and I mean code, not imports) to get the same things done. The brief syntax and higher-order programming eliminate so much from my screen that, in hindsight, served as little more than visual framing for sparse bits of code.
- RyanZAG 11y ago
  It's kind of like building a model with clay or with sticks. The clay model is going to look nicer with rounded curves and no wasted points. The stick model will have bits sticking out. However when you get 30 people working together to make a big model, the clay model is going to start looking very strange indeed. Different people will have different finger pressure and shape the clay differently. Some people will make more right angled parts with the clay which will clash with the parts of the clay that are carefully smoothed out. Someone might spend a week fixing a perfect parabola for a curve at the top of the model just for someone else to bend it off to the side with an ugly twist to fit with the other side of the model. 30 people on the stick model? Not really a problem. All the sticks look the same. Anybody who needs to attach one side of the stick model to something else just needs to grab a stick and tie it on. You won't see a perfect stick parabola, but then you also won't be stuck with a strangely twisted piece of clay. You can always create a style guide for using the clay - always keep between a 30 and 60 degree angle on joins. Apply one finger of pressure when doing joins. And so on. Sometimes this works with very diligent reviews that keep everything in place, but as soon as a crisis comes up or the model needs to pivot urgently? Everyone rushes to touch up the model in the quickest way possible and your model starts to get a bit thinner on one side than the other, etc. So there's nothing wrong with choosing a great clay model that fits well into your hand and you can shape effortlessly - but you need to remember that sometimes the 'painfulness' of using a model made out of sticks can be a big blessing when you're trying to massage a bit of clay that someone else pressed a bit too firmly on. EDIT: Also, when you've been making and using models for awhile, the stick models get an added benefit: you can browse through the stick models really fast based on only their outline and be able to understand very quickly how they work. Pretty clear why: the rigid sticks form a very distinct outline (think interfaces and forced directory structure in Java). Being a stick model expert therefore has very good job mobility and high demand. Each clay model is generally more unique and requires some study to understand even if it does look nicer.

daxfohl 11y ago

I'd be interested to understand how Korma solves their problem. It's macro-based so I wouldn't think it would be particularly good at dynamically restructuring queries based on runtime variables.

thom 11y ago
Yeah, clojureql was much better for this sort of thing but is to all intents and purposes abandoned.

stcredzero 11y ago

Now, choosing a programming language for a project is usually decided on a mix of what a team is most comfortable with, beer-fueled debates on dynamic vs static typing, benchmarks of marginal relevance and that elusive quest for Dev-Hipster points. How about this for an interview strategy: Have these discussions. Hire the people that realize most of the above mentioned things are meaningless. (Clarification: Only the "comfortable" part isn't meaningless. But there too, it's only as meaningful as the placebo effect is significant.)

ionforce 11y ago
How is "dynamic vs static typing" meaningless?
- stcredzero 11y ago
  For most applications, either can work. They both have to be managed correctly.
  - ionforce 11y ago
    What are "most applications"? How do they have to be managed?
    - stcredzero 11y ago
      http://bfy.tw/36aT http://bfy.tw/36aT
throwaway999888 11y ago
I don't think an interview strategy of the form "has the same values and opinions as stcredzero" is going to work for everyone.
- stcredzero 11y ago
  Of course not. Some people really like their illusions. But seriously: Having been part of a community that thought their language made them "super-programmers" I can say this: It's possible that on some level, the language does give a significant tooling advantage. However, this is often going to be completely swamped by various soft/human factors. A baroque quartet consisting of very mature musicians who work supremely well together is still going to produce far better music than a "super-group" of ego-driven martinets who secretly despise each other. It wouldn't matter how technologically advanced their instruments were over the 1st group, even if those instruments played themselves. It's the group that truly gels, has healthy interpersonal dynamics, has relevant experience, and healthy practice habits that is going to be the better musical group. Resolving the analogy left as an exercise.

programminggeek 11y ago

You know what else is easy to work with and deploy? PHP.

lectrick 11y ago
I wouldn't call a language that encourages writing spaghetti code and creating dependency hell to be "easy to work with", much less one which doesn't even pass its own test suite, which implies that even if you do everything 100% correctly, the spec itself is faulty, so you WILL have non-deterministic "3AM phone call, up till 3PM fixing" type bugs... No thanks

mark_l_watson 11y ago

I read this article yesterday because I am having a language choice issue, between Ruby and Clojure. I am in my (almost) mid 60s, and except for being on retainer for advice I don't really accept consulting work anymore. I have a long list of projects that I want to do, mostly open source, and settling on one language for a few years would make things easier. Clojure and Ruby both support DSL development very well which is important to me and both play very well with Java (via JRuby). I am starting to ask myself though if I wouldn't just be better off with Java 8, even without good support for writing DSLs. I have mostly stopped using Haskell and Scala. When I was much younger I totally enjoyed learning and using many programming languages but as I get older I feel more like just getting stuff done, especially since I dedicate much less time now to technical pursuits. So in a sense I am facing roughly the same type of decision as the Metabase people.

nkassis 11y ago
I'd personally look to find what language fills the gap that the current set of languages I know don't fill. Clojure is my filler for lisp language and language with good macros support. I use Ruby in the slot that many have Python in but with added coolness of metaprogramming not found in most scripting languages. And to add to that a bit, I found that recently it's been getting harder finding new languages with enough new patterns and paradigms to keep me interested in learning and mastering them.
bribri 11y ago
Out of curiosity, why not Haskell or Scala if you already know them?
daxfohl 11y ago
I follow your blog. I'm a month from 40 and agree. After several years working mostly in Clojure, Scala, and F#, I am finding myself most productive in traditional languages even though they're less interesting. And am finding projects more interesting than implementations, which is probably the way it should be. I'm finding now that OOP is actually a pretty good model for most things. Loops and mutation reflect reality: you do a task N times and write down each result; you don't take the first N results from an infinite seq of tasks. IoC is useful. In addition having a much larger set of things to copy/paste from github/stackoverflow is nice. That said, most (all?) of my paid work is pretty banal, nothing that's really begging for functional style. And for whatever reason I've never had much success with REPLs. Getting them updated correctly with the code and dependency changes, typing in what I want to test, then running it always seems to take me longer than just executing the app and testing changes directly.

iagooar 11y ago

It's great to see such an "exotic" programming language like Clojure getting some serious traction. And by exotic I mean a language that stands out from the imperative or object-oriented crowd. I like how Clojure challenges established paradigms with a surprisingly mature ecosystem. Being a hosted language on the JVM has its pros and cons, but the cleverness and pragmatism of Rich Hickey (creator of Clojure) really stands out. He decided to go with an industry strong, battle-tested environment, effectively standing on the shoulders of giants, while providing a lot of fresh ideas. Also, the fact of being a Lisp can be seen from different angles: for some it's lots of weirdly nested parentheses, for others it has a beautifully simple, yet extremely powerful and expressive syntax. But don't let those aspects fool you. Once you scratch a little bit the surface, you will see the extraordinary power of its persistent data structures. For those who don't know what persistent data structures are: in Clojure, oversimplifying, there is no mutation to data structures. Every time you "add" or "remove" a key from a hash map, you are actually creating a copy of it. And in order to make this efficient, you need special data structures with "superpowers", that are not actually creating a whole copy of themselves, but rather modify atomically only a part (hint: trees). And having those data structures leads to the next big feature of Clojure: concurrency. Not only does Clojure provide a mechanism to dispatch thread-like constructs (core.async), but the whole language is designed around supporting concurrency. I also want to add that what really surprised me about Clojure is its thriving community and smart people using it. I am pretty sure that if you want to hire great, highly motivated talent, going Clojure will help you with that.

troym 11y ago
> Once you scratch a little bit the surface, you will see the extraordinary power of its persistent data structures. I've heard this repeatedly, yet in the examples I've read (for web servers and for web ui work with cljs), they're using atoms with `swap!` to provide "current state". Could you provide an example of using persistent data structures in a web app? Or am I mis-understanding?
- danneu 11y ago
  You're compare-and-swapping out persistent values with an atom (it holds a succession of immutable values over time). It's not an example of people shirking persistent data structures for mutation.

giancarlostoro 11y ago

One of the things that I love about Clojure is Leiningen. It just takes away the headaches of setting up projects in Clojure. I think the only other tool I'm aware of for another language that is quite as useful is probably NuGet. I've never had issues using lein, just drop it into my ~/bin/ folder (I'm using openSUSE so that folder is in the system path by default), open up a terminal and type in lein and it works. I don't even need clojure installed, just the JVM, it will download clojure for you as needed! Props to those working on lein.

chii 11y ago
And yet maven does the same thing but gets no love from anyone (including me!)
- giancarlostoro 11y ago
  Not sure if Maven can be compared to lein, never personally used it, not knowingly at least. I've heard great things about Gradle though.
  - vorg 11y ago
    If Gradle's so great, then why does it use a JVM language that wasn't considered by Metabase. > The shortlist really came down to “something on the JVM” [...] However, we had a strong aversion to using Java itself, so the real short list was Scala or Clojure Gradle requires the configuration files to be written in Groovy, and is itself written in Java and lots of Groovy.
    - giancarlostoro 11y ago
      I do know Android Studio likes to rock Gradle. Clojure has Lein though, why would you use Maven then? :)
chii 11y ago
And yet maven does the same thing but gets no love from anyone (including me!)

johansch 11y ago

Has anyone here ever heard of Metabase before? Their github page indicates there are 3 (three) contributors. Wouldn't it be safe to assume this post is primarily a marketing venture?

escherize 11y ago
I've used Metabase before, it's a super easy way for a non-technical to do sql queries.
escherize 11y ago
I've used Metabase before, it's a super easy way for a non-technical to do sql queries.

staticelf 11y ago

This is probably going to be a very unpopular opinion here, but I think Clojure is a horrible language to work with. I don't have the experience with non imperitive languages which makes learning clojure a really steep curve. Also the error you can recieve with clojure are many times a big mystery, I've never seen as bad error reporting as I've seen with Clojure. Other issues I have with it is that Clojure-fans tends to want to write EVERYTHING in clojure or lisp style code. That includes stuff like Javascript, HTML and similar stuff. Things that just makes it even more complicated. I have really tried to learn it several times but got so frustrated I quit. We have a small thing at work which is written in Clojure and no one knows how to fix issues with it since no one has any experience with Clojure. Clojure does not fit my brain.

timroy 11y ago
I work only with Clojure and ClojureScript (including a simple DSL which lets me write HTML in Clojure-syntax). Error reporting in Clojure is bad - no two ways about it. Once I got used to scanning the JVM stack trace, it wasn't so bad. But it could be better. I really prefer writing ClojureScript to JS: using one language, with the same data structures, is a seamless experience. I like writing HTML in Clojure data structures, as it makes manipulating the data much easier. But your mileage may vary here! Clojure won't be my last language. :-)
chrislloyd 11y ago
I think your criticisms are definitely valid. The learning curve is steep (intentionally so) but definitely worth it. Your programs will become more reliable and you'll get more done as a consequence. Please push through! The error messages are terrible, and there are a lot of ongoing conversations about them at the moment in the Clojure community[1]. The Clojure[script] toolchain does tend to be all-in but there's work happening now to break that up — stay tuned. StackOverflow and the Clojure Slack channel also full of people who know how to fix issues :) Stick with it, when you'll have an "ah-ha!" moment soon enough! [1]: This talk is a good backstory as to why the errors are so bad, and useful no matter what language you use https://www.youtube.com/watch?v=FihU5JxmnBg https://www.youtube.com/watch?v=FihU5JxmnBg
jdeisenberg 11y ago

melipone 11y ago

Lisp has been around a while, Java has been around a while. Clojure is the perfect combination and I predict it will be around a while.

melipone 11y ago

Lisp has been around a while. Java has been around a while. Clojure is the perfect combination and I predict it will be around a while. It's exhilarating to program in Clojure. That's the only word I can find to describe what I feel to program in that language.

lectrick 11y ago

Why wasn't Erlang/Elixir even considered? It checks off all their initial set of requirements