8 ms·

Salient line: While Storm's Clojure implementation served it well for many years, it was often cited as a barrier for entry to new contributors. Storm's codeba

Salient line: While Storm's Clojure implementation served it well for many years, it was often cited as a barrier for entry to new contributors. Storm's codebase is now more accessible to developers who don't want to learn Clojure in order to contribute. Very common story that I hear from a lot of shops that tried to go Clojure first. Great language, but too little penetration amongst developers. Makes it very difficult to effectively recruit employees/contributors. At some point, you just accept that it's more important to get a new warm body than to continue pursuing some idealized programming perfection.

maltalex 7y ago
That's not the only point though, is it? > The new Java-based implementation has improved performance significantly, and made Storm's internal APIs more maintainable and extensible.
- jsmeaton 7y ago
  It’s unclear if the author meant that performance is better “because java” or “because rewrite”.
dleavitt 7y ago
This is salient too though: The new Java-based implementation has improved performance significantly, and made Storm's internal APIs more maintainable and extensible. It's not totally clear if this is because it's been rewritten Java, but the intrinsic qualities of the language do matter; there are real tradeoffs between maintainability and dynamism/flexibility. I feel like this sometimes gets shortchanged in these narratives.
- danarmak 7y ago
  The announcement says they have a new architecture. At the very least it's not a direct performance comparison to the same architecture implemented in Clojure. > Storm 2.0.0 introduces a new core featuring a leaner threading model, a blazing fast messaging subsystem and a lightweight back pressure model. It is designed to push boundaries on throughput, latency and energy consumption while maintaining backward compatibility. The design was motivated by the observation that existing hardware remains capable of much more than what the best streaming engines can deliver. Storm 2.0 is the first streaming engine to break the 1 microsecond latency barrier.
- bjoli 7y ago
  Idiomatic java is like idiomatic C: pretty fast. Most smaller languages prioritize other things. Clojure has immutability and Haskell has purity. I have noticed this with every "X produces code faster than C": you begin with two programs that use a suboptimal algorithm, then you take your non-c language and try to write C in it. The result is always awful and removes most reasons not to use C in the first place. This has somewhat changed with rust and in some sense C++,but for other languages my point still stands. They have a nice idiomatic golden path that is fast enough for most cases. Once you need performance badly enough you have to treat you language as an assembler, and then you will always lose to languages that actually are good at that. I say this as a scheme/Haskell weenie. Writing really performance scheme and Haskell code often means writing ugly code.
  - vbezhenar 7y ago
    Kotlin generally is not slower than Java. One exception is that it inserts a lot of null checks. And there are features that will make it even faster than Java, I'm talking about inline lambdas.
    - pvorb 7y ago
      The compiler is significantly slower than Java's, though.
      - vbezhenar 7y ago
        That's the sad truth for a lot of modern compiled languages (I think Go is an exception). While I did not dig into their compiler internals, it just seems an inevitable consequence from a powerful language (Go is pretty simple in that regard). C++, Rust, Scala, Kotlin, Swift, they all have significantly longer compilation times compared to their predecessors. Probably that's the price we have to pay.
        pjmlp 7y ago
        Not really. .NET Native, D, Delphi, Ada and Eiffel also have pretty fast compilers, while offering quite powerful languages. Lets even pick Turbo Pascal 7.0 for MS-DOS, which was quite feature rich, and was able to compile very large programs in a couple of seconds. Also Visual C++ with modules preview support, incremental compilation and linking is also reasonably faster than the FOSS alternatives. It is only a matter how much money one is willing to invest into improving tooling support.
- roenxi 7y ago
  Clojure is marketed as a very fast language (bunch of clever optimisation gets done because data structures are immutable). If the gains were from switching to Java they'd probably have said so less ambiguously. Surely what they mean is they've rewritten it to be faster, and switched to Java at the same time because they want to open up the contributor pool.
  - 0815test 7y ago
    BTW, if anyone's looking for another language that stresses immutability on the JVM, you might want to check out Frege, a Haskell-like language with some "tweaks" that make it work well in that environment. https://github.com/Frege/frege https://github.com/Frege/frege And as a sibling comment mentions, the announcement does clearly state that a rather extensive rewrite was done, including switching to a new architecture, with likely performance improvements coming from that - it's nowhere close to a pure Java vs. Clojure comparison.
    - fiddlerwoaroof 7y ago
      My impression is that Eta is a better choice, if only because it is compatible with a wider variety of Haskell libraries. https://eta-lang.org/ https://eta-lang.org/
      - 0815test 7y ago
        Eta is a better choice, if you care about Haskell compatibility in the strictest sense. Frege doesn't really pursue this, but the flip side is that it optimizes for working as smoothly as possible with the JVM ecosystem.
- didibus 7y ago
  I did find that bit interesting. It seems it's missing a leading "due to the commiters' better familiarity with Java." Since you can argue knowledge of a language does help with maintenance and extensibility. With this, it also makes more sense for the following bit about finding people able to contribute. No contributors is pretty bad for maintenance. At my work, we use Clojure, but I do often wonder what would happen if the current Clojure devs left. New devs unfamiliar with Lisps, FP, and Clojure would most likely have a hard time maintaining the code base and extending it, and if they try to rush it before really learning the language, would quickly degrade the quality of the code base, compounding the effect. I think as long as you continue to have one strong Clojure dev on the projects, you'll be fine, as they can direct newcomers and help them transition to Clojure, but if you lost that, I think a rewrite in a different language would make sense, or there's a risk of the code base degrading quite quickly.
nnnmnten 7y ago
Most CS programs have a course on functional programming or programming paradigms including functional programming. I don't understand why it would be a challenge to pick up Clojure.
- Mirioron 7y ago
  I don't know how it is elsewhere, but I saw many (most?) students struggle with those classes. These are people who have learned to successfully program, but they struggled quite significantly with functional languages (Haskell in this case). Even when they pass, many of them probably wouldn't want to pick up a functional language afterwards.
  - danieldk 7y ago
    But Scheme and Common Lisp (and its dialects) are drastically easier to learn than Haskell. Furthermore, I think it also has to do with the order in which these courses are taught. When I did a small introduction to FP at my university, a lot of people who had no prior experience in programming picked up Haskell much more easily than those who already had a background in imperative/OO programming. (Though that strengthens the point - FP is often quite a leap for programmers who have used imperative languages for a long time.)
- tanilama 7y ago
  What learnt in school is insufficient to meet with all real world challenges. To claim as a developer, you need to understand the internals, that is the ability to poke and trace to the root of the problem deep underneath. Debugging in FP feels different to most people, where our thoughts is tuned to the procedural model with observing states and form our hypothesis from there. Another disadvantage with niche programming language is not enough Stack Overflow help there. That alone will drive most people away.
- rienbdj 7y ago
  Many cs courses teach ML style functional programming, not Lisp or Clojure
- 7y ago
TeMPOraL 7y ago
It's a sad state of reality, because it drives down the quality of software to the common denominator of fresh junior devs, which is not much. Another view of this would be: progress in this industry is made by those who succeed while sticking to their guns and not optimizing their infrastructure for the available worker pool.
- JustSomeNobody 7y ago
  So, you’re saying the quality of the code dropped with the move to java? Could there be other reasons? Maybe they rushed?
  - dillonmckay 7y ago
    He is not saying Java caused it, just that the priority was cheaper labor, and a more popular language makes that more likely. This is in the context of software development in general, not this specific project per se.
- pron 7y ago
  > It's a sad state of reality, because it drives down the quality of software to the common denominator of fresh junior devs, which is not much. There's no reality here, just myth. That switching to Java would drive quality up makes at least as much sense as it driving it down. If you're referring to the particular languages, then the reality is that no big effect has been found for the choice of programming language on software quality. As lack of evidence in favor of a big effect is evidence of its lack, the most probable explanation is that no such effect exists (it's possible a small effect exists, but we don't know in which language's favor). And, indeed, while there is a theory explaining (and even predicting ahead of time) why there is no big effect to the choice of the programming language, not only is there no empirical evidence suggesting there is such an effect, there is no theory to predict it, either. That language has a big effect on quality is, at this point, no more than wishful thinking among a minority of developers who are big programming language fans. It's a bedtime story. (I am not saying that language didn't ever have a big effect or that it never will, just that both theory and observation show that there isn't one currently among "reasonable" languages in common use; also, I have used and I like both Java and Clojure) And if you're only referring to the priority of making the project more accessible, then I don't understand your comment at all. There are many, many more experienced Java developers than experienced Clojure developers, and experienced developers are at least as likely not to bother learning a new language in order to contribute to a project (although perhaps for different reasons). Picking a popular language makes the project more accessible to experienced developers. What has been shown to drive quality up or down is process. If you have a good process, then making the project more accessible can help, and if you don't then you're screwed anyway.
Jach 7y ago
At another point you just accept you'll have to hire people who don't fit the classic job requirement of "X years experience in Lang" and hire people who don't know Lang at all but want to learn. A growing Clojure shop in Seattle seems to be having success with this, onboarding the language is a small fraction of overall onboarding. On the more meta level it's weird what people expect devs to learn on the job or know up front. Companies expect employees to keep up with the latest nonsense in web dev, but can't pick up a new straightforward language? Unless maybe another Algol like Go or TypeScript? This Steve Yegge quote captures some of it: "For some reason, programmers love to learn new stuff, as long as it's not syntax."
- amortize 7y ago
  > A growing Clojure shop in Seattle seems to be having success with this, onboarding the language is a small fraction of overall onboarding. I can attest to this fact, as part of a unicorn upstart with dev offices in Sao Paulo and Berlin. Clojure onboarding has never been a major roadblock for new engineers. And we never hire asking for "X years of lang experience". Almost everyone in the (~300 strong) engineering team started with zero-to-little Clojure background.
  - mooreds 7y ago
    What do you look for? I mean the "X years of experience" is a shorthand proxy for "can get stuff done in the language we use". So I imagine you must have some other way of getting an idea of how a candidate can get things done. Would love to hear more about it.
    - amortize 7y ago
      What we have found is that it is enough to verify 'can get stuff done', and leave out 'in the language we use'. So, much of the interview turns out to be a process of validating two things: 1. can do what is claimed in the language of choice. 2. gauge enthusiasm and interest to pick up our language.
      - 7y ago
_cs2017_ 7y ago
How long does it take a strong developer to learn Clojure? Is it really a big a barrier to new contributors? Presumably people who consider contributing to the open source have some spare time and love programming. Wouldn't they find it exciting to contribute while also learning a cool new language?
- skywhopper 7y ago
  Maybe, but learning a new language by contributing to a large project built in that language is asking for trouble. Programming languages aren’t just syntax and stdlib. The relevant idioms and patterns may be significantly different. Even making small changes can be hard and intimidating. And then, even if you wanted to learn something new, to a lot of people, it matters what that something is. Unless the person is really set on contributing to Storm specifically, then the barrier of Clojure is still significant. They may prefer to spend their new language learning bandwidth on something with more momentum.
- yogthos 7y ago
  My experience from hiring and training co-op students for many years is that it takes around 2 weeks to learn with a bit of guidance. when a co-op student with little to know programming experience can learn Clojure in that time I find it surreal that there are professional developers who cannot.
jcadam 7y ago
As a Clojure enthusiast who has never been able to find a FT job using the language, I find it hard to believe that the demand for Clojure devs outstrips the supply. And even if you have to hire devs who don't already know the language I really don't understand the difficulty in learning Lisp - few languages are simpler or easier to grok.
- puredanger 7y ago
  The demand and supply for Clojure both exist, but they are not equally distributed geographically.
  - fnordsensei 7y ago
    Things are progressing, but the cultural addiction to shipping meat around is still too high in my opinion. Geography is a weird and often arbitrary constraint for knowledge work.
- emidln 7y ago
  I didn't have any problems finding work in Clojure when I was last looking (Aug-Sept 2018) with offers in four locales across the US. I was not able to find remote work at comparable salary for a US based org. (Some context) I have 7 years experience using Clojure professionally across 3 orgs (16 years overall) and was looking for individual contributor roles. Some things I observed about the Clojure companies: - Their HR seemed slower than the non-Clojure companies I was dealing with. I don't think this has anything to do with Clojure, but I am no longer working in Clojure full-time because the most credible offers took too long and I've found non-Clojure work I love. - Outside of trading firms, I did the most whiteboard coding questions with Clojure firms compared to shops looking to hire for Python or Elixir. I like whiteboard work, so this was fine for me. - 6 Clojure companies, 4 different conccurrent processing models and 6 different service frameworks. Neither of these is necessarily bad, but the community was much more fractured than I realized. - Of the 6 Clojure orgs, 4 were actively hiring (had open roles) and 2 were passively hiring whenever someone materialized on IRC/Slack/Twitter. This leads me to think if you want a Clojure job, you should probably reach out to people you know inside Clojure orgs and, barring that, the HR people at these orgs directly. This isn't a Clojure-specific thing, but worth keeping in mind. - 4 of 6 didn't seem to have dedicates DevOps, leading to them looking for candidates also familiar with deploying, monitoring, and debugging Linux hosts. I think I was given heavier weight at some orgs because of a background doing this type of work for Clojure stacks. Might be an angle worth looking into if you don't have production Clojure but you do have Linux and Java deployment/monitoring/debugging/tuning experience.
yogthos 7y ago
My team has been using Clojure for close to a decade, and we found the opposite to be the case. While the pool of applicants is smaller, so is the noise ratio. Clojure being niche means that you get people who are willing to look outside the mainstream, and are typically genuinely interested in programming. We also rarely hire devs who already know Clojure, and we typically just train on the job. It's really not hard to do when you already have people who know the language on the team. In fact, if somebody can't learn Clojure, I would question if they can learn to work on a large project in general. Dev practices, architecture, code styles, tooling, and so on tends to change quite a bit from company to company. The language is only small part of that. In case of Storm, Apache commons is run by Java devs who have zero interest in learning Clojure. So, it's not surprising they would rewrite Storm in their preferred language.
- fnordsensei 7y ago
  Same (sure; anecdotal) experience here. Hiring for Clojure has been so much easier for us than hiring for JavaScript. Granted, these are two different products, and maybe the Clojure one is just more attractive to the right kind of person. Nevetheless, in terms of quality vs. quantity of applicants, Clojure has been massively winning.
lvh 7y ago
FWIW, I have a very different read on this, based on both my experience with Clojure and some of the leading Storm-using orgs (and the Apache process). Firstly: Clojure isn't actually a problem. We hired people who are highly junior (one who even refused to call themselves a developer) and taught them Clojure. It was fine. It is true that you need a strong dev lead to guide them through that, but honestly, if you don't have a strong dev lead your Python/Java project is screwed too. Secondly: look at Storm's increased success at Java data processing sweatshop^H^H^H^H^H^Hconsultancies. Of course "can write Java" is what you're going to get if most of your business revolves around the lowest bidder.
Fellshard 7y ago
Same thing I repeat again and again: There is a false cost assigned to learning a language. Developers are too unwilling to even try stepping beyond the boundaries of the first thing they learned. The cost is always lower than they may think, and the benefits far surpassing what they may think. We've got to work at showing developers those benefits early; it's as important to creating software effectively as any other engineer's basic toolkit.
- mgkimsal 7y ago
  > Developers are too unwilling to even try stepping beyond the boundaries of the first thing they learned. The cost is always lower than they may think, and the benefits far surpassing what they may think. and yet... almost every time I've come in to a project where developers got past those boundaries.... they inevitably recreated patterns/idioms from tech 1 in to tech 2, and make tech 2 horrifically bad, and made it even harder for anyone who actually knew tech 2 to come in to the project and help. adding other languages is more than just 'get over it' - without guidance from experienced people who can understand good/efficient ways to model the problem domains unique to a business idiomatically in the new tech, you're very likely adding even more technical debt that you won't even know about for months/years. And... IME, the "let's use new stack X" are moved on to another project or company by the time the technical debt is apparent to everyone.
- Kye 7y ago
  I suspect programming has something similar to Gear/Plugin Acquisition Syndrome in photography and music. It's easy to let tinkering with new toys stand in for doing the actual work of creating. At some point, you have to sit down and do the work with the tools you have if you want to get anything done. If I were to get serious about programming, I would use C# or Go because I found them the most accessible after trying lots of languages. There's nothing wrong with finding a niche.
  - Fellshard 7y ago
    True, but I expect you to know which two or three /distinct/ tools you can work with effectively, and which situations they are most suitable for. Not only that, but knowing multiple languages has a multiplicative effect, massively expanding how you think of programming in all the languages you know, even the stable standbys like C#.
noir_lord 7y ago
> At some point, you just accept that it's more important to get a new warm body than to continue pursuing some idealized programming perfection. I think that is why Kotlin and Typescript are doing so well generally, they build on top of already established bases offering good new things without throwing out everything else. The trade off is they are constrained from how far they can diverge from their roots but on the flip side that is a strength as well. I like both, they feel productive without feeling totally alien.
moocowtruck 7y ago
another interesting part: The new Java-based implementation has improved performance significantly, and made Storm's internal APIs more maintainable and extensible.
hugi 7y ago
> At some point, you just accept that it's more important to get a new warm body than to continue pursuing some idealized programming perfection. Describes my dating life pretty well.
zmmmmm 7y ago
A more glass-half-full view of it is that it's a magnificent success for Clojure. Yes, it got redone in Java, but the fact is that Clojure took it from nothing to one of the world's premier frameworks for distributed computing - something that might have been impossible with the same resources if Java was its starting language. And because of the commonality of the underlying JVM, choosing Clojure allowed the functional approach with far less risk because the transition later was (I am assuming) much less risky and smoother. Looking at it in that light, this would be a huge boon to people considering starting new projects in Clojure (or possibly other JVM languages): it proves if they succeed and get so popular that moving to Java is the right thing, it can be done.
solussd 7y ago
Sometimes that backfires. - Clojure dev

maltalex 7y ago

That's not the only point though, is it? > The new Java-based implementation has improved performance significantly, and made Storm's internal APIs more maintainable and extensible.

jsmeaton 7y ago
It’s unclear if the author meant that performance is better “because java” or “because rewrite”.

dleavitt 7y ago

This is salient too though: The new Java-based implementation has improved performance significantly, and made Storm's internal APIs more maintainable and extensible. It's not totally clear if this is because it's been rewritten Java, but the intrinsic qualities of the language do matter; there are real tradeoffs between maintainability and dynamism/flexibility. I feel like this sometimes gets shortchanged in these narratives.

danarmak 7y ago
The announcement says they have a new architecture. At the very least it's not a direct performance comparison to the same architecture implemented in Clojure. > Storm 2.0.0 introduces a new core featuring a leaner threading model, a blazing fast messaging subsystem and a lightweight back pressure model. It is designed to push boundaries on throughput, latency and energy consumption while maintaining backward compatibility. The design was motivated by the observation that existing hardware remains capable of much more than what the best streaming engines can deliver. Storm 2.0 is the first streaming engine to break the 1 microsecond latency barrier.
bjoli 7y ago
Idiomatic java is like idiomatic C: pretty fast. Most smaller languages prioritize other things. Clojure has immutability and Haskell has purity. I have noticed this with every "X produces code faster than C": you begin with two programs that use a suboptimal algorithm, then you take your non-c language and try to write C in it. The result is always awful and removes most reasons not to use C in the first place. This has somewhat changed with rust and in some sense C++,but for other languages my point still stands. They have a nice idiomatic golden path that is fast enough for most cases. Once you need performance badly enough you have to treat you language as an assembler, and then you will always lose to languages that actually are good at that. I say this as a scheme/Haskell weenie. Writing really performance scheme and Haskell code often means writing ugly code.
- vbezhenar 7y ago
  Kotlin generally is not slower than Java. One exception is that it inserts a lot of null checks. And there are features that will make it even faster than Java, I'm talking about inline lambdas.
  - pvorb 7y ago
    The compiler is significantly slower than Java's, though.
    - vbezhenar 7y ago
      That's the sad truth for a lot of modern compiled languages (I think Go is an exception). While I did not dig into their compiler internals, it just seems an inevitable consequence from a powerful language (Go is pretty simple in that regard). C++, Rust, Scala, Kotlin, Swift, they all have significantly longer compilation times compared to their predecessors. Probably that's the price we have to pay.
      - pjmlp 7y ago
        Not really. .NET Native, D, Delphi, Ada and Eiffel also have pretty fast compilers, while offering quite powerful languages. Lets even pick Turbo Pascal 7.0 for MS-DOS, which was quite feature rich, and was able to compile very large programs in a couple of seconds. Also Visual C++ with modules preview support, incremental compilation and linking is also reasonably faster than the FOSS alternatives. It is only a matter how much money one is willing to invest into improving tooling support.
roenxi 7y ago
Clojure is marketed as a very fast language (bunch of clever optimisation gets done because data structures are immutable). If the gains were from switching to Java they'd probably have said so less ambiguously. Surely what they mean is they've rewritten it to be faster, and switched to Java at the same time because they want to open up the contributor pool.
- 0815test 7y ago
  BTW, if anyone's looking for another language that stresses immutability on the JVM, you might want to check out Frege, a Haskell-like language with some "tweaks" that make it work well in that environment. https://github.com/Frege/frege https://github.com/Frege/frege And as a sibling comment mentions, the announcement does clearly state that a rather extensive rewrite was done, including switching to a new architecture, with likely performance improvements coming from that - it's nowhere close to a pure Java vs. Clojure comparison.
  - fiddlerwoaroof 7y ago
    My impression is that Eta is a better choice, if only because it is compatible with a wider variety of Haskell libraries. https://eta-lang.org/ https://eta-lang.org/
    - 0815test 7y ago
      Eta is a better choice, if you care about Haskell compatibility in the strictest sense. Frege doesn't really pursue this, but the flip side is that it optimizes for working as smoothly as possible with the JVM ecosystem.
didibus 7y ago
I did find that bit interesting. It seems it's missing a leading "due to the commiters' better familiarity with Java." Since you can argue knowledge of a language does help with maintenance and extensibility. With this, it also makes more sense for the following bit about finding people able to contribute. No contributors is pretty bad for maintenance. At my work, we use Clojure, but I do often wonder what would happen if the current Clojure devs left. New devs unfamiliar with Lisps, FP, and Clojure would most likely have a hard time maintaining the code base and extending it, and if they try to rush it before really learning the language, would quickly degrade the quality of the code base, compounding the effect. I think as long as you continue to have one strong Clojure dev on the projects, you'll be fine, as they can direct newcomers and help them transition to Clojure, but if you lost that, I think a rewrite in a different language would make sense, or there's a risk of the code base degrading quite quickly.

nnnmnten 7y ago

Most CS programs have a course on functional programming or programming paradigms including functional programming. I don't understand why it would be a challenge to pick up Clojure.

Mirioron 7y ago
I don't know how it is elsewhere, but I saw many (most?) students struggle with those classes. These are people who have learned to successfully program, but they struggled quite significantly with functional languages (Haskell in this case). Even when they pass, many of them probably wouldn't want to pick up a functional language afterwards.
- danieldk 7y ago
  But Scheme and Common Lisp (and its dialects) are drastically easier to learn than Haskell. Furthermore, I think it also has to do with the order in which these courses are taught. When I did a small introduction to FP at my university, a lot of people who had no prior experience in programming picked up Haskell much more easily than those who already had a background in imperative/OO programming. (Though that strengthens the point - FP is often quite a leap for programmers who have used imperative languages for a long time.)
tanilama 7y ago
What learnt in school is insufficient to meet with all real world challenges. To claim as a developer, you need to understand the internals, that is the ability to poke and trace to the root of the problem deep underneath. Debugging in FP feels different to most people, where our thoughts is tuned to the procedural model with observing states and form our hypothesis from there. Another disadvantage with niche programming language is not enough Stack Overflow help there. That alone will drive most people away.
rienbdj 7y ago
Many cs courses teach ML style functional programming, not Lisp or Clojure
7y ago

TeMPOraL 7y ago

It's a sad state of reality, because it drives down the quality of software to the common denominator of fresh junior devs, which is not much. Another view of this would be: progress in this industry is made by those who succeed while sticking to their guns and not optimizing their infrastructure for the available worker pool.

JustSomeNobody 7y ago
So, you’re saying the quality of the code dropped with the move to java? Could there be other reasons? Maybe they rushed?
- dillonmckay 7y ago
  He is not saying Java caused it, just that the priority was cheaper labor, and a more popular language makes that more likely. This is in the context of software development in general, not this specific project per se.
pron 7y ago
> It's a sad state of reality, because it drives down the quality of software to the common denominator of fresh junior devs, which is not much. There's no reality here, just myth. That switching to Java would drive quality up makes at least as much sense as it driving it down. If you're referring to the particular languages, then the reality is that no big effect has been found for the choice of programming language on software quality. As lack of evidence in favor of a big effect is evidence of its lack, the most probable explanation is that no such effect exists (it's possible a small effect exists, but we don't know in which language's favor). And, indeed, while there is a theory explaining (and even predicting ahead of time) why there is no big effect to the choice of the programming language, not only is there no empirical evidence suggesting there is such an effect, there is no theory to predict it, either. That language has a big effect on quality is, at this point, no more than wishful thinking among a minority of developers who are big programming language fans. It's a bedtime story. (I am not saying that language didn't ever have a big effect or that it never will, just that both theory and observation show that there isn't one currently among "reasonable" languages in common use; also, I have used and I like both Java and Clojure) And if you're only referring to the priority of making the project more accessible, then I don't understand your comment at all. There are many, many more experienced Java developers than experienced Clojure developers, and experienced developers are at least as likely not to bother learning a new language in order to contribute to a project (although perhaps for different reasons). Picking a popular language makes the project more accessible to experienced developers. What has been shown to drive quality up or down is process. If you have a good process, then making the project more accessible can help, and if you don't then you're screwed anyway.

Jach 7y ago

At another point you just accept you'll have to hire people who don't fit the classic job requirement of "X years experience in Lang" and hire people who don't know Lang at all but want to learn. A growing Clojure shop in Seattle seems to be having success with this, onboarding the language is a small fraction of overall onboarding. On the more meta level it's weird what people expect devs to learn on the job or know up front. Companies expect employees to keep up with the latest nonsense in web dev, but can't pick up a new straightforward language? Unless maybe another Algol like Go or TypeScript? This Steve Yegge quote captures some of it: "For some reason, programmers love to learn new stuff, as long as it's not syntax."

amortize 7y ago
> A growing Clojure shop in Seattle seems to be having success with this, onboarding the language is a small fraction of overall onboarding. I can attest to this fact, as part of a unicorn upstart with dev offices in Sao Paulo and Berlin. Clojure onboarding has never been a major roadblock for new engineers. And we never hire asking for "X years of lang experience". Almost everyone in the (~300 strong) engineering team started with zero-to-little Clojure background.
- mooreds 7y ago
  What do you look for? I mean the "X years of experience" is a shorthand proxy for "can get stuff done in the language we use". So I imagine you must have some other way of getting an idea of how a candidate can get things done. Would love to hear more about it.
  - amortize 7y ago
    What we have found is that it is enough to verify 'can get stuff done', and leave out 'in the language we use'. So, much of the interview turns out to be a process of validating two things: 1. can do what is claimed in the language of choice. 2. gauge enthusiasm and interest to pick up our language.
    - 7y ago

_cs2017_ 7y ago

How long does it take a strong developer to learn Clojure? Is it really a big a barrier to new contributors? Presumably people who consider contributing to the open source have some spare time and love programming. Wouldn't they find it exciting to contribute while also learning a cool new language?

skywhopper 7y ago
Maybe, but learning a new language by contributing to a large project built in that language is asking for trouble. Programming languages aren’t just syntax and stdlib. The relevant idioms and patterns may be significantly different. Even making small changes can be hard and intimidating. And then, even if you wanted to learn something new, to a lot of people, it matters what that something is. Unless the person is really set on contributing to Storm specifically, then the barrier of Clojure is still significant. They may prefer to spend their new language learning bandwidth on something with more momentum.
yogthos 7y ago
My experience from hiring and training co-op students for many years is that it takes around 2 weeks to learn with a bit of guidance. when a co-op student with little to know programming experience can learn Clojure in that time I find it surreal that there are professional developers who cannot.

jcadam 7y ago

As a Clojure enthusiast who has never been able to find a FT job using the language, I find it hard to believe that the demand for Clojure devs outstrips the supply. And even if you have to hire devs who don't already know the language I really don't understand the difficulty in learning Lisp - few languages are simpler or easier to grok.

puredanger 7y ago
The demand and supply for Clojure both exist, but they are not equally distributed geographically.
- fnordsensei 7y ago
  Things are progressing, but the cultural addiction to shipping meat around is still too high in my opinion. Geography is a weird and often arbitrary constraint for knowledge work.
emidln 7y ago
I didn't have any problems finding work in Clojure when I was last looking (Aug-Sept 2018) with offers in four locales across the US. I was not able to find remote work at comparable salary for a US based org. (Some context) I have 7 years experience using Clojure professionally across 3 orgs (16 years overall) and was looking for individual contributor roles. Some things I observed about the Clojure companies: - Their HR seemed slower than the non-Clojure companies I was dealing with. I don't think this has anything to do with Clojure, but I am no longer working in Clojure full-time because the most credible offers took too long and I've found non-Clojure work I love. - Outside of trading firms, I did the most whiteboard coding questions with Clojure firms compared to shops looking to hire for Python or Elixir. I like whiteboard work, so this was fine for me. - 6 Clojure companies, 4 different conccurrent processing models and 6 different service frameworks. Neither of these is necessarily bad, but the community was much more fractured than I realized. - Of the 6 Clojure orgs, 4 were actively hiring (had open roles) and 2 were passively hiring whenever someone materialized on IRC/Slack/Twitter. This leads me to think if you want a Clojure job, you should probably reach out to people you know inside Clojure orgs and, barring that, the HR people at these orgs directly. This isn't a Clojure-specific thing, but worth keeping in mind. - 4 of 6 didn't seem to have dedicates DevOps, leading to them looking for candidates also familiar with deploying, monitoring, and debugging Linux hosts. I think I was given heavier weight at some orgs because of a background doing this type of work for Clojure stacks. Might be an angle worth looking into if you don't have production Clojure but you do have Linux and Java deployment/monitoring/debugging/tuning experience.

yogthos 7y ago

My team has been using Clojure for close to a decade, and we found the opposite to be the case. While the pool of applicants is smaller, so is the noise ratio. Clojure being niche means that you get people who are willing to look outside the mainstream, and are typically genuinely interested in programming. We also rarely hire devs who already know Clojure, and we typically just train on the job. It's really not hard to do when you already have people who know the language on the team. In fact, if somebody can't learn Clojure, I would question if they can learn to work on a large project in general. Dev practices, architecture, code styles, tooling, and so on tends to change quite a bit from company to company. The language is only small part of that. In case of Storm, Apache commons is run by Java devs who have zero interest in learning Clojure. So, it's not surprising they would rewrite Storm in their preferred language.

fnordsensei 7y ago
Same (sure; anecdotal) experience here. Hiring for Clojure has been so much easier for us than hiring for JavaScript. Granted, these are two different products, and maybe the Clojure one is just more attractive to the right kind of person. Nevetheless, in terms of quality vs. quantity of applicants, Clojure has been massively winning.

lvh 7y ago

FWIW, I have a very different read on this, based on both my experience with Clojure and some of the leading Storm-using orgs (and the Apache process). Firstly: Clojure isn't actually a problem. We hired people who are highly junior (one who even refused to call themselves a developer) and taught them Clojure. It was fine. It is true that you need a strong dev lead to guide them through that, but honestly, if you don't have a strong dev lead your Python/Java project is screwed too. Secondly: look at Storm's increased success at Java data processing sweatshop^H^H^H^H^H^Hconsultancies. Of course "can write Java" is what you're going to get if most of your business revolves around the lowest bidder.

Fellshard 7y ago

Same thing I repeat again and again: There is a false cost assigned to learning a language. Developers are too unwilling to even try stepping beyond the boundaries of the first thing they learned. The cost is always lower than they may think, and the benefits far surpassing what they may think. We've got to work at showing developers those benefits early; it's as important to creating software effectively as any other engineer's basic toolkit.

mgkimsal 7y ago
> Developers are too unwilling to even try stepping beyond the boundaries of the first thing they learned. The cost is always lower than they may think, and the benefits far surpassing what they may think. and yet... almost every time I've come in to a project where developers got past those boundaries.... they inevitably recreated patterns/idioms from tech 1 in to tech 2, and make tech 2 horrifically bad, and made it even harder for anyone who actually knew tech 2 to come in to the project and help. adding other languages is more than just 'get over it' - without guidance from experienced people who can understand good/efficient ways to model the problem domains unique to a business idiomatically in the new tech, you're very likely adding even more technical debt that you won't even know about for months/years. And... IME, the "let's use new stack X" are moved on to another project or company by the time the technical debt is apparent to everyone.
Kye 7y ago
I suspect programming has something similar to Gear/Plugin Acquisition Syndrome in photography and music. It's easy to let tinkering with new toys stand in for doing the actual work of creating. At some point, you have to sit down and do the work with the tools you have if you want to get anything done. If I were to get serious about programming, I would use C# or Go because I found them the most accessible after trying lots of languages. There's nothing wrong with finding a niche.
- Fellshard 7y ago
  True, but I expect you to know which two or three /distinct/ tools you can work with effectively, and which situations they are most suitable for. Not only that, but knowing multiple languages has a multiplicative effect, massively expanding how you think of programming in all the languages you know, even the stable standbys like C#.

noir_lord 7y ago

> At some point, you just accept that it's more important to get a new warm body than to continue pursuing some idealized programming perfection. I think that is why Kotlin and Typescript are doing so well generally, they build on top of already established bases offering good new things without throwing out everything else. The trade off is they are constrained from how far they can diverge from their roots but on the flip side that is a strength as well. I like both, they feel productive without feeling totally alien.

moocowtruck 7y ago

another interesting part: The new Java-based implementation has improved performance significantly, and made Storm's internal APIs more maintainable and extensible.

hugi 7y ago

> At some point, you just accept that it's more important to get a new warm body than to continue pursuing some idealized programming perfection. Describes my dating life pretty well.

zmmmmm 7y ago

A more glass-half-full view of it is that it's a magnificent success for Clojure. Yes, it got redone in Java, but the fact is that Clojure took it from nothing to one of the world's premier frameworks for distributed computing - something that might have been impossible with the same resources if Java was its starting language. And because of the commonality of the underlying JVM, choosing Clojure allowed the functional approach with far less risk because the transition later was (I am assuming) much less risky and smoother. Looking at it in that light, this would be a huge boon to people considering starting new projects in Clojure (or possibly other JVM languages): it proves if they succeed and get so popular that moving to Java is the right thing, it can be done.

solussd 7y ago

Sometimes that backfires. - Clojure dev