15 ms·

Storm 2.0

215 points by mnming 7y ago

pjmlp 7y ago
Sad to see it happen, however it is yet another example how platform languages always win in the end, even if due to their nature and market size they tend to be more conservative regarding feature adoption.
- skywhopper 7y ago
  I dare say it may be because they are more conservative regarding feature adoption.
avinium 7y ago
Salient line: While Storm's Clojure implementation served it well for many years, it was often cited as a barrier for entry to new contributors. Storm's codebase is now more accessible to developers who don't want to learn Clojure in order to contribute. Very common story that I hear from a lot of shops that tried to go Clojure first. Great language, but too little penetration amongst developers. Makes it very difficult to effectively recruit employees/contributors. At some point, you just accept that it's more important to get a new warm body than to continue pursuing some idealized programming perfection.
- maltalex 7y ago
  That's not the only point though, is it? > The new Java-based implementation has improved performance significantly, and made Storm's internal APIs more maintainable and extensible.
  - jsmeaton 7y ago
    It’s unclear if the author meant that performance is better “because java” or “because rewrite”.
- dleavitt 7y ago
  This is salient too though: The new Java-based implementation has improved performance significantly, and made Storm's internal APIs more maintainable and extensible. It's not totally clear if this is because it's been rewritten Java, but the intrinsic qualities of the language do matter; there are real tradeoffs between maintainability and dynamism/flexibility. I feel like this sometimes gets shortchanged in these narratives.
  - danarmak 7y ago
    The announcement says they have a new architecture. At the very least it's not a direct performance comparison to the same architecture implemented in Clojure. > Storm 2.0.0 introduces a new core featuring a leaner threading model, a blazing fast messaging subsystem and a lightweight back pressure model. It is designed to push boundaries on throughput, latency and energy consumption while maintaining backward compatibility. The design was motivated by the observation that existing hardware remains capable of much more than what the best streaming engines can deliver. Storm 2.0 is the first streaming engine to break the 1 microsecond latency barrier.
  - bjoli 7y ago
    Idiomatic java is like idiomatic C: pretty fast. Most smaller languages prioritize other things. Clojure has immutability and Haskell has purity. I have noticed this with every "X produces code faster than C": you begin with two programs that use a suboptimal algorithm, then you take your non-c language and try to write C in it. The result is always awful and removes most reasons not to use C in the first place. This has somewhat changed with rust and in some sense C++,but for other languages my point still stands. They have a nice idiomatic golden path that is fast enough for most cases. Once you need performance badly enough you have to treat you language as an assembler, and then you will always lose to languages that actually are good at that. I say this as a scheme/Haskell weenie. Writing really performance scheme and Haskell code often means writing ugly code.
    - vbezhenar 7y ago
      Kotlin generally is not slower than Java. One exception is that it inserts a lot of null checks. And there are features that will make it even faster than Java, I'm talking about inline lambdas.
- nnnmnten 7y ago
  Most CS programs have a course on functional programming or programming paradigms including functional programming. I don't understand why it would be a challenge to pick up Clojure.
  - Mirioron 7y ago
    I don't know how it is elsewhere, but I saw many (most?) students struggle with those classes. These are people who have learned to successfully program, but they struggled quite significantly with functional languages (Haskell in this case). Even when they pass, many of them probably wouldn't want to pick up a functional language afterwards.
    - danieldk 7y ago
      But Scheme and Common Lisp (and its dialects) are drastically easier to learn than Haskell. Furthermore, I think it also has to do with the order in which these courses are taught. When I did a small introduction to FP at my university, a lot of people who had no prior experience in programming picked up Haskell much more easily than those who already had a background in imperative/OO programming. (Though that strengthens the point - FP is often quite a leap for programmers who have used imperative languages for a long time.)
  - tanilama 7y ago
    What learnt in school is insufficient to meet with all real world challenges. To claim as a developer, you need to understand the internals, that is the ability to poke and trace to the root of the problem deep underneath. Debugging in FP feels different to most people, where our thoughts is tuned to the procedural model with observing states and form our hypothesis from there. Another disadvantage with niche programming language is not enough Stack Overflow help there. That alone will drive most people away.
  - rienbdj 7y ago
    Many cs courses teach ML style functional programming, not Lisp or Clojure
  - 7y ago
- TeMPOraL 7y ago
  It's a sad state of reality, because it drives down the quality of software to the common denominator of fresh junior devs, which is not much. Another view of this would be: progress in this industry is made by those who succeed while sticking to their guns and not optimizing their infrastructure for the available worker pool.
  - JustSomeNobody 7y ago
    So, you’re saying the quality of the code dropped with the move to java? Could there be other reasons? Maybe they rushed?
    - dillonmckay 7y ago
      He is not saying Java caused it, just that the priority was cheaper labor, and a more popular language makes that more likely. This is in the context of software development in general, not this specific project per se.
  - pron 7y ago
    > It's a sad state of reality, because it drives down the quality of software to the common denominator of fresh junior devs, which is not much. There's no reality here, just myth. That switching to Java would drive quality up makes at least as much sense as it driving it down. If you're referring to the particular languages, then the reality is that no big effect has been found for the choice of programming language on software quality. As lack of evidence in favor of a big effect is evidence of its lack, the most probable explanation is that no such effect exists (it's possible a small effect exists, but we don't know in which language's favor). And, indeed, while there is a theory explaining (and even predicting ahead of time) why there is no big effect to the choice of the programming language, not only is there no empirical evidence suggesting there is such an effect, there is no theory to predict it, either. That language has a big effect on quality is, at this point, no more than wishful thinking among a minority of developers who are big programming language fans. It's a bedtime story. (I am not saying that language didn't ever have a big effect or that it never will, just that both theory and observation show that there isn't one currently among "reasonable" languages in common use; also, I have used and I like both Java and Clojure) And if you're only referring to the priority of making the project more accessible, then I don't understand your comment at all. There are many, many more experienced Java developers than experienced Clojure developers, and experienced developers are at least as likely not to bother learning a new language in order to contribute to a project (although perhaps for different reasons). Picking a popular language makes the project more accessible to experienced developers. What has been shown to drive quality up or down is process. If you have a good process, then making the project more accessible can help, and if you don't then you're screwed anyway.
- Jach 7y ago
  At another point you just accept you'll have to hire people who don't fit the classic job requirement of "X years experience in Lang" and hire people who don't know Lang at all but want to learn. A growing Clojure shop in Seattle seems to be having success with this, onboarding the language is a small fraction of overall onboarding. On the more meta level it's weird what people expect devs to learn on the job or know up front. Companies expect employees to keep up with the latest nonsense in web dev, but can't pick up a new straightforward language? Unless maybe another Algol like Go or TypeScript? This Steve Yegge quote captures some of it: "For some reason, programmers love to learn new stuff, as long as it's not syntax."
  - amortize 7y ago
    > A growing Clojure shop in Seattle seems to be having success with this, onboarding the language is a small fraction of overall onboarding. I can attest to this fact, as part of a unicorn upstart with dev offices in Sao Paulo and Berlin. Clojure onboarding has never been a major roadblock for new engineers. And we never hire asking for "X years of lang experience". Almost everyone in the (~300 strong) engineering team started with zero-to-little Clojure background.
    - mooreds 7y ago
      What do you look for? I mean the "X years of experience" is a shorthand proxy for "can get stuff done in the language we use". So I imagine you must have some other way of getting an idea of how a candidate can get things done. Would love to hear more about it.
      - amortize 7y ago
        What we have found is that it is enough to verify 'can get stuff done', and leave out 'in the language we use'. So, much of the interview turns out to be a process of validating two things: 1. can do what is claimed in the language of choice. 2. gauge enthusiasm and interest to pick up our language.
        7y ago
- _cs2017_ 7y ago
  How long does it take a strong developer to learn Clojure? Is it really a big a barrier to new contributors? Presumably people who consider contributing to the open source have some spare time and love programming. Wouldn't they find it exciting to contribute while also learning a cool new language?
  - skywhopper 7y ago
    Maybe, but learning a new language by contributing to a large project built in that language is asking for trouble. Programming languages aren’t just syntax and stdlib. The relevant idioms and patterns may be significantly different. Even making small changes can be hard and intimidating. And then, even if you wanted to learn something new, to a lot of people, it matters what that something is. Unless the person is really set on contributing to Storm specifically, then the barrier of Clojure is still significant. They may prefer to spend their new language learning bandwidth on something with more momentum.
  - yogthos 7y ago
    My experience from hiring and training co-op students for many years is that it takes around 2 weeks to learn with a bit of guidance. when a co-op student with little to know programming experience can learn Clojure in that time I find it surreal that there are professional developers who cannot.
- jcadam 7y ago
  As a Clojure enthusiast who has never been able to find a FT job using the language, I find it hard to believe that the demand for Clojure devs outstrips the supply. And even if you have to hire devs who don't already know the language I really don't understand the difficulty in learning Lisp - few languages are simpler or easier to grok.
  - puredanger 7y ago
    The demand and supply for Clojure both exist, but they are not equally distributed geographically.
    - fnordsensei 7y ago
      Things are progressing, but the cultural addiction to shipping meat around is still too high in my opinion. Geography is a weird and often arbitrary constraint for knowledge work.
  - emidln 7y ago
    I didn't have any problems finding work in Clojure when I was last looking (Aug-Sept 2018) with offers in four locales across the US. I was not able to find remote work at comparable salary for a US based org. (Some context) I have 7 years experience using Clojure professionally across 3 orgs (16 years overall) and was looking for individual contributor roles. Some things I observed about the Clojure companies: - Their HR seemed slower than the non-Clojure companies I was dealing with. I don't think this has anything to do with Clojure, but I am no longer working in Clojure full-time because the most credible offers took too long and I've found non-Clojure work I love. - Outside of trading firms, I did the most whiteboard coding questions with Clojure firms compared to shops looking to hire for Python or Elixir. I like whiteboard work, so this was fine for me. - 6 Clojure companies, 4 different conccurrent processing models and 6 different service frameworks. Neither of these is necessarily bad, but the community was much more fractured than I realized. - Of the 6 Clojure orgs, 4 were actively hiring (had open roles) and 2 were passively hiring whenever someone materialized on IRC/Slack/Twitter. This leads me to think if you want a Clojure job, you should probably reach out to people you know inside Clojure orgs and, barring that, the HR people at these orgs directly. This isn't a Clojure-specific thing, but worth keeping in mind. - 4 of 6 didn't seem to have dedicates DevOps, leading to them looking for candidates also familiar with deploying, monitoring, and debugging Linux hosts. I think I was given heavier weight at some orgs because of a background doing this type of work for Clojure stacks. Might be an angle worth looking into if you don't have production Clojure but you do have Linux and Java deployment/monitoring/debugging/tuning experience.
- yogthos 7y ago
  My team has been using Clojure for close to a decade, and we found the opposite to be the case. While the pool of applicants is smaller, so is the noise ratio. Clojure being niche means that you get people who are willing to look outside the mainstream, and are typically genuinely interested in programming. We also rarely hire devs who already know Clojure, and we typically just train on the job. It's really not hard to do when you already have people who know the language on the team. In fact, if somebody can't learn Clojure, I would question if they can learn to work on a large project in general. Dev practices, architecture, code styles, tooling, and so on tends to change quite a bit from company to company. The language is only small part of that. In case of Storm, Apache commons is run by Java devs who have zero interest in learning Clojure. So, it's not surprising they would rewrite Storm in their preferred language.
  - fnordsensei 7y ago
    Same (sure; anecdotal) experience here. Hiring for Clojure has been so much easier for us than hiring for JavaScript. Granted, these are two different products, and maybe the Clojure one is just more attractive to the right kind of person. Nevetheless, in terms of quality vs. quantity of applicants, Clojure has been massively winning.
- lvh 7y ago
  FWIW, I have a very different read on this, based on both my experience with Clojure and some of the leading Storm-using orgs (and the Apache process). Firstly: Clojure isn't actually a problem. We hired people who are highly junior (one who even refused to call themselves a developer) and taught them Clojure. It was fine. It is true that you need a strong dev lead to guide them through that, but honestly, if you don't have a strong dev lead your Python/Java project is screwed too. Secondly: look at Storm's increased success at Java data processing sweatshop^H^H^H^H^H^Hconsultancies. Of course "can write Java" is what you're going to get if most of your business revolves around the lowest bidder.
- Fellshard 7y ago
  Same thing I repeat again and again: There is a false cost assigned to learning a language. Developers are too unwilling to even try stepping beyond the boundaries of the first thing they learned. The cost is always lower than they may think, and the benefits far surpassing what they may think. We've got to work at showing developers those benefits early; it's as important to creating software effectively as any other engineer's basic toolkit.
  - mgkimsal 7y ago
    > Developers are too unwilling to even try stepping beyond the boundaries of the first thing they learned. The cost is always lower than they may think, and the benefits far surpassing what they may think. and yet... almost every time I've come in to a project where developers got past those boundaries.... they inevitably recreated patterns/idioms from tech 1 in to tech 2, and make tech 2 horrifically bad, and made it even harder for anyone who actually knew tech 2 to come in to the project and help. adding other languages is more than just 'get over it' - without guidance from experienced people who can understand good/efficient ways to model the problem domains unique to a business idiomatically in the new tech, you're very likely adding even more technical debt that you won't even know about for months/years. And... IME, the "let's use new stack X" are moved on to another project or company by the time the technical debt is apparent to everyone.
  - Kye 7y ago
    I suspect programming has something similar to Gear/Plugin Acquisition Syndrome in photography and music. It's easy to let tinkering with new toys stand in for doing the actual work of creating. At some point, you have to sit down and do the work with the tools you have if you want to get anything done. If I were to get serious about programming, I would use C# or Go because I found them the most accessible after trying lots of languages. There's nothing wrong with finding a niche.
    - Fellshard 7y ago
      True, but I expect you to know which two or three /distinct/ tools you can work with effectively, and which situations they are most suitable for. Not only that, but knowing multiple languages has a multiplicative effect, massively expanding how you think of programming in all the languages you know, even the stable standbys like C#.
- noir_lord 7y ago
  > At some point, you just accept that it's more important to get a new warm body than to continue pursuing some idealized programming perfection. I think that is why Kotlin and Typescript are doing so well generally, they build on top of already established bases offering good new things without throwing out everything else. The trade off is they are constrained from how far they can diverge from their roots but on the flip side that is a strength as well. I like both, they feel productive without feeling totally alien.
- moocowtruck 7y ago
  another interesting part: The new Java-based implementation has improved performance significantly, and made Storm's internal APIs more maintainable and extensible.
- hugi 7y ago
  > At some point, you just accept that it's more important to get a new warm body than to continue pursuing some idealized programming perfection. Describes my dating life pretty well.
- zmmmmm 7y ago
  A more glass-half-full view of it is that it's a magnificent success for Clojure. Yes, it got redone in Java, but the fact is that Clojure took it from nothing to one of the world's premier frameworks for distributed computing - something that might have been impossible with the same resources if Java was its starting language. And because of the commonality of the underlying JVM, choosing Clojure allowed the functional approach with far less risk because the transition later was (I am assuming) much less risky and smoother. Looking at it in that light, this would be a huge boon to people considering starting new projects in Clojure (or possibly other JVM languages): it proves if they succeed and get so popular that moving to Java is the right thing, it can be done.
- solussd 7y ago
  Sometimes that backfires. - Clojure dev
zenlot 7y ago
"While Storm's Clojure implementation served it well for many years, it was often cited as a barrier for entry to new contributors. Storm's codebase is now more accessible to developers who don't want to learn Clojure in order to contribute." - interesting phrasing. Seems that the authors assuming that new contributors are more willing to learn Java instead of Clojure. And those who know Java find Clojure difficult to learn? Clojure, in fact is quite easy to grasp - especially for someone coming from Java, as you would still be working on JVM and always have Java Interop. I guess same thing goes to someone coming from Erlang to Elixir, where you are still working in the familiar ecosystem.
- dvfjsdhgfv 7y ago
  > Seems that the authors assuming that new contributors are more willing to learn Java instead of Clojure. No, many more people know Java than Clojure, that's it. Whether it's the right decision for the future of the project, time will show, but I don't think it was taken lightly.
- twblalock 7y ago
  Most new contributors already know Java.
- cygned 7y ago
  > Clojure, in fact is quite easy to grasp - especially for someone coming from Java I disagree on that point. In my experience as a programming language teacher, people often struggle to learn functional programming and even more to learn a Lisp, especially if they are used to a C-like language. And learning both at the same time is even more difficult. I liked working with Clojure but I have yet to meet someone who easily grasps the language and - more important - development patterns.
  - adz5a 7y ago
    Hello, this is an interesting take. Clojure has been a hobby of mine for some time but now I am a professional Clojure developper (clj/cljs). I knew the language before starting my new job but I found it really different than know the "platform": how a project is layed out, tests are written, some libs, how to pinpoint potential technical debt/suboptimal code accurately and so on... But I guess this would have been true if I had transition to any other new language. Would you have any example of what your trainees stumbled upon or found difficult in the language? That would be useful I guess when we try to onboard new people on the project. Have a nice day :)
agumonkey 7y ago
poor lisp, just like reddit ... always dropped for another language.
- sitkack 7y ago
  What if they are phases of evolution? A small team that can speak a lightweight language can quickly change velocity (speed and direction), iterating until a PoC is a useful system. Once the interfaces and the protocols fall into place, calcify or harden the design by converting to something more static, more literal. Imagine going from something like lisp to Ada? Or lisp to Rust? Write quick prototypes in lisp, solve the hard problems of understanding the domain, then convert portions that require correctness to a language that is amenable to correctness checking (property testing, model checking, etc). I think gradual typing [0] (not Gradual Typing) or language embedding [1] are steps in that direction. In a way, lisp can be thought of as a CASE tool for bootstrapping new systems. The great thing about language platforms (JVM, Beam, Graal, Wasm, Racket) is that many languages can easily exist in the same system so that in many cases, code can be converted on a function by function basis. [0] http://mypy-lang.org http://mypy-lang.org [1] http://terralang.org http://terralang.org
- fulafel 7y ago
  The SW world is full of Java/C++ rewrites of products that originally started out in a dynamic & productive language like Lisp or Python. It might make sense in the lifecycle phase when you're past making big changes under make-or-break pressure, and are in the "provide predictable 10 year maintainance roadmap for this large codebaes" stage. You end up employing different kinds of programmers too because many of the original team may be reassigned to something more exciting/important to do. But it's in no way inevitable, of course, there are many old Lisp, Python, etc codebases in production too. BTW. I've been trying to rediscover this blog post and can't seem to find it again, anyone? It described different kinds of software projects, the (1) kind that try to deliver incremental improvements (eg 10% cost savings), and the (2) kind that try to make 10x improvements, and the difference in how those projects are run (the 1-type project can't go over budget because it will then deliver negative value etc).
  - 7y ago
juskrey 7y ago
Apache products have always been a result of a blind majority rule, and thanks to that many of them are a corporate-style incomprehensible mess. They favor the process instead of a result: Apache things need to show constant progress to be afloat. Indeed, adopting Clojure for the developer product of this style is a dead end. Clojure is a minority rule. A slow tinkerer's and a lazy (in a good sense) practitioner's tool, whose aim is to survive in a real world, not to show a steady movement no matter what.
- Shmebulock 7y ago
  These Talebian references got me thinking. Couldn't you argue that it shows a certain arrogance and non-results oriented "falling in love with ideals and tools" detachment from reality to use Clojure instead of good ol' Java?
  - krn 7y ago
    > non-results oriented I think Clojure's ecosystem is more results-oriented than almost any other currently available[1]. It just requires a completely different mindset to understand, which an average software developer lacks. Therefore, Java is a much better fit for an Apache project. [1] https://news.ycombinator.com/item?id=19728502 https://news.ycombinator.com/item?id=19728502
  - jrs95 7y ago
    Oh I'm sure people make that argument all the time. I just don't think it holds up to reality. It's sort of just a dismissal of better options. The reality in my experience is that Java is just a poor choice for a lot of software. That's why you end up with frameworks built upon frameworks & AspectJ to basically make it behave like a dynamic language and try to hide some of the ugliness of the underlying APIs from you. And it's just because it's corporate approved and "everyone knows Java". I don't see how it can reasonably be considered the pragmatic choice. It's just not productive, and the final product isn't something that I would consider easy to understand or maintain most of the time either.
  - TeMPOraL 7y ago
    > Couldn't you argue that it shows a certain arrogance and non-results oriented "falling in love with ideals and tools" detachment from reality to use Clojure instead of good ol' Java? Since when trying to do better than using a tool designed for mass-producing software with dumb and cheap labor is "falling in love with ideals and tools"? It's kind of like saying that using an excavator over a bunch of people with shovels is "falling in love with ideals and tools". No, it just allows a single person to get more work done faster and better. Our industry is weird. When recruiting, companies claim they want "the best of the best" and will often test you on ridiculous stuff. But when it comes to actual work, on industry scale, learning, growing professionally, and doing things in a clean and efficient way is frowned upon. Best take the weakest tool available and compensate for its shortcomings with third party services.
ian1321 7y ago
Java is statically typed. Clojure is not. To me, this is the biggest difference and the reason to choose Java.
- deleted 7y ago
  [deleted]
- jnwkw 7y ago
  Is this true or false? I don't understand the downvotes.
  - Rovanion 7y ago
    While it is true that Java is statically typed and Clojure is dynamically typed it is perhaps one of the less important differences between the two languages. If you in fact would like to add a static type system on top of Clojure there is a way to do that: Typed Clojure. https://typedclojure.org/ https://typedclojure.org/
  - pyb 7y ago
    Many people disagree with "the reason to choose Java".
  - CodeArtisan 7y ago
    https://danluu.com/empirical-pl/ https://danluu.com/empirical-pl/
  - innocentoldguy 7y ago
    I'm not going to downvote the comment, but I think the "statically typed" argument is fantastically overrated. I spent years of my life coding in Java and C# and other years of my life writing Clojure, Python, Ruby, and Elixir code. I've never once thought to myself of the latter, "Damn! This would be so much better if I could have static types." It has literally never, not once, ever been an issue for me.
- roshan_naik 7y ago
  In the critical path many things can matter that ordinarily don't. When working on perf improvements for Storm 1.0 release, profiling revealed hotspots in Clojure code. Adding type hints to avoid dynamic typing fixed some of these hotspots. In other places, Clojure primitives showed up as a bottleneck. These required rewriting some code in Java and invoking it from Clojure. This was mentioned briefly here: https://hortonworks.com/blog/microbenchmarking-storm-1-0-performance/ https://hortonworks.com/blog/microbenchmarking-storm-1-0-per... The Clojure based core has served the community well for some time. It was still powered by Disruptor (a fast Java messaging library). So there was always some back and forth between Java and Clojure. For Storm 2.0, the Java rewrite happened first. The re-architecture work might have not happened without it (at least for the foreseeable near future)... primarily due to Java expertise being more easily available.
jaimex2 7y ago
If you're going to be on the JVM, stick to what the J stands for :)
- lvh 7y ago
  The newest JVM out of Oracle, GraalVM, has its polyglossy as its central feature. And Clojure has always been a lot closer to Java (in terms of quality of interop) than even e.g. Scala, let alone something like Jython. (Notable exception JRuby.)
rcaught 7y ago
I struggle to see how you pick Java over Kotlin on the JVM these days. It's everything that Java is, minus everything Java shouldn't be, plus everything that Java should be.
- jrs95 7y ago
  It's the lowest common denominator. It's the same reason I haven't been able to adopt Kotlin at work. Everyone acknowledges it would be better, but "everyone knows Java". Seems almost like "nobody ever got fired for buying IBM" to me.
  - cutler 7y ago
    ... or, worse, "everyone here uses Java <= 8".
- ivan_gammel 7y ago
  Syntactic sugar does not solve programming problems. Kotlin does not offer any paradigm changes which could make switch worth it.
  - deleted 7y ago
    [deleted]
  - vbezhenar 7y ago
    The only Kotlin feature that it's really hard to live without is properties. Otherwise sure, it's better than Java 11, but I can live without all those niceties. And (non)nullable types even get in my way! I just don't understand where there's syntax sugar for propeties in Java. There are lot of unneeded stuff added, that I don't really care about. But my classes are still full of autogenerated getters/setters. I recently tried to find some JEP and I did not find one. There was some talk around Java 7, but nothing since that. Nobody even wants to add properties support for Java?
    - sverhagen 7y ago
      And (non)nullable types even get in my way! Interesting how that goes...: the sibling comment to yours said the exact opposite. How does this feature get in your way?
zubairq 7y ago
We did the same with our open source project. We ditched Clojure and Clojurescript for NodeJS and JavaScript, for the same reason as it was a huge barrier to entry for new users
unlogic 7y ago
To me, this makes total sense as the project moved to Apache. Obviously, much more people will be able to consider contributing when it's in Java. Apache goal is sustainability and long-term viability, and Java would work better for that. I also consider this a success story for Clojure. It gives Clojure another usecase: a "production-ready prototype" language where the resulting "prototype" can last for eight years and benefit thousands of developers until it gets rewritten to something else when all the hard questions are answered, and most experimentation/wandering is over.
- sitkack 7y ago
  See my response inspired by your comment and agumonkey here https://news.ycombinator.com/item?id=20074532 https://news.ycombinator.com/item?id=20074532 I am drawn to the idea of scalable languages or languages with zero friction for interop. This thread on Pallene is interesting https://news.ycombinator.com/item?id=18038619 https://news.ycombinator.com/item?id=18038619
- usgroup 7y ago
  I think it’s noteworthy that whatever the benefits of clojure it didn’t outweigh the adoption issue. It’s quite a mature language so I’m not sure that bodes we’ll for its prospects this late in the game.
  - stingraycharles 7y ago
    What if Clojure is a language that bodes well especially early in a project’s lifecycle?
    - usgroup 7y ago
      I think it’s very difficult to motivate going out of your way to hire decent devs that know clojure and java only to throw away the clojure some time down the road. Begs the question, why not just write it in Java? Then at least its more likely to be a refactor down the road rather than a rewrite. I think clojure ought to be a production ready language that scales well. That’s what it was designed to be. However lisp seems to dichotomise devs into those that get it and those that don’t and thus alienates many would be team members.
hanswesterbeek 7y ago
Clojure just seems "too weird" to too many people, albeit for all the wrong reasons. Its syntax looks like a 6ft hurdle to newcomers, especially if they have lots of experience in C-synthax-languages. But anyone who just /tries/ to jump it finds out the hurdle really isn't that high at all. This dynamic will probably never change. Clojure will remain an acquired taste, which is fine, I guess. Those 'in the know' will happily continue to develop effectively with it.
keymone 7y ago
Bet they had to reimplement half of Clojure when rewriting storm core.
- juskrey 7y ago
  Yep, at last a lot of progress to report to Apache Board of Directors
- eecc 7y ago
  I see what you did there
readme3 7y ago
Apart from the language part , i am super excited about trying storm-2.0. In many ways the buzz in the ecosystem has been that storm was dead!. We run lot of prod workloads on storm and spark , storm has worked like a wonderful workhorse without quirks. The ecosystem has moved on , but i think storm may still continue having loyalists!
- scribu 7y ago
  > The ecosystem has moved on Just ouf of curiosity, what tools are people using instead of Storm?
  - tanilama 7y ago
    Flink is a strong competitor. Spark itself also claims to be a runner up with minibatching. But streaming model is more widely adopted outside of frameworks. AWS Lambda or Fass platform can be viewed as streaming model with simple topology.
  - nimrody 7y ago
    Flink is popular. I believe Samza also fills a similar niche. There's also Kafka Streams which is suitable for many cases that do not warrant a full cluster with all the management and trouble required.
- grkg8tr 7y ago
  I thought Heron was the Storm follow-up. I'm surprised to see a Storm 2.0.
fulafel 7y ago
cutting off 1.0 immediately sounds quite harsh, I wonder what kind of user base it has? "EOL for 1.0.x With the release of 2.0.0 the 1.0.x version line will no longer be maintained."
- Hendrikto 7y ago
  > It is designed to push boundaries on throughput, latency and energy consumption while maintaining backward compatibility.
- JanecekPetr 7y ago
  That's because they'll be maintaining the 1.1.x and 1.2.x branches...
- kabhwan 7y ago
  Many big projects tend to manage around 3 version lines. Since Storm 2.0 is out, there're three version lines for 2.x, 1.2.x, 1.1.x. 1.0.x could be EOL-ed.
mrkeen 7y ago
In this thread: * Non-Java JVM languages are too different for people to use. Also: * It's not worth switching from Java to other languages because they're just Java with syntactic sugar.
- skywhopper 7y ago
  Clojure and Kotlin are two very different languages. I don’t see anyone making the broad statements you are calling out.
  - mrkeen 7y ago
    No, of course not. I misread > Syntactic sugar does not solve programming problems. Kotlin does not offer any paradigm changes which could make switch worth it. and > Also, java 8 and above is getting a lot leaner to use, api, linguistics .. so the gap with clojure is a bit less than before.
gigatexal 7y ago
How am I just learning about this? This seems to be rather novel. Real-time bolt on for any database? Pretty cool stuff.
grego 7y ago
In the old days a project would often start in Lisp (Common Lisp for example), and then later be rewritten in C. This is the same pattern, and one should embrace it. Write the first cut, explore the problem space and prototype in a dynamic language. Once the design is proven, rewrite in statically typed language for performance and to get the possible remaining bugs out. If the design was perfect from the get go, you could just write it in assembler to start with.
amortize 7y ago
> While Storm's Clojure implementation served it well for many years, it was often cited as a barrier for entry to new contributors. Storm's codebase is now more accessible to developers who don't want to learn Clojure in order to contribute. It is interesting to contrast this with state of affairs in Apache Spark. Spark has thrived well in spite of being a Scala project; Scala arguably has a higher barrier for entry compared to Clojure (although the flavour of Scala used within Spark closely resembles Java).
_cs2017_ 7y ago
Why change the title of the post? Almost every one of the ~100 comments is in response to the original title that Storm switched from Clojure to Java. Now this context is lost.
- lvh 7y ago
  Because the previous title was editorialized? Furthermore, perhaps the conversation is about the old title because it was, and now the conversation can go be about Storm itself :)
  - _cs2017_ 7y ago
    I think the rule about not editorializing needs an exception so that someone can point out an important or interesting fact using an article whose original title is too generic. Another example would be "Press release from XYZ". If it talks about XYZ reaching some major breakthrough, the title should be allowed to reflect that. It's a waste of everyone's time to post the article titled "Press release". And arguably this situation should fall under such an exception.
jacobtracey 7y ago
What exactly are you doing?
ww520 7y ago
While most comments touch on the developer issue, the significant performance gain deserves more discussion. I guess Java's static type, pragmatic mutability with less garbage produced, and efficient parallelism give substantial boost to performance. It would be interesting to see what the upcoming blogs have to say about the performance gain.
- yogthos 7y ago
  My guess would be that it's always easier to make a rewrite performant because you already have a working version and know exactly where bottlenecks are. I imagine rewriting it in Clojure, or any other language, would've resulted in a significant performance boost as well.
grkg8tr 7y ago
How does this relate to Apache Heron? I thought Heron was essentially Storm 2.0. Which one should I be using?
- roshan_naik 7y ago
  Not well known is that Heron's performance claims in the "flying faster" blog and SIGMOD paper were made by comparing against a fork of an older (pre apache) version of Storm even though much more performant version was already out. IMO (biased?) Heron makes surprisingly poor design choices for their key distinguishing features like threading model and backpressure. And it shows when you benchmark it. More specifics: https://github.com/apache/storm/pull/2241#issuecomment-317879907 https://github.com/apache/storm/pull/2241#issuecomment-31787...
- kabhwan 7y ago
  Given there're so many streaming frameworks with different characteristics, you should be the one who understands the difference and select one fits for your business use case. There's no silver bullet.
didibus 7y ago
Can someone enlighten me on the inner working of Apache? Do they hire developers to maintain and work on the projects? If not, is there community leads in charge of each one? How are they appointed?
- geodel 7y ago
  http://www.apache.org/foundation/governance/ http://www.apache.org/foundation/governance/
  - didibus 7y ago
    Interesting, where would you go to see the list of committers for a given project and find information about them? I'm curious if the commiters are Clojure developers who chose to switch to Java, or if they are Java devs who decided to rewrite the project to a language they are more familiar and comfortable in. Or if they are Clojure devs looking to transition the project to other commiters and they struggled to find other Clojure devs willing to take it over.
snorremd 7y ago
> While Storm's Clojure implementation served it well for many years, it was often cited as a barrier for entry to new contributors. Storm's codebase is now more accessible to developers who don't want to learn Clojure in order to contribute. While I appreciate that programmers have limited free time and may not want to learn a new language to contribute to open source, I feel like learning Clojure could be of value to the programmer in itself. The supposition that Clojure acts as a big barrier against contribution from developers might be true, but I really feel like developers who only write code in the ALGOL family of languages are missing out. Learning a lisp language is not an impossible task and can improve your skills as a developer.
didibus 7y ago
It is interesting to read the history of Storm from the original author (Nathan Marz) himself: http://nathanmarz.com/blog/history-of-apache-storm-and-lessons-learned.html http://nathanmarz.com/blog/history-of-apache-storm-and-lesso... Clojure allowed him to build the initial Storm release by himself in only 5 months. > I made all of Storm's APIs in Java, but implemented Storm in Clojure. By keeping Storm's APIs 100% Java, Storm was ensured to have a very large amount of potential users. By doing the implementation in Clojure, I was able to be a lot more productive and get the project working sooner. Also interesting is that Nathan has now founded a startup working on something which sounds like the next evolution of Storm, and he's building it in Clojure again: https://redplanetlabs.com/ https://redplanetlabs.com/ They recently got 5M in funding and are hiring. Building a Clojure team for it. Personally, I feel the Storm 2.0 release rewrite to Java is really just a case of the new maintainers not knowing Clojure very well, and being primarily Java devs, looking for contributions from people using Storm, which tend itself to be mostly Java shops. Clojure is not quick to pick up, and very different from Java, so unlike Scala, Kotlin, C#, Go, etc. Someone with a Java background and no Clojure or Lisp experience won't be able to contribute as easily. And that's fine, and possibly best for Storm's future now that Nathan has moved on. Unless a Clojure dev had stepped up and adopted Storm, this change isn't at all surprising.
sandra101 7y ago
My ex ruined my credit due to his incessant extravagant spending spree, I found myself in a big mess. I talked to a credit repair company and I was told that it would take me non less than a year to fix my credit. I was devastated, that's a very long time which I can't cope with. I looked online and came across Neo's contact, took the bold step of messaging him even though I was skeptical about the whole thing and to my greatest surprise, my credit was repaired 3 working days, my credit was restored from 337 -710. Tbh I was shocked and it didn't cost me too much. I advise you to contact him on (neogonzalezhacks at g mail dot com )for all credit issues, he replies quickly and you can text him on +1 773 688 9662 .
mikewills23 7y ago
be careful of whom you contact to help you, he is a a special hack which have is own ways of hacking that nobody has ever imagine in this hacking world with his spare idea he can hack anything. he is totally secured and your security comes first. Hacking a website is a job for pro like (neogonzalezhacks@gmail.com) or text him +1 773 688 9662 - ANY TYPE LOANS -Database hack. Remove criminal records. facebook hack. -gmail hack -whatsapp hack -tracking calls -online hacking lectures -Cloning of phones -increasing credit cards scores -Twitter hack EMAIL : NEOGONZALEZHACKS@GMAIL.COM
fulafel 7y ago
For anyone interested in the Clojure history, there's this interview of Nathan Marz, originalcreator of Storm, from 5 years ago: https://www.infoq.com/interviews/marz-lambda-architecture/ https://www.infoq.com/interviews/marz-lambda-architecture/
kabhwan 7y ago
DISCLAIMER: I'm one of PMC member of Apache Storm. Even though Clojure has been Nathan's first-class language, not everyone in contributors and even committers couldn't become native on Clojure. (Though I guess part of committers could feel Clojure as native.) It might not be a problem when Nathan builds most of things in Storm (I'll link his blog post which describes lesson learned from Storm on his perspective http://nathanmarz.com/blog/history-of-apache-storm-and-lessons-learned.html http://nathanmarz.com/blog/history-of-apache-storm-and-lesso...), but when Storm becomes popular and also no longer his toy project, Clojure matters for scalibility of the project. Personally, I had a hard time understanding plenty of macros and even chained macros - that might not be a problem if I were allowed to have plenty of times to study Clojure and get used to it, but possible contributors don't get a chance when they're ready. In many cases, the first time they read the code on what they're using is when they encounter issues on stage/production and have to dig with stack trace. For me, I had to investigate on Storm because my team suffered from lack of documentation and also lack of knowledge on internal which ended up just taking workarounds. It was pretty hard time learning language to just read and understand the details to track down the issue. I even couldn't think of writing code on Clojure. When I decided to contribute Apache Storm, I started from connectors and CI build which wasn't written on Clojure. After some months I was able to read and understand the code to get into details but I still couldn't be native for Clojure, so still required N times of efforts to read and write the code even after I became one of PMC member. Don't get me wrong. I'm not saying something is better than some other one. In scalability point of view on project, I guess we had to do this, even though it ended up teaching me some lesson that migrating language is not the thing which can be considered easily, especially for a big project. Lots of efforts were spent here. Someone could claim for Apache Spark's case for Scala. Yes, I'm also contributing to Apache Spark and I had to learn Scala (fortunately I had a chance to do that before contributing) but the case is different. If you read Databricks Scala guide in details, you might realize that the doc is actually not encouraging contributors to use advanced features (a.k.a the features only Scala geeks might feel natural) on Scala. Same goes on reading codebase. Apache Spark community has been trying to keep necessary knowledges on Scala to contribute Apache Spark low enough, so that non-expert of Scala engineer could easily try out contribution. If they have been requiring hard understanding of Scala to accept contributions - Spark may not be able to get its amazing reputation.

215 points by mnming 7y ago

Sad to see it happen, however it is yet another example how platform languages always win in the end, even if due to their nature and market size they tend to be more conservative regarding feature adoption.

skywhopper 7y ago
I dare say it may be because they are more conservative regarding feature adoption.

avinium 7y ago

Salient line: While Storm's Clojure implementation served it well for many years, it was often cited as a barrier for entry to new contributors. Storm's codebase is now more accessible to developers who don't want to learn Clojure in order to contribute. Very common story that I hear from a lot of shops that tried to go Clojure first. Great language, but too little penetration amongst developers. Makes it very difficult to effectively recruit employees/contributors. At some point, you just accept that it's more important to get a new warm body than to continue pursuing some idealized programming perfection.

maltalex 7y ago
That's not the only point though, is it? > The new Java-based implementation has improved performance significantly, and made Storm's internal APIs more maintainable and extensible.
- jsmeaton 7y ago
  It’s unclear if the author meant that performance is better “because java” or “because rewrite”.
dleavitt 7y ago
This is salient too though: The new Java-based implementation has improved performance significantly, and made Storm's internal APIs more maintainable and extensible. It's not totally clear if this is because it's been rewritten Java, but the intrinsic qualities of the language do matter; there are real tradeoffs between maintainability and dynamism/flexibility. I feel like this sometimes gets shortchanged in these narratives.
- danarmak 7y ago
  The announcement says they have a new architecture. At the very least it's not a direct performance comparison to the same architecture implemented in Clojure. > Storm 2.0.0 introduces a new core featuring a leaner threading model, a blazing fast messaging subsystem and a lightweight back pressure model. It is designed to push boundaries on throughput, latency and energy consumption while maintaining backward compatibility. The design was motivated by the observation that existing hardware remains capable of much more than what the best streaming engines can deliver. Storm 2.0 is the first streaming engine to break the 1 microsecond latency barrier.
- bjoli 7y ago
  Idiomatic java is like idiomatic C: pretty fast. Most smaller languages prioritize other things. Clojure has immutability and Haskell has purity. I have noticed this with every "X produces code faster than C": you begin with two programs that use a suboptimal algorithm, then you take your non-c language and try to write C in it. The result is always awful and removes most reasons not to use C in the first place. This has somewhat changed with rust and in some sense C++,but for other languages my point still stands. They have a nice idiomatic golden path that is fast enough for most cases. Once you need performance badly enough you have to treat you language as an assembler, and then you will always lose to languages that actually are good at that. I say this as a scheme/Haskell weenie. Writing really performance scheme and Haskell code often means writing ugly code.
  - vbezhenar 7y ago
    Kotlin generally is not slower than Java. One exception is that it inserts a lot of null checks. And there are features that will make it even faster than Java, I'm talking about inline lambdas.
nnnmnten 7y ago
Most CS programs have a course on functional programming or programming paradigms including functional programming. I don't understand why it would be a challenge to pick up Clojure.
- Mirioron 7y ago
  I don't know how it is elsewhere, but I saw many (most?) students struggle with those classes. These are people who have learned to successfully program, but they struggled quite significantly with functional languages (Haskell in this case). Even when they pass, many of them probably wouldn't want to pick up a functional language afterwards.
  - danieldk 7y ago
    But Scheme and Common Lisp (and its dialects) are drastically easier to learn than Haskell. Furthermore, I think it also has to do with the order in which these courses are taught. When I did a small introduction to FP at my university, a lot of people who had no prior experience in programming picked up Haskell much more easily than those who already had a background in imperative/OO programming. (Though that strengthens the point - FP is often quite a leap for programmers who have used imperative languages for a long time.)
- tanilama 7y ago
  What learnt in school is insufficient to meet with all real world challenges. To claim as a developer, you need to understand the internals, that is the ability to poke and trace to the root of the problem deep underneath. Debugging in FP feels different to most people, where our thoughts is tuned to the procedural model with observing states and form our hypothesis from there. Another disadvantage with niche programming language is not enough Stack Overflow help there. That alone will drive most people away.
- rienbdj 7y ago
  Many cs courses teach ML style functional programming, not Lisp or Clojure
- 7y ago
TeMPOraL 7y ago
It's a sad state of reality, because it drives down the quality of software to the common denominator of fresh junior devs, which is not much. Another view of this would be: progress in this industry is made by those who succeed while sticking to their guns and not optimizing their infrastructure for the available worker pool.
- JustSomeNobody 7y ago
  So, you’re saying the quality of the code dropped with the move to java? Could there be other reasons? Maybe they rushed?
  - dillonmckay 7y ago
    He is not saying Java caused it, just that the priority was cheaper labor, and a more popular language makes that more likely. This is in the context of software development in general, not this specific project per se.
- pron 7y ago
  > It's a sad state of reality, because it drives down the quality of software to the common denominator of fresh junior devs, which is not much. There's no reality here, just myth. That switching to Java would drive quality up makes at least as much sense as it driving it down. If you're referring to the particular languages, then the reality is that no big effect has been found for the choice of programming language on software quality. As lack of evidence in favor of a big effect is evidence of its lack, the most probable explanation is that no such effect exists (it's possible a small effect exists, but we don't know in which language's favor). And, indeed, while there is a theory explaining (and even predicting ahead of time) why there is no big effect to the choice of the programming language, not only is there no empirical evidence suggesting there is such an effect, there is no theory to predict it, either. That language has a big effect on quality is, at this point, no more than wishful thinking among a minority of developers who are big programming language fans. It's a bedtime story. (I am not saying that language didn't ever have a big effect or that it never will, just that both theory and observation show that there isn't one currently among "reasonable" languages in common use; also, I have used and I like both Java and Clojure) And if you're only referring to the priority of making the project more accessible, then I don't understand your comment at all. There are many, many more experienced Java developers than experienced Clojure developers, and experienced developers are at least as likely not to bother learning a new language in order to contribute to a project (although perhaps for different reasons). Picking a popular language makes the project more accessible to experienced developers. What has been shown to drive quality up or down is process. If you have a good process, then making the project more accessible can help, and if you don't then you're screwed anyway.
Jach 7y ago
At another point you just accept you'll have to hire people who don't fit the classic job requirement of "X years experience in Lang" and hire people who don't know Lang at all but want to learn. A growing Clojure shop in Seattle seems to be having success with this, onboarding the language is a small fraction of overall onboarding. On the more meta level it's weird what people expect devs to learn on the job or know up front. Companies expect employees to keep up with the latest nonsense in web dev, but can't pick up a new straightforward language? Unless maybe another Algol like Go or TypeScript? This Steve Yegge quote captures some of it: "For some reason, programmers love to learn new stuff, as long as it's not syntax."
- amortize 7y ago
  > A growing Clojure shop in Seattle seems to be having success with this, onboarding the language is a small fraction of overall onboarding. I can attest to this fact, as part of a unicorn upstart with dev offices in Sao Paulo and Berlin. Clojure onboarding has never been a major roadblock for new engineers. And we never hire asking for "X years of lang experience". Almost everyone in the (~300 strong) engineering team started with zero-to-little Clojure background.
  - mooreds 7y ago
    What do you look for? I mean the "X years of experience" is a shorthand proxy for "can get stuff done in the language we use". So I imagine you must have some other way of getting an idea of how a candidate can get things done. Would love to hear more about it.
    - amortize 7y ago
      What we have found is that it is enough to verify 'can get stuff done', and leave out 'in the language we use'. So, much of the interview turns out to be a process of validating two things: 1. can do what is claimed in the language of choice. 2. gauge enthusiasm and interest to pick up our language.
      - 7y ago
_cs2017_ 7y ago
How long does it take a strong developer to learn Clojure? Is it really a big a barrier to new contributors? Presumably people who consider contributing to the open source have some spare time and love programming. Wouldn't they find it exciting to contribute while also learning a cool new language?
- skywhopper 7y ago
  Maybe, but learning a new language by contributing to a large project built in that language is asking for trouble. Programming languages aren’t just syntax and stdlib. The relevant idioms and patterns may be significantly different. Even making small changes can be hard and intimidating. And then, even if you wanted to learn something new, to a lot of people, it matters what that something is. Unless the person is really set on contributing to Storm specifically, then the barrier of Clojure is still significant. They may prefer to spend their new language learning bandwidth on something with more momentum.
- yogthos 7y ago
  My experience from hiring and training co-op students for many years is that it takes around 2 weeks to learn with a bit of guidance. when a co-op student with little to know programming experience can learn Clojure in that time I find it surreal that there are professional developers who cannot.
jcadam 7y ago
As a Clojure enthusiast who has never been able to find a FT job using the language, I find it hard to believe that the demand for Clojure devs outstrips the supply. And even if you have to hire devs who don't already know the language I really don't understand the difficulty in learning Lisp - few languages are simpler or easier to grok.
- puredanger 7y ago
  The demand and supply for Clojure both exist, but they are not equally distributed geographically.
  - fnordsensei 7y ago
    Things are progressing, but the cultural addiction to shipping meat around is still too high in my opinion. Geography is a weird and often arbitrary constraint for knowledge work.
- emidln 7y ago
  I didn't have any problems finding work in Clojure when I was last looking (Aug-Sept 2018) with offers in four locales across the US. I was not able to find remote work at comparable salary for a US based org. (Some context) I have 7 years experience using Clojure professionally across 3 orgs (16 years overall) and was looking for individual contributor roles. Some things I observed about the Clojure companies: - Their HR seemed slower than the non-Clojure companies I was dealing with. I don't think this has anything to do with Clojure, but I am no longer working in Clojure full-time because the most credible offers took too long and I've found non-Clojure work I love. - Outside of trading firms, I did the most whiteboard coding questions with Clojure firms compared to shops looking to hire for Python or Elixir. I like whiteboard work, so this was fine for me. - 6 Clojure companies, 4 different conccurrent processing models and 6 different service frameworks. Neither of these is necessarily bad, but the community was much more fractured than I realized. - Of the 6 Clojure orgs, 4 were actively hiring (had open roles) and 2 were passively hiring whenever someone materialized on IRC/Slack/Twitter. This leads me to think if you want a Clojure job, you should probably reach out to people you know inside Clojure orgs and, barring that, the HR people at these orgs directly. This isn't a Clojure-specific thing, but worth keeping in mind. - 4 of 6 didn't seem to have dedicates DevOps, leading to them looking for candidates also familiar with deploying, monitoring, and debugging Linux hosts. I think I was given heavier weight at some orgs because of a background doing this type of work for Clojure stacks. Might be an angle worth looking into if you don't have production Clojure but you do have Linux and Java deployment/monitoring/debugging/tuning experience.
yogthos 7y ago
My team has been using Clojure for close to a decade, and we found the opposite to be the case. While the pool of applicants is smaller, so is the noise ratio. Clojure being niche means that you get people who are willing to look outside the mainstream, and are typically genuinely interested in programming. We also rarely hire devs who already know Clojure, and we typically just train on the job. It's really not hard to do when you already have people who know the language on the team. In fact, if somebody can't learn Clojure, I would question if they can learn to work on a large project in general. Dev practices, architecture, code styles, tooling, and so on tends to change quite a bit from company to company. The language is only small part of that. In case of Storm, Apache commons is run by Java devs who have zero interest in learning Clojure. So, it's not surprising they would rewrite Storm in their preferred language.
- fnordsensei 7y ago
  Same (sure; anecdotal) experience here. Hiring for Clojure has been so much easier for us than hiring for JavaScript. Granted, these are two different products, and maybe the Clojure one is just more attractive to the right kind of person. Nevetheless, in terms of quality vs. quantity of applicants, Clojure has been massively winning.
lvh 7y ago
FWIW, I have a very different read on this, based on both my experience with Clojure and some of the leading Storm-using orgs (and the Apache process). Firstly: Clojure isn't actually a problem. We hired people who are highly junior (one who even refused to call themselves a developer) and taught them Clojure. It was fine. It is true that you need a strong dev lead to guide them through that, but honestly, if you don't have a strong dev lead your Python/Java project is screwed too. Secondly: look at Storm's increased success at Java data processing sweatshop^H^H^H^H^H^Hconsultancies. Of course "can write Java" is what you're going to get if most of your business revolves around the lowest bidder.
Fellshard 7y ago
Same thing I repeat again and again: There is a false cost assigned to learning a language. Developers are too unwilling to even try stepping beyond the boundaries of the first thing they learned. The cost is always lower than they may think, and the benefits far surpassing what they may think. We've got to work at showing developers those benefits early; it's as important to creating software effectively as any other engineer's basic toolkit.
- mgkimsal 7y ago
  > Developers are too unwilling to even try stepping beyond the boundaries of the first thing they learned. The cost is always lower than they may think, and the benefits far surpassing what they may think. and yet... almost every time I've come in to a project where developers got past those boundaries.... they inevitably recreated patterns/idioms from tech 1 in to tech 2, and make tech 2 horrifically bad, and made it even harder for anyone who actually knew tech 2 to come in to the project and help. adding other languages is more than just 'get over it' - without guidance from experienced people who can understand good/efficient ways to model the problem domains unique to a business idiomatically in the new tech, you're very likely adding even more technical debt that you won't even know about for months/years. And... IME, the "let's use new stack X" are moved on to another project or company by the time the technical debt is apparent to everyone.
- Kye 7y ago
  I suspect programming has something similar to Gear/Plugin Acquisition Syndrome in photography and music. It's easy to let tinkering with new toys stand in for doing the actual work of creating. At some point, you have to sit down and do the work with the tools you have if you want to get anything done. If I were to get serious about programming, I would use C# or Go because I found them the most accessible after trying lots of languages. There's nothing wrong with finding a niche.
  - Fellshard 7y ago
    True, but I expect you to know which two or three /distinct/ tools you can work with effectively, and which situations they are most suitable for. Not only that, but knowing multiple languages has a multiplicative effect, massively expanding how you think of programming in all the languages you know, even the stable standbys like C#.
noir_lord 7y ago
> At some point, you just accept that it's more important to get a new warm body than to continue pursuing some idealized programming perfection. I think that is why Kotlin and Typescript are doing so well generally, they build on top of already established bases offering good new things without throwing out everything else. The trade off is they are constrained from how far they can diverge from their roots but on the flip side that is a strength as well. I like both, they feel productive without feeling totally alien.
moocowtruck 7y ago
another interesting part: The new Java-based implementation has improved performance significantly, and made Storm's internal APIs more maintainable and extensible.
hugi 7y ago
> At some point, you just accept that it's more important to get a new warm body than to continue pursuing some idealized programming perfection. Describes my dating life pretty well.
zmmmmm 7y ago
A more glass-half-full view of it is that it's a magnificent success for Clojure. Yes, it got redone in Java, but the fact is that Clojure took it from nothing to one of the world's premier frameworks for distributed computing - something that might have been impossible with the same resources if Java was its starting language. And because of the commonality of the underlying JVM, choosing Clojure allowed the functional approach with far less risk because the transition later was (I am assuming) much less risky and smoother. Looking at it in that light, this would be a huge boon to people considering starting new projects in Clojure (or possibly other JVM languages): it proves if they succeed and get so popular that moving to Java is the right thing, it can be done.
solussd 7y ago
Sometimes that backfires. - Clojure dev

zenlot 7y ago

"While Storm's Clojure implementation served it well for many years, it was often cited as a barrier for entry to new contributors. Storm's codebase is now more accessible to developers who don't want to learn Clojure in order to contribute." - interesting phrasing. Seems that the authors assuming that new contributors are more willing to learn Java instead of Clojure. And those who know Java find Clojure difficult to learn? Clojure, in fact is quite easy to grasp - especially for someone coming from Java, as you would still be working on JVM and always have Java Interop. I guess same thing goes to someone coming from Erlang to Elixir, where you are still working in the familiar ecosystem.

dvfjsdhgfv 7y ago
> Seems that the authors assuming that new contributors are more willing to learn Java instead of Clojure. No, many more people know Java than Clojure, that's it. Whether it's the right decision for the future of the project, time will show, but I don't think it was taken lightly.
twblalock 7y ago
Most new contributors already know Java.
cygned 7y ago
> Clojure, in fact is quite easy to grasp - especially for someone coming from Java I disagree on that point. In my experience as a programming language teacher, people often struggle to learn functional programming and even more to learn a Lisp, especially if they are used to a C-like language. And learning both at the same time is even more difficult. I liked working with Clojure but I have yet to meet someone who easily grasps the language and - more important - development patterns.
- adz5a 7y ago
  Hello, this is an interesting take. Clojure has been a hobby of mine for some time but now I am a professional Clojure developper (clj/cljs). I knew the language before starting my new job but I found it really different than know the "platform": how a project is layed out, tests are written, some libs, how to pinpoint potential technical debt/suboptimal code accurately and so on... But I guess this would have been true if I had transition to any other new language. Would you have any example of what your trainees stumbled upon or found difficult in the language? That would be useful I guess when we try to onboard new people on the project. Have a nice day :)

agumonkey 7y ago

poor lisp, just like reddit ... always dropped for another language.

sitkack 7y ago
What if they are phases of evolution? A small team that can speak a lightweight language can quickly change velocity (speed and direction), iterating until a PoC is a useful system. Once the interfaces and the protocols fall into place, calcify or harden the design by converting to something more static, more literal. Imagine going from something like lisp to Ada? Or lisp to Rust? Write quick prototypes in lisp, solve the hard problems of understanding the domain, then convert portions that require correctness to a language that is amenable to correctness checking (property testing, model checking, etc). I think gradual typing [0] (not Gradual Typing) or language embedding [1] are steps in that direction. In a way, lisp can be thought of as a CASE tool for bootstrapping new systems. The great thing about language platforms (JVM, Beam, Graal, Wasm, Racket) is that many languages can easily exist in the same system so that in many cases, code can be converted on a function by function basis. [0] http://mypy-lang.org http://mypy-lang.org [1] http://terralang.org http://terralang.org
fulafel 7y ago
The SW world is full of Java/C++ rewrites of products that originally started out in a dynamic & productive language like Lisp or Python. It might make sense in the lifecycle phase when you're past making big changes under make-or-break pressure, and are in the "provide predictable 10 year maintainance roadmap for this large codebaes" stage. You end up employing different kinds of programmers too because many of the original team may be reassigned to something more exciting/important to do. But it's in no way inevitable, of course, there are many old Lisp, Python, etc codebases in production too. BTW. I've been trying to rediscover this blog post and can't seem to find it again, anyone? It described different kinds of software projects, the (1) kind that try to deliver incremental improvements (eg 10% cost savings), and the (2) kind that try to make 10x improvements, and the difference in how those projects are run (the 1-type project can't go over budget because it will then deliver negative value etc).
- 7y ago

juskrey 7y ago

Apache products have always been a result of a blind majority rule, and thanks to that many of them are a corporate-style incomprehensible mess. They favor the process instead of a result: Apache things need to show constant progress to be afloat. Indeed, adopting Clojure for the developer product of this style is a dead end. Clojure is a minority rule. A slow tinkerer's and a lazy (in a good sense) practitioner's tool, whose aim is to survive in a real world, not to show a steady movement no matter what.

Shmebulock 7y ago
These Talebian references got me thinking. Couldn't you argue that it shows a certain arrogance and non-results oriented "falling in love with ideals and tools" detachment from reality to use Clojure instead of good ol' Java?
- krn 7y ago
  > non-results oriented I think Clojure's ecosystem is more results-oriented than almost any other currently available[1]. It just requires a completely different mindset to understand, which an average software developer lacks. Therefore, Java is a much better fit for an Apache project. [1] https://news.ycombinator.com/item?id=19728502 https://news.ycombinator.com/item?id=19728502
- jrs95 7y ago
  Oh I'm sure people make that argument all the time. I just don't think it holds up to reality. It's sort of just a dismissal of better options. The reality in my experience is that Java is just a poor choice for a lot of software. That's why you end up with frameworks built upon frameworks & AspectJ to basically make it behave like a dynamic language and try to hide some of the ugliness of the underlying APIs from you. And it's just because it's corporate approved and "everyone knows Java". I don't see how it can reasonably be considered the pragmatic choice. It's just not productive, and the final product isn't something that I would consider easy to understand or maintain most of the time either.
- TeMPOraL 7y ago
  > Couldn't you argue that it shows a certain arrogance and non-results oriented "falling in love with ideals and tools" detachment from reality to use Clojure instead of good ol' Java? Since when trying to do better than using a tool designed for mass-producing software with dumb and cheap labor is "falling in love with ideals and tools"? It's kind of like saying that using an excavator over a bunch of people with shovels is "falling in love with ideals and tools". No, it just allows a single person to get more work done faster and better. Our industry is weird. When recruiting, companies claim they want "the best of the best" and will often test you on ridiculous stuff. But when it comes to actual work, on industry scale, learning, growing professionally, and doing things in a clean and efficient way is frowned upon. Best take the weakest tool available and compensate for its shortcomings with third party services.

ian1321 7y ago

Java is statically typed. Clojure is not. To me, this is the biggest difference and the reason to choose Java.

deleted 7y ago
[deleted]
jnwkw 7y ago
Is this true or false? I don't understand the downvotes.
- Rovanion 7y ago
  While it is true that Java is statically typed and Clojure is dynamically typed it is perhaps one of the less important differences between the two languages. If you in fact would like to add a static type system on top of Clojure there is a way to do that: Typed Clojure. https://typedclojure.org/ https://typedclojure.org/
- pyb 7y ago
  Many people disagree with "the reason to choose Java".
- CodeArtisan 7y ago
  https://danluu.com/empirical-pl/ https://danluu.com/empirical-pl/
- innocentoldguy 7y ago
  I'm not going to downvote the comment, but I think the "statically typed" argument is fantastically overrated. I spent years of my life coding in Java and C# and other years of my life writing Clojure, Python, Ruby, and Elixir code. I've never once thought to myself of the latter, "Damn! This would be so much better if I could have static types." It has literally never, not once, ever been an issue for me.
roshan_naik 7y ago
In the critical path many things can matter that ordinarily don't. When working on perf improvements for Storm 1.0 release, profiling revealed hotspots in Clojure code. Adding type hints to avoid dynamic typing fixed some of these hotspots. In other places, Clojure primitives showed up as a bottleneck. These required rewriting some code in Java and invoking it from Clojure. This was mentioned briefly here: https://hortonworks.com/blog/microbenchmarking-storm-1-0-performance/ https://hortonworks.com/blog/microbenchmarking-storm-1-0-per... The Clojure based core has served the community well for some time. It was still powered by Disruptor (a fast Java messaging library). So there was always some back and forth between Java and Clojure. For Storm 2.0, the Java rewrite happened first. The re-architecture work might have not happened without it (at least for the foreseeable near future)... primarily due to Java expertise being more easily available.

jaimex2 7y ago

If you're going to be on the JVM, stick to what the J stands for :)

lvh 7y ago
The newest JVM out of Oracle, GraalVM, has its polyglossy as its central feature. And Clojure has always been a lot closer to Java (in terms of quality of interop) than even e.g. Scala, let alone something like Jython. (Notable exception JRuby.)

rcaught 7y ago

I struggle to see how you pick Java over Kotlin on the JVM these days. It's everything that Java is, minus everything Java shouldn't be, plus everything that Java should be.

jrs95 7y ago
It's the lowest common denominator. It's the same reason I haven't been able to adopt Kotlin at work. Everyone acknowledges it would be better, but "everyone knows Java". Seems almost like "nobody ever got fired for buying IBM" to me.
- cutler 7y ago
  ... or, worse, "everyone here uses Java <= 8".
ivan_gammel 7y ago
Syntactic sugar does not solve programming problems. Kotlin does not offer any paradigm changes which could make switch worth it.
- deleted 7y ago
  [deleted]
- vbezhenar 7y ago
  The only Kotlin feature that it's really hard to live without is properties. Otherwise sure, it's better than Java 11, but I can live without all those niceties. And (non)nullable types even get in my way! I just don't understand where there's syntax sugar for propeties in Java. There are lot of unneeded stuff added, that I don't really care about. But my classes are still full of autogenerated getters/setters. I recently tried to find some JEP and I did not find one. There was some talk around Java 7, but nothing since that. Nobody even wants to add properties support for Java?
  - sverhagen 7y ago
    And (non)nullable types even get in my way! Interesting how that goes...: the sibling comment to yours said the exact opposite. How does this feature get in your way?

zubairq 7y ago

We did the same with our open source project. We ditched Clojure and Clojurescript for NodeJS and JavaScript, for the same reason as it was a huge barrier to entry for new users

unlogic 7y ago

To me, this makes total sense as the project moved to Apache. Obviously, much more people will be able to consider contributing when it's in Java. Apache goal is sustainability and long-term viability, and Java would work better for that. I also consider this a success story for Clojure. It gives Clojure another usecase: a "production-ready prototype" language where the resulting "prototype" can last for eight years and benefit thousands of developers until it gets rewritten to something else when all the hard questions are answered, and most experimentation/wandering is over.

sitkack 7y ago
See my response inspired by your comment and agumonkey here https://news.ycombinator.com/item?id=20074532 https://news.ycombinator.com/item?id=20074532 I am drawn to the idea of scalable languages or languages with zero friction for interop. This thread on Pallene is interesting https://news.ycombinator.com/item?id=18038619 https://news.ycombinator.com/item?id=18038619
usgroup 7y ago
I think it’s noteworthy that whatever the benefits of clojure it didn’t outweigh the adoption issue. It’s quite a mature language so I’m not sure that bodes we’ll for its prospects this late in the game.
- stingraycharles 7y ago
  What if Clojure is a language that bodes well especially early in a project’s lifecycle?
  - usgroup 7y ago
    I think it’s very difficult to motivate going out of your way to hire decent devs that know clojure and java only to throw away the clojure some time down the road. Begs the question, why not just write it in Java? Then at least its more likely to be a refactor down the road rather than a rewrite. I think clojure ought to be a production ready language that scales well. That’s what it was designed to be. However lisp seems to dichotomise devs into those that get it and those that don’t and thus alienates many would be team members.

hanswesterbeek 7y ago

Clojure just seems "too weird" to too many people, albeit for all the wrong reasons. Its syntax looks like a 6ft hurdle to newcomers, especially if they have lots of experience in C-synthax-languages. But anyone who just /tries/ to jump it finds out the hurdle really isn't that high at all. This dynamic will probably never change. Clojure will remain an acquired taste, which is fine, I guess. Those 'in the know' will happily continue to develop effectively with it.

keymone 7y ago

Bet they had to reimplement half of Clojure when rewriting storm core.

juskrey 7y ago
Yep, at last a lot of progress to report to Apache Board of Directors
eecc 7y ago
I see what you did there

readme3 7y ago

Apart from the language part , i am super excited about trying storm-2.0. In many ways the buzz in the ecosystem has been that storm was dead!. We run lot of prod workloads on storm and spark , storm has worked like a wonderful workhorse without quirks. The ecosystem has moved on , but i think storm may still continue having loyalists!

scribu 7y ago
> The ecosystem has moved on Just ouf of curiosity, what tools are people using instead of Storm?
- tanilama 7y ago
  Flink is a strong competitor. Spark itself also claims to be a runner up with minibatching. But streaming model is more widely adopted outside of frameworks. AWS Lambda or Fass platform can be viewed as streaming model with simple topology.
- nimrody 7y ago
  Flink is popular. I believe Samza also fills a similar niche. There's also Kafka Streams which is suitable for many cases that do not warrant a full cluster with all the management and trouble required.
grkg8tr 7y ago
I thought Heron was the Storm follow-up. I'm surprised to see a Storm 2.0.

fulafel 7y ago

cutting off 1.0 immediately sounds quite harsh, I wonder what kind of user base it has? "EOL for 1.0.x With the release of 2.0.0 the 1.0.x version line will no longer be maintained."

Hendrikto 7y ago
> It is designed to push boundaries on throughput, latency and energy consumption while maintaining backward compatibility.
JanecekPetr 7y ago
That's because they'll be maintaining the 1.1.x and 1.2.x branches...
kabhwan 7y ago
Many big projects tend to manage around 3 version lines. Since Storm 2.0 is out, there're three version lines for 2.x, 1.2.x, 1.1.x. 1.0.x could be EOL-ed.

mrkeen 7y ago

In this thread: * Non-Java JVM languages are too different for people to use. Also: * It's not worth switching from Java to other languages because they're just Java with syntactic sugar.

skywhopper 7y ago
Clojure and Kotlin are two very different languages. I don’t see anyone making the broad statements you are calling out.
- mrkeen 7y ago
  No, of course not. I misread > Syntactic sugar does not solve programming problems. Kotlin does not offer any paradigm changes which could make switch worth it. and > Also, java 8 and above is getting a lot leaner to use, api, linguistics .. so the gap with clojure is a bit less than before.

gigatexal 7y ago

How am I just learning about this? This seems to be rather novel. Real-time bolt on for any database? Pretty cool stuff.

grego 7y ago

In the old days a project would often start in Lisp (Common Lisp for example), and then later be rewritten in C. This is the same pattern, and one should embrace it. Write the first cut, explore the problem space and prototype in a dynamic language. Once the design is proven, rewrite in statically typed language for performance and to get the possible remaining bugs out. If the design was perfect from the get go, you could just write it in assembler to start with.

amortize 7y ago

> While Storm's Clojure implementation served it well for many years, it was often cited as a barrier for entry to new contributors. Storm's codebase is now more accessible to developers who don't want to learn Clojure in order to contribute. It is interesting to contrast this with state of affairs in Apache Spark. Spark has thrived well in spite of being a Scala project; Scala arguably has a higher barrier for entry compared to Clojure (although the flavour of Scala used within Spark closely resembles Java).

_cs2017_ 7y ago

Why change the title of the post? Almost every one of the ~100 comments is in response to the original title that Storm switched from Clojure to Java. Now this context is lost.

lvh 7y ago
Because the previous title was editorialized? Furthermore, perhaps the conversation is about the old title because it was, and now the conversation can go be about Storm itself :)
- _cs2017_ 7y ago
  I think the rule about not editorializing needs an exception so that someone can point out an important or interesting fact using an article whose original title is too generic. Another example would be "Press release from XYZ". If it talks about XYZ reaching some major breakthrough, the title should be allowed to reflect that. It's a waste of everyone's time to post the article titled "Press release". And arguably this situation should fall under such an exception.

jacobtracey 7y ago

What exactly are you doing?

ww520 7y ago

While most comments touch on the developer issue, the significant performance gain deserves more discussion. I guess Java's static type, pragmatic mutability with less garbage produced, and efficient parallelism give substantial boost to performance. It would be interesting to see what the upcoming blogs have to say about the performance gain.

yogthos 7y ago
My guess would be that it's always easier to make a rewrite performant because you already have a working version and know exactly where bottlenecks are. I imagine rewriting it in Clojure, or any other language, would've resulted in a significant performance boost as well.

grkg8tr 7y ago

How does this relate to Apache Heron? I thought Heron was essentially Storm 2.0. Which one should I be using?

roshan_naik 7y ago
Not well known is that Heron's performance claims in the "flying faster" blog and SIGMOD paper were made by comparing against a fork of an older (pre apache) version of Storm even though much more performant version was already out. IMO (biased?) Heron makes surprisingly poor design choices for their key distinguishing features like threading model and backpressure. And it shows when you benchmark it. More specifics: https://github.com/apache/storm/pull/2241#issuecomment-317879907 https://github.com/apache/storm/pull/2241#issuecomment-31787...
kabhwan 7y ago
Given there're so many streaming frameworks with different characteristics, you should be the one who understands the difference and select one fits for your business use case. There's no silver bullet.

didibus 7y ago

Can someone enlighten me on the inner working of Apache? Do they hire developers to maintain and work on the projects? If not, is there community leads in charge of each one? How are they appointed?

geodel 7y ago
http://www.apache.org/foundation/governance/ http://www.apache.org/foundation/governance/
- didibus 7y ago
  Interesting, where would you go to see the list of committers for a given project and find information about them? I'm curious if the commiters are Clojure developers who chose to switch to Java, or if they are Java devs who decided to rewrite the project to a language they are more familiar and comfortable in. Or if they are Clojure devs looking to transition the project to other commiters and they struggled to find other Clojure devs willing to take it over.

snorremd 7y ago

> While Storm's Clojure implementation served it well for many years, it was often cited as a barrier for entry to new contributors. Storm's codebase is now more accessible to developers who don't want to learn Clojure in order to contribute. While I appreciate that programmers have limited free time and may not want to learn a new language to contribute to open source, I feel like learning Clojure could be of value to the programmer in itself. The supposition that Clojure acts as a big barrier against contribution from developers might be true, but I really feel like developers who only write code in the ALGOL family of languages are missing out. Learning a lisp language is not an impossible task and can improve your skills as a developer.

didibus 7y ago

It is interesting to read the history of Storm from the original author (Nathan Marz) himself: http://nathanmarz.com/blog/history-of-apache-storm-and-lessons-learned.html http://nathanmarz.com/blog/history-of-apache-storm-and-lesso... Clojure allowed him to build the initial Storm release by himself in only 5 months. > I made all of Storm's APIs in Java, but implemented Storm in Clojure. By keeping Storm's APIs 100% Java, Storm was ensured to have a very large amount of potential users. By doing the implementation in Clojure, I was able to be a lot more productive and get the project working sooner. Also interesting is that Nathan has now founded a startup working on something which sounds like the next evolution of Storm, and he's building it in Clojure again: https://redplanetlabs.com/ https://redplanetlabs.com/ They recently got 5M in funding and are hiring. Building a Clojure team for it. Personally, I feel the Storm 2.0 release rewrite to Java is really just a case of the new maintainers not knowing Clojure very well, and being primarily Java devs, looking for contributions from people using Storm, which tend itself to be mostly Java shops. Clojure is not quick to pick up, and very different from Java, so unlike Scala, Kotlin, C#, Go, etc. Someone with a Java background and no Clojure or Lisp experience won't be able to contribute as easily. And that's fine, and possibly best for Storm's future now that Nathan has moved on. Unless a Clojure dev had stepped up and adopted Storm, this change isn't at all surprising.

sandra101 7y ago

My ex ruined my credit due to his incessant extravagant spending spree, I found myself in a big mess. I talked to a credit repair company and I was told that it would take me non less than a year to fix my credit. I was devastated, that's a very long time which I can't cope with. I looked online and came across Neo's contact, took the bold step of messaging him even though I was skeptical about the whole thing and to my greatest surprise, my credit was repaired 3 working days, my credit was restored from 337 -710. Tbh I was shocked and it didn't cost me too much. I advise you to contact him on (neogonzalezhacks at g mail dot com )for all credit issues, he replies quickly and you can text him on +1 773 688 9662 .

mikewills23 7y ago

be careful of whom you contact to help you, he is a a special hack which have is own ways of hacking that nobody has ever imagine in this hacking world with his spare idea he can hack anything. he is totally secured and your security comes first. Hacking a website is a job for pro like (neogonzalezhacks@gmail.com) or text him +1 773 688 9662 - ANY TYPE LOANS -Database hack. Remove criminal records. facebook hack. -gmail hack -whatsapp hack -tracking calls -online hacking lectures -Cloning of phones -increasing credit cards scores -Twitter hack EMAIL : NEOGONZALEZHACKS@GMAIL.COM

fulafel 7y ago

For anyone interested in the Clojure history, there's this interview of Nathan Marz, originalcreator of Storm, from 5 years ago: https://www.infoq.com/interviews/marz-lambda-architecture/ https://www.infoq.com/interviews/marz-lambda-architecture/

kabhwan 7y ago

DISCLAIMER: I'm one of PMC member of Apache Storm. Even though Clojure has been Nathan's first-class language, not everyone in contributors and even committers couldn't become native on Clojure. (Though I guess part of committers could feel Clojure as native.) It might not be a problem when Nathan builds most of things in Storm (I'll link his blog post which describes lesson learned from Storm on his perspective http://nathanmarz.com/blog/history-of-apache-storm-and-lessons-learned.html http://nathanmarz.com/blog/history-of-apache-storm-and-lesso...), but when Storm becomes popular and also no longer his toy project, Clojure matters for scalibility of the project. Personally, I had a hard time understanding plenty of macros and even chained macros - that might not be a problem if I were allowed to have plenty of times to study Clojure and get used to it, but possible contributors don't get a chance when they're ready. In many cases, the first time they read the code on what they're using is when they encounter issues on stage/production and have to dig with stack trace. For me, I had to investigate on Storm because my team suffered from lack of documentation and also lack of knowledge on internal which ended up just taking workarounds. It was pretty hard time learning language to just read and understand the details to track down the issue. I even couldn't think of writing code on Clojure. When I decided to contribute Apache Storm, I started from connectors and CI build which wasn't written on Clojure. After some months I was able to read and understand the code to get into details but I still couldn't be native for Clojure, so still required N times of efforts to read and write the code even after I became one of PMC member. Don't get me wrong. I'm not saying something is better than some other one. In scalability point of view on project, I guess we had to do this, even though it ended up teaching me some lesson that migrating language is not the thing which can be considered easily, especially for a big project. Lots of efforts were spent here. Someone could claim for Apache Spark's case for Scala. Yes, I'm also contributing to Apache Spark and I had to learn Scala (fortunately I had a chance to do that before contributing) but the case is different. If you read Databricks Scala guide in details, you might realize that the doc is actually not encouraging contributors to use advanced features (a.k.a the features only Scala geeks might feel natural) on Scala. Same goes on reading codebase. Apache Spark community has been trying to keep necessary knowledges on Scala to contribute Apache Spark low enough, so that non-expert of Scala engineer could easily try out contribution. If they have been requiring hard understanding of Scala to accept contributions - Spark may not be able to get its amazing reputation.