7 ms·
Clojure vs Scala - Anecdote
- abscondment 15y agoIt's interesting that rewriting in Clojure was "less work and more timely" than migrating to Akka or waiting for Scala 2.10. Any insight as to why? Having written no Scala, I'm not sure what the Akka changes would entail.
- jzoidberg 15y agoAkka actors are the accepted production solution in Scala. I wonder how the author translated actor heavy code to clojure - I guess the closest available concept would be agents.
- seancorfield 15y agoThe original Scala code was written back in November 2009. Akka wasn't really the "accepted production solution" back then. There's no question that it is _today_. We used actors to create a pool of workers in order to run some slow processes in parallel. The simplest equivalent in Clojure was to run the processes via pmap. Cheap, quick and very effective. We didn't have as much control over the parallelism but it didn't really matter for our use case. We had about 1,000 lines of Scala that we'd developed as we learned Scala. Today we have more experience in Clojure than we did in Scala back then. Writing 260 lines of Clojure with the Scala code as a guide took a couple of days, including writing a new set of unit tests. Learning enough about Akka to migrate to it would have taken us some time - and would still have left us with three languages in the mix. By replacing this Scala code with Clojure, we only had two languages in the mix, which was also a win.
- Uchikoma 15y ago[edit: See my comment below on how this might be a culture thing] I wonder how 15 hours in the post become a couple of days here on HN. But of course, the post was written as an anecdote and should be taken as that. Could you elaborate what makes Akka "a language" for you? I did assume Akka actors are for all practical reasons a drop in replacement for scala actors. More people should btw. read "Anecdotal Evidence and Other Fairy Tales" in Hacknots book: http://www.scribd.com/doc/459372/hacknot-book-a4 http://www.scribd.com/doc/459372/hacknot-book-a4
- seancorfield 15y ago15 hours = 2 x 7.5 hours - 2 days? (seems like basic math to me :) Akka is _not_ a language - I'm not sure where you got that impression? (please explain so I can respond to it - no judgement). Based on the Akka documentation (pre-1.0), it did not seem to be completely a "drop in replacement for scala actors" but I hope it's moving that way (and I'm very pleased that Scala is adopting Akka as the default implementation!). Heh, yes, anecdotes are meant to be just that....
- deleted 15y ago[deleted]
- Uchikoma 15y agoAh, then the reason of our misunderstanding would be my insufficient knowledge of English. In German I would have said "two days" if it was 2 days and use a "a couple of days" for - lets say 4 or 5. Interesting how culture differs. Sorry, I thought you were using Scala and Java and had to chose if you replace Scala with Clojure (Java + Clojure) or add Akka to Scala (Java + Scala). So I've assumed you counted Akka as a language (Java + Scala + Akka) as you've mentioned 3 languages. I probably didn't read your comments correctly.
- lightcatcher 15y agoI also thought you were considering Akka as a language. The sentence where you mention "leaving 3 languages in the mix" gave me this impression.
- seancorfield 15y agoAh, I see. No, we have another language in the mix which is not relevant to this discussion. The "One True Language" comment was a bit tongue in cheek since I doubt we'd really end up with just a single language across all tiers. I had initially hoped Scala would become our primary backend language but it looks like that "honor" will go to Clojure as we're doing more and more work with that.
- charliegriefer 15y ago
- sreque 15y agoI'm surprised that you chose to use actors to parallelize your code. I always thought they were more valuable for concurrency. Back before Scala 2.8 came out with parallel collections, when I need parallelism I wrote my own pmap-like functions. It was really easy, probably easier than learning the whole concept of actors, let alone an implementation of them, and I got the performance speedup I was looking for. It sounds to me like you got a large code reduction because in Scala you used a heavyweight concurrency solution to a lightweight parallelism problem.
- seancorfield 15y agoI just went back and looked at the code overhead of using actors over what it might look like with 2.9's parallel collections and the difference would be about 25-30 lines so I don't think that counts as a potentially "large code reduction". Wrapping three bits of code in actors doesn't add much ceremony (which is in fact a great example of how concise Scala can be when adding concurrency!). (And parallel collections came in 2.9, not 2.8 - we'd already migrated, painfully, from 2.7 to 2.8)
- snprbob86 15y agoI'm not so sure this is a Clojure vs Scala story, as much as it is a Write One To Throw Away story. "The form of the Clojure code mostly follows the form of the Scala code, most of the same functions" Well evidently not. Otherwise, the code would not have seen a 75% reduction in lines of code. What specific transformations resulted in this delta?
- rjd 15y agoI was a little disappointed in the point where he said he rewrote it in 15 hours, and was 75% lighter code... and didn't explain why. Makes me think is think a few things... - the original code was bloated and needed a rewrite anyway, and hence was performing badly. Most importantly having written lots of code that has evolved myself over the years... I have taken my own code (written in the same language) and cleaned them unto be half the size of they where before hand. - The closure rewrite was incomplete as compared to the scala version and totally invalidates his comparison - would the out of memory issue and performance difference be solved by not using actors with known memory leaks and swapping to akka. Would love a follow up on this as I'm slowly learning scala as a hobby ATM
- seancorfield 15y agoIt's Clojure, not "closure". Yes, switching to Akka would almost certainly have solved the OoM problems. That's the whole point. We had to switch from the original code. The choices were: move to Akka, wait for 2.10 (when Akka replaces the default actors implementation - and whatever migration was involved), or rewrite. Given that we're getting heavily into Clojure, it seemed a worthwhile spike to create a Clojure version to see how it performed... Was the original code bloated? I don't think so. I've written a LOT of code over 30+ years in dozens of languages. When the Scala code was written two years ago I was very impressed with its conciseness compared to Java (and other languages). My experience has been that Scala is 3-4x more concise than Java in general (and sometimes as much as 10x more concise) but Clojure is even more concise. If you're learning Scala, don't let this put you off: Scala is an incredibly impressive language. Odersky and his team have done an amazing job.
- Uchikoma 15y agoExcellent, the title says anecdote to set the right context. There have been other discussions on how much the knowledge of writing one solution helps in writing another solution in a different languages and what is attributable to the new language itself.
- Uchikoma 15y agoWanted to add one link to this where I did get the idea from the first time: http://www.oreillynet.com/ruby/blog/2007/09/7_reasons_i_switched_back_to_p_1.html http://www.oreillynet.com/ruby/blog/2007/09/7_reasons_i_swit... "#7 - PROGRAMMING LANGUAGES ARE LIKE GIRLFRIENDS: THE NEW ONE IS BETTER BECAUSE YOU ARE BETTER"
- Uchikoma 15y agoIt takes a little away from the story that it took them a lot of time to discover a widely published performance and memory problem with scalas actors. "after a lot of poking around, we became fairly convinced it was due (at least in part) to the default actor implementation in Scala"
- seancorfield 15y ago"widely published performance and memory problems with scala actors" - remember the code dates back to November 2009 when that was not common knowledge...
- Uchikoma 15y agoThe post was probably a little unclear, I did not understand you poked around to find the oOms in 2009, I just assumed it was more recent as the Clojure post was from September 2011.
- seancorfield 15y agoNo, we didn't hit the OoM regularly until we had more traffic as we moved more sites onto the new system and it wasn't clear that there was an admitted problem with the Scala actors at first either. These days folks seem much more comfortable about admitting there's a problem with the built-in actor implementation. Is that clear enough for you?
- DannoHung 15y agoGuess you weren't really using the type system if you were able to translate your Scala code directly to Clojure.
- seancorfield 15y agoNot sure what you mean - if code is statically type safe, it will also be dynamically type safe. Moving from Scala to Clojure is not really affected by the type system.
- vrode 15y agoPeople and their one-liners.
- rbranson 15y agoA story presenting objective truth through measured experimentation and empirical data.
- deleted 15y ago[deleted]
- schiptsov 15y agoNumbers of lines of code and improved readability alone are worth switching. As a consequence, less code means less consumption of resources, for Java it is very true.)) Just by looking at the code one could see that Clojure makes the development process for JVM less annoying and frustrating while Scala makes it even more verbose and complicated. ^_^
- ambrosebs 15y agoLisps make it fairly difficult to reason about resource consumption based on code size. And in general, is not a good metric for such a measurement.
- foobarbazetc 15y agoWithout seeing the code for both implementations, this posting means absolutely nothing. I wish I could unread these types of submissions.