5 ms·

Question - there was a recent post about how the dailymail group moved 140k lines of java code to 4k lines in clojure. Since I have never programmed much in Ja

Question - there was a recent post about how the dailymail group moved 140k lines of java code to 4k lines in clojure. Since I have never programmed much in Java (I went from c++ to ruby!), could someone tell me where the gains are coming from ? I know that eliminating the setters and getters (and the private/protected/etc system) would eliminate a lot of code, but I dont understand if it would be that much. Possibly, I'm also trying to understand if they would have gotten similar improvements in kloc if they had switched to.. ruby/python for instance.

deleted 13y ago
[deleted]
PurplePanda 13y ago
Some reduction in lines of code may have come if they had rewritten it in Java too, due to better understanding the problem space after having written the first version. Java has a fair amount of boilerplate type code other than getters and setters. For example forcing you to use "classes" for everything, even when you really want a procedural or functional API. Some people try to write semi functional style programs anyway using object oriented "patterns", which can sometimes lead to really just a lof more lines of code too. Also explicit type annotations take up space. And the Java style of using exceptions can be quite verbose. There may also be better designed libraries to do a lot of things for you automatically in clojure. Macros are very helpful too. But I'm just making all of this up, haven't actually seen the codebases to compare them.
cliftonc 13y ago
Hi, I'm the CTO at the MailOnline so can comment. The biggest reduction in loc was due to the move from a relational DB to elastic search containing flatter representations of our content. This basically meant that we could delete thousands of lines of boilerplate and create a very simple processing pipeline to render content, and for this clojure really shines of course. The REPL made the first week of exploring the problem and solution space really effective, and even as a node guy I was impressed with what could be done. I can say that there is definitely a big productivity boost, but as others have said it isn't purely the language, but rather thinking about the problem in a completely new way that really did it. I do credit clojure though with giving our engineers the ability to think differently, as that is often the hardest part. There was a comment I made in a speech recently about functional programming being frictionless for content management system design, this is what I was alluding to. I don't want to start a flame war, but we built an earlier version in ruby (with one of the best OO guys I know leading it), and it was much more complex through attempting to model our home page as a set of inter related objects, using visitor patterns to traverse etc., than the clojure version.
- sandGorgon 13y ago
  very interesting. I had a similar experience with elasticsearch in modelling a ecommerce catalog - however that was with jruby. In fact, it took me quite some convincing to not go with something like mongodb - elasticsearch served us quite well. Could you comment a bit more on the interaction of functional programming to content management ? Intuitively, it feels to me that a lot of it has to do with the NOSQL-like data model of elasticsearch. Being a lisp beginner, I feel that a visitor pattern is very analogous to "map" in functional programming - again, without all the OO getter/setter functions (e.g. "accept", "visit").
  - cliftonc 13y ago
    A lot of the simplification was definitely down to flattening the structure that was represented in 20-30 tables down into a smaller set of documents in ES. But the functional approach to then rendering these documents simplified it further. A functional map is definitely analogous to a visitor pattern, but don't underestimate how much of a productivity boost arrives by being able to see all of the code in a small set of terse statements in one place, vs having a set of fairly abstract classes with accept/visit scattered all over the place. We just found that the OO solution resulted in more code and it was more difficult to hold the resulting model in your head as the number of things that could be visited grew - we run a very big site. The other big enabler vs a traditional CMS approach is that we have a very clear separation between the part of our stack that allows editors to create content, and the part that renders it (connected via messaging) - so we aren't mixing r/w domains and our rendering pipeline isn't burdened with the additional complexity of having to understand how content changes, it just renders what it is given as fast as it can.
T-R 13y ago
Java (at least prior to Java 7), has some severe limitations in how code can be abstracted. As an example, it doesn't allow you to pass functions as arguments to other functions directly. As a result, the canonical way to handle situations where you would do this (such as passing a comparison operator to a sort function), is to declare an interface, define the function in a class that implements that interface (command pattern, etc.), and then instantiate an instance of the class and pass that. Closures are imitated by explicitly passing data to class constructors and storing it in member variables, and currying imitated through patterns like the onerous abstract factory pattern. Additionally, the abstraction features Java does offer are implemented with some heavy language syntax, and often can't be generated at runtime or pre-compile time, at least not easily, and so there's a limit to the abstraction, even taking into account Java Generics. That's something where homoiconic languages like Clojure have a particular advantage. As a result, abstracting things in these ways (which in other languages is incredibly common) gets to be painfully verbose. Often, things that would be beautifully abstracted in these ways are just left unabstracted (as in-line boilerplate) - the verbosity introduced makes the code less readable, rather than more, and so there's a stigma against "over-abstracted" code. The result is that code that does, or could, make use of these things takes exceedingly more code in Java than in a language like Clojure. Of course, conventions in Java of, e.g., giving each curly brace its own line aren't helping its case in line counts, either.
- thaumasiotes 13y ago
  > Of course, conventions in Java of, e.g., giving each curly brace its own line aren't helping its case in line counts, either. My comment is basically a free-association from this. In C and Java both (and, um, CSS), the people I've encountered seem to strongly prefer giving each matched pair of curly braces one line (that is, the opening { doesn't get its own line, but the closing } does). In lisps (to my knowledge) it's more normal to have all your closing )s stuck together at the end of the previous line. Using a clump of )))))s to discern exactly which scope you're in afterwards is tricky (I'm pretty sure that's exactly the "problem" that the C/Java style is trying to solve), so you look at the indentation instead. Python formalized the indentation-is-scope paradigm, which means a malicious programmer can't trick you into believing you're looking at a different scope than you are, but my Python code always ends up being a lot taller than I feel it should be, so I have a lingering sense that Python took it a little too far. It turns out that I occasionally appreciate the option you get with lisps to put a short bit of code all on one line and mark the parse tree with parens instead of with line-breaks-plus-indentation. (All that said, I do like Python a lot.)
d0m 13y ago
I think it's also in the mentality as in it's perfectly fine in lisp to use simple lists as data structure, whereas we mostly use wrapper objects around primitive or collections in Java. For instance, you'll often see a "Person" class definition with getters, setters around the .adds collection function. The idea is to encapsulate the collections as much as possible so that if it changes down the road, nothing else is affected but that simple method. In lisp, however, you'd probably just (let (people '("tom" "john"))) and be done with it. I'm obviously exaggerating, but hopefully you get my point. It's similar in Python where I use dicts all the time, but feel bad using it in Java or C++ without abstracting or encapsulating it. It's not so much a syntax issue (even though clojure syntax is less verbose than Java) but mostly the mentality behind the idioms IMHO.
kazagistar 13y ago
Its very possible they would have gotten significant (though possibly not equivalent) reduction if they reprogrammed it directly in Java. An old codebase often has a lot of useless cruft and over-abstractions and so on, and you do better each time you code something. The experimental control is not the old code base, it is translating the same code base back to itself with similar engineering effort. No wonder few people actually try to gather that sort of data.
benjaminwootton 13y ago
There is a lot of ceremony in Java. If I want a single polymorphic function I'll create an interface and a bunch of classes to implement it, instantiate those classes and pass them to where they need to be. In a functional program, I'll just create the function and pass it around anonymously if needed. The function will in itself will likely be simpler than the Java equivalent and stateless. There will also be less DTO type class code for carrying around dumb collections of data, replaced with simple records and lists. I'm not at the stage where functional programming has really clicked for me, but after writing some Clojure and Haskell recently, the sheer amount of code you have to type in in Java to express yourself is really starting to bug me!

deleted 13y ago

[deleted]

PurplePanda 13y ago

Some reduction in lines of code may have come if they had rewritten it in Java too, due to better understanding the problem space after having written the first version. Java has a fair amount of boilerplate type code other than getters and setters. For example forcing you to use "classes" for everything, even when you really want a procedural or functional API. Some people try to write semi functional style programs anyway using object oriented "patterns", which can sometimes lead to really just a lof more lines of code too. Also explicit type annotations take up space. And the Java style of using exceptions can be quite verbose. There may also be better designed libraries to do a lot of things for you automatically in clojure. Macros are very helpful too. But I'm just making all of this up, haven't actually seen the codebases to compare them.

cliftonc 13y ago

Hi, I'm the CTO at the MailOnline so can comment. The biggest reduction in loc was due to the move from a relational DB to elastic search containing flatter representations of our content. This basically meant that we could delete thousands of lines of boilerplate and create a very simple processing pipeline to render content, and for this clojure really shines of course. The REPL made the first week of exploring the problem and solution space really effective, and even as a node guy I was impressed with what could be done. I can say that there is definitely a big productivity boost, but as others have said it isn't purely the language, but rather thinking about the problem in a completely new way that really did it. I do credit clojure though with giving our engineers the ability to think differently, as that is often the hardest part. There was a comment I made in a speech recently about functional programming being frictionless for content management system design, this is what I was alluding to. I don't want to start a flame war, but we built an earlier version in ruby (with one of the best OO guys I know leading it), and it was much more complex through attempting to model our home page as a set of inter related objects, using visitor patterns to traverse etc., than the clojure version.

sandGorgon 13y ago
very interesting. I had a similar experience with elasticsearch in modelling a ecommerce catalog - however that was with jruby. In fact, it took me quite some convincing to not go with something like mongodb - elasticsearch served us quite well. Could you comment a bit more on the interaction of functional programming to content management ? Intuitively, it feels to me that a lot of it has to do with the NOSQL-like data model of elasticsearch. Being a lisp beginner, I feel that a visitor pattern is very analogous to "map" in functional programming - again, without all the OO getter/setter functions (e.g. "accept", "visit").
- cliftonc 13y ago
  A lot of the simplification was definitely down to flattening the structure that was represented in 20-30 tables down into a smaller set of documents in ES. But the functional approach to then rendering these documents simplified it further. A functional map is definitely analogous to a visitor pattern, but don't underestimate how much of a productivity boost arrives by being able to see all of the code in a small set of terse statements in one place, vs having a set of fairly abstract classes with accept/visit scattered all over the place. We just found that the OO solution resulted in more code and it was more difficult to hold the resulting model in your head as the number of things that could be visited grew - we run a very big site. The other big enabler vs a traditional CMS approach is that we have a very clear separation between the part of our stack that allows editors to create content, and the part that renders it (connected via messaging) - so we aren't mixing r/w domains and our rendering pipeline isn't burdened with the additional complexity of having to understand how content changes, it just renders what it is given as fast as it can.

T-R 13y ago

Java (at least prior to Java 7), has some severe limitations in how code can be abstracted. As an example, it doesn't allow you to pass functions as arguments to other functions directly. As a result, the canonical way to handle situations where you would do this (such as passing a comparison operator to a sort function), is to declare an interface, define the function in a class that implements that interface (command pattern, etc.), and then instantiate an instance of the class and pass that. Closures are imitated by explicitly passing data to class constructors and storing it in member variables, and currying imitated through patterns like the onerous abstract factory pattern. Additionally, the abstraction features Java does offer are implemented with some heavy language syntax, and often can't be generated at runtime or pre-compile time, at least not easily, and so there's a limit to the abstraction, even taking into account Java Generics. That's something where homoiconic languages like Clojure have a particular advantage. As a result, abstracting things in these ways (which in other languages is incredibly common) gets to be painfully verbose. Often, things that would be beautifully abstracted in these ways are just left unabstracted (as in-line boilerplate) - the verbosity introduced makes the code less readable, rather than more, and so there's a stigma against "over-abstracted" code. The result is that code that does, or could, make use of these things takes exceedingly more code in Java than in a language like Clojure. Of course, conventions in Java of, e.g., giving each curly brace its own line aren't helping its case in line counts, either.

thaumasiotes 13y ago
> Of course, conventions in Java of, e.g., giving each curly brace its own line aren't helping its case in line counts, either. My comment is basically a free-association from this. In C and Java both (and, um, CSS), the people I've encountered seem to strongly prefer giving each matched pair of curly braces one line (that is, the opening { doesn't get its own line, but the closing } does). In lisps (to my knowledge) it's more normal to have all your closing )s stuck together at the end of the previous line. Using a clump of )))))s to discern exactly which scope you're in afterwards is tricky (I'm pretty sure that's exactly the "problem" that the C/Java style is trying to solve), so you look at the indentation instead. Python formalized the indentation-is-scope paradigm, which means a malicious programmer can't trick you into believing you're looking at a different scope than you are, but my Python code always ends up being a lot taller than I feel it should be, so I have a lingering sense that Python took it a little too far. It turns out that I occasionally appreciate the option you get with lisps to put a short bit of code all on one line and mark the parse tree with parens instead of with line-breaks-plus-indentation. (All that said, I do like Python a lot.)

d0m 13y ago

I think it's also in the mentality as in it's perfectly fine in lisp to use simple lists as data structure, whereas we mostly use wrapper objects around primitive or collections in Java. For instance, you'll often see a "Person" class definition with getters, setters around the .adds collection function. The idea is to encapsulate the collections as much as possible so that if it changes down the road, nothing else is affected but that simple method. In lisp, however, you'd probably just (let (people '("tom" "john"))) and be done with it. I'm obviously exaggerating, but hopefully you get my point. It's similar in Python where I use dicts all the time, but feel bad using it in Java or C++ without abstracting or encapsulating it. It's not so much a syntax issue (even though clojure syntax is less verbose than Java) but mostly the mentality behind the idioms IMHO.

kazagistar 13y ago

Its very possible they would have gotten significant (though possibly not equivalent) reduction if they reprogrammed it directly in Java. An old codebase often has a lot of useless cruft and over-abstractions and so on, and you do better each time you code something. The experimental control is not the old code base, it is translating the same code base back to itself with similar engineering effort. No wonder few people actually try to gather that sort of data.

benjaminwootton 13y ago

There is a lot of ceremony in Java. If I want a single polymorphic function I'll create an interface and a bunch of classes to implement it, instantiate those classes and pass them to where they need to be. In a functional program, I'll just create the function and pass it around anonymously if needed. The function will in itself will likely be simpler than the Java equivalent and stateless. There will also be less DTO type class code for carrying around dumb collections of data, replaced with simple records and lists. I'm not at the stage where functional programming has really clicked for me, but after writing some Clojure and Haskell recently, the sheer amount of code you have to type in in Java to express yourself is really starting to bug me!