10 ms·
Revisiting Prechelt’s paper comparing Java, Lisp, C/C++ and scripting languages
- lisper 5y agoI am Ron Garret (f.k.a. Erann Gat), the author of the followup study mentioned in TFA. I am, frankly, amazed that this work is getting the amount of attention that it is over 20 years after it was published. I have a couple of comments about this latest critique. > Given that the participants were recruited from internet groups, I doubt that the subjects were monitored in place, which makes me think that the Development time data collected should be taken with a giant grain of salt. > Based on a small study like Garret’s, I find that the premise that Lisp is so great is unwarranted. That's a fair criticism. In my defense I will say that I didn't actually claim that Lisp is great, only that it seems plausible that it might offer the development speed and runtime safety advantages of Java and the execution speed of C, and so it is worth paying some attention to it, and maybe doing further studies. I certainly believed (and continue to believe) that Lisp is great, but I certainly concede that my data didn't (and doesn't) prove it. Renato seems to agree: > However, the data does suggest that Lisp is about as productive as the scripting languages of the day and has a speed comparable to C and C++, which is indeed a remarkable achievement. FWIW, my study was a throwaway side-project conducted on the spur of the moment in my spare time. It was intended more to get people's attention and make the case that Lisp ought to be considered alongside Java as an alternative to C than to be a rigorous scientific case for Lisp's superiority over Java. At the time, Lisp was quite a bit more mature than Java, and indeed one could have argued (and indeed could argue today) that Java was an inferior re-invention of many of the ideas that Lisp had pioneered, with automatic memory management being at the top of that list. I was frustrated that Java was getting all the attention because it was the shiny new thing and Lisp was being ignored because at the time Java actually sucked pretty badly. (IMHO it sucks pretty badly today too, but that is now a much harder case to make.) In any case, what astonishes me most is that AFAICT no one has ever done a follow-up to either of these studies that was better designed and used more rigorous methods. Instead, people just seem to keep citing these papers again and again as if squeezing these turnips harder might yield something worthwhile now that could not be extracted from them when they were first published. This is a very fertile field for further research. Someone should cultivate it.
- 37ef_ced3 5y agoWhat do you think of Go (particularly as an alternative to C and C++)?
- exdsq 5y agoOn a similar note, what about Rust?
- lisper 5y agoI am not very familiar with either one (I've read about them both but never written any code) so I can't really answer that. Same for C#, F#, Swift, in fact, pretty much anything that has come out in the last 10 years. Old dog, new tricks, something something... I will say, however, that IMHO just about anything is better than C++, including C.
- Gibbon1 5y agoA random anecdote back in the early 90's I worked with a mad lad self taught non-degreed engineer. He did an embedded project where faced with memory constraints based on unit cost. Ported a lisp interpreter to PIC assembly[1] and then write most of the code in lisp. Performance was about 1/2 to 1/5 that of straight C[2]. But the memory footprint was about 50% less. Which meant it fit into flash. He went on to use that for a few other products. [1] Took him three months to port it. [2] A 50% difference in speed usually doen't cause firmware to fail performance metrics.
- jhgb 5y agoSounds like there's a decent chance that one could achieve similar things with something like Modula's M-code. Also compressing a large program into a small computer by having a part of the program written in compact interpreted code dates back at the very least to Apollo's Guidance Computer's INTERPRETER component.
- kragen 5y ago
- btilly 5y agoI remember loving these papers back in the day. First, I would be far less skeptical than this author is over the tradeoffs of scripting languages. Given the tools offered by scripting languages, there is one obvious way to do things. The overhead of building a fancy "just right" data structure almost always makes things perform worse than the straightforward solution using dictionaries, arrays and scalars. Given that, people go directly to that solution, leading to less variation in length of code, time and performance. It is always possible to beat a scripting language's performance, but a surprising number of programmers won't. This is a lesson that generation after generation of programmers have learned in a variety of scripting languages. With the latest winner being Python. But the differences between popular scripting languages generally matter a lot less than the commonalities. (There are exceptions, specialized languages like Lua, R and Julia have very specific strengths and weaknesses despite being aimed at scripting.) Second, there are two basic reasons why Lisp loses. The first is the disadvantage of being an image based system. This has a lot of upsides, but the downside is that people lose access to a long toolchain that they are used to for dealing with files. Tools including editors, source control, and so on. Programming expertise is fragile, and that is a pretty big shock. The second is that Lisp attracts people who want a perfect language. Unfortunately people's ideas of perfection different. So we get a large number of related dialects of Lisp, which are almost compatible with each other. And fail to get critical mass behind any of them. That said, how big a deal is achieving critical mass? Well obviously it matters. If you're using a mainstream language you can find packages for things that you need, answers on StackOverflow, and so on. But there are diminishing returns. The size of CPAN didn't keep Ruby from rising, and JavaScript and Python in turn have seen meteoric growth. But that said, there is a fundamental tradeoff. Working in a popular language matters more if you're planning on leveraging lots of existing work. The more your project is building on what you yourself have built, the less important the easy start of external packages is relative to your own development familiarity. The more specialized the thing that you're building, the more it makes sense to use the perfect language for you/your space, rather than going with what's popular.
- pfdietz 5y agoCommon Lisp implementations typically aren't image based systems, although one can dump images. Code is in files, which are loaded. The typical workflow involves modifying files, compiling/loading the changed files and their dependencies (using asdf), and running tests. Nothing persistent is in the image.
- jeffbee 5y agoI've been asked to whiteboard this or very similar problems in 45-minute job interviews. Based on my experience in those interviews I am not surprised to see it typically takes 3-10 hours to do a proper job of it.
- actually_a_dog 5y agoI was about to comment the same thing. It really shows up the absurdity of those Leetcode-style, whiteboard interviews, doesn’t it?
- amusedcyclist 5y agoI mean of course Java is slower if you're including JVM start up time. A more modern version of the problem would ask for a service which performs the same function and averages time taken over N requests. I suspect modern java would look pretty close to C++/rust there
- bluejekyll 5y agoThis is always an interesting question, about including the JVM startup time or not. In this context doesn't it seem like the JVM startup time should be included? During JVM startup, one thing it's generally doing is allocating space for the heap from the OS. For the C/C++/Rust application, this could be implemented either as a large arena allocation, or allocations inline during the program. That is a programming choice, but the allocation cost is still there. It would be unfair to not include that allocation for Java whereas always including it (whether inline or upfront) in the native C/C++/Rust versions. For long lived programs, like server-side things, the JVM startup time probably isn't relevant, but for short lived programs, it seems reasonable to include the startup time as that would be part of the time running the program from the CLI.
- zmmmmm 5y ago> for short lived programs, it seems reasonable to include the startup time but then one can say, if the whole problem is so trivial that startup time is a significant chunk of the execution then is it really a problem worth optimising in the first place? The only situation where it would matter is if you are putting this in a tight loop on the command line or something like that (which is a thing, but its a reasonably small thing).
- tsimionescu 5y agoThere are actually important cases where the JVM startup time is a huge problem. For example, my team chose Go instead of Java for a microservice solution because, among others, the startup time of a microservice after a scaling event or crash matters a lot for user perception. The difference between a 100ms startup time and a 500ms startup time (or more) is often the difference between a system that seems to 'just work' and a system that seems to 'constantly stutter'.
- bch 5y agoJohn Ousterhout (creator of Tcl (mentioned in OP paper)) also wrote an interesting paper about the “scripting” vs “system” language dichotomy, “Scripting: Higher Level Programming for the 21st Century”[0] which some may find interesting. [0] http://users.ece.utexas.edu/~adnan/top/ousterhout-scripting.pdf http://users.ece.utexas.edu/~adnan/top/ousterhout-scripting....
- robertwt7 5y agoIt’s interesting that even a system programming language like rust is actually not faster than java if not written / implemented correctly. People always thought that when they needed to go faster they go for c++/rust. I guess advance knowledge in one programming language is better than switching to another one in the case where you need to make your app faster Also isn’t this similar to this study? https://benchmarksgame-team.pages.debian.net/benchmarksgame/index.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
- igouy 5y ago"faster" is not a property of programming languages. "faster" is a property of programs & programming language implementations on some hardware & operating environment. The programs shown on the benchmarks game have been crowd sourced, contributed to the project by an ever-changing self-selected group. The programs shown on the blog post were written by the blogger.
- ncmncm 5y agoAnywhere you see "C/C++", treated as if a single language, you know that you will be not getting good information. It is common to claim "almost as fast as C", when it is known that C is now a low bar. For speed, nowadays, you use C++. (Or Rust, for identical reasons.) The most characteristic quality of any publication about the performance of Lisp, as for anything like it, is exaggeration. In particular, indirect costs of garbage collection are invariably concealed.
- User23 5y agoGarbage collection is not part of the Common Lisp standard. DISASSEMBLE is magic and practically speaking it’s easier to write machine language performant code in CL than any other language. Yes of course you can look at =as= output in your favored language, but it drops you out of your flow.
- ludston 5y ago> Garbage collection is not part of the Common Lisp standard. Every important implementation of Common Lisp (Allegro, LispWorks, SBCL, CCL, ABCL, ECL, CLisp and Clasp to name a few) all have garbage collection.
- User23 5y agoSure, but the standard doesn't say anything about it, leaving implementers rather a lot of freedom. LispWorks even claims to have a real-time GC![1] [1] http://www.lispworks.com/products/myths_and_legends.html http://www.lispworks.com/products/myths_and_legends.html
- Jach 5y agoI agree that "C/C++" isn't a good sign, though it was more forgivable when C++ just meant C++98... For speed, nowadays, you use X. That is, many languages can be fast, especially if they have access to intrinsics, the real question is how much special knowledge and herculean effort do you have to spend? It's still true that for many problems idiomatic modern C++ will give a very nice result that may be hard to beat (though be careful if you forgot cin.sync_with_stdio(false) or you may be slower than Python!). But it's also true that C, Rust, Java, Common Lisp, and even JS all do very well out of the box these days, while languages like Python and Ruby have lagged. For this problem, the author had to spend quite a bit of effort in a followup post to get Rust to match some 20 year old unoptimized CL. If one wanted to optimize Lisp, a simple place to start is with some type declarations, at least with the SBCL implementation. It can even be kind of fun to have the compiler yell at you because e.g. it couldn't infer a type somewhere and was forced to do a generic add, so you tell it some things are fixnums, see the message go away, and verify if you want with DISASSEMBLE that it's now using LEA instead of a CALL. For an example of going to quite a bit of trouble (relatively) with adding inline hints, removing unneeded array bounds checks, and SIMD instructions, see https://github.com/telekons/42nd-at-threadmill https://github.com/telekons/42nd-at-threadmill which I believe is quite a bit faster than a similar "blazing fast" Rust project featured on HN not long ago. But my point again isn't so much that CL is the fastest or has some fast projects, just that it can be made as fast as you're probably going to want. This applies to a lot of other languages too, though CL still feels pretty unique in supporting both high level and low level constructs. Your GC comment is weird, since any publications looking at total performance invariable include the GC overhead, nothing is hidden. The TIME macro for micro-benchmarking even typically reports on allocated memory, which can be a proxy for estimating GC pressure, and both SBCL and CCL report the actual time (if any) spent in GC. Why not complain that C++ benchmarks hide the indirect costs of memory fragmentation, which is a real bane for long-running C++ programs like application servers? But I'll admit that the GC can be a big weakness and it's no fun to be forced to code around its limitations, and historically some GCs used by some Lisps were really bad (that is, huge pause times). I've been looking at the herculean GC work being done in the Java ecosystem for years with a jealous eye, and even the newer Nim with its swappable GCs when you want to make certain tradeoffs without having to code them in.
- jillesvangurp 5y agoIt was an interesting distinction in the nineties to make whether languages were interpreted or compiled. Then things got complicated when we started compiling things just in time. And then things got even more complicated when we started compiling compiled languages to imaginary instruction sets that we then interpreted at run time only to compile them just in time. To make things even more complicated, Java now has an ahead of time compiler too. Java was merely the one of such languages and instruction sets. If you look at WASM, it's exactly the same vision: a portable instruction set that can be run anywhere. Except this time it's optimized not for just one language but any language and the most popular languages on it currently are C++ and Rust, which are compiled languages not intended for this kind of thing. There are other ways to compare languages but their run times are a bit apples and oranges. Typing systems have evolved too. Java now has inferred types and ruby and python now have type annotations. The lines blurred. Purely dynamically typed languages still exist but are increasingly sidelined in favor of at least partially statically typed languages (e.g. Javascript vs Typescript). Assemblyscript takes that to the next level and compiles to WASM instead of Javascript. You see similar evolution in the Ruby ecosystem (e.g. Crystal). And the python ecosystem, which has multiple compilers. The distinction has ceased to be relevant. It's more about what kind of limitations you are willing to impose in exchange for performance. Rust makes a few very clever tradeoffs here where they do a lot of heavy lifting with templates to provide so-called zero cost abstractions. People even attempted REPLs for it. It's a scripting language!
- hoseja 5y ago>)))))) Well that's certainly on par.
- Jach 5y agoWhat's the problem? Take out the BiConsumer that doesn't exist in the Lisp version and the Java version has the same number of closing marks.