7 ms·
The speed, size and dependability of programming languages
- herdrick 17y agoThis is great. If you've looked at Peter Norvig's spell checker shootout you won't be surprised much. Ex., the story on Common Lisp and Scheme implementations is the same here: MzScheme is the mediocre best of an unimpressive lot. (I want to see Clojure and Arc on this.) Python looks great again. F# looks weak here but wins big in the spell checker with reasonable-looking code. Sounds interesting. What's the deal with Squeak? I had thought that it very concise. The meta-message here is best of all: people are measuring language conciseness.
- silentbicycle 17y agoWell, arc's running on top of MzScheme, so it's likely to be a bit slower. Also, Stalin in the left column is a Scheme implementation, albeit one that is very restrictive for sake of raw speed. Also, it's worth noting that the features in a language that make large systems' codebases manageable (such as good module systems) are different from the features that allow you to pare down small scripts further. It's hard to show features supporting conciseness-in-the-large in a one page benchmark, so IMHO the latter is often greatly overemphasized. Being able to knock 2 lines off 20 is often just a parlor trick, though, while 50kloc off of 200kloc can mean life or death for a project. The programming world would be a happier place if the size of every legacy C++ / Java codebase were cut by a double-digit percentage.
- granular 17y ago> > (I want to see Clojure and Arc on this.) > Well, arc's running on top of MzScheme, so it's likely to be a bit slower. I'd be interested to see how Arc fares on the code-size axis.
- silentbicycle 17y agoAre there any large projects using it?
- granular 17y agoI don't know much about the Shootout, but my guess is that, if someone wants Arc there, they need to go and write whatever programs the Shootout requires you to write in order to participate in the Shootout. I'd try it, but I don't know Arc (or Lisp yet, for that matter). If someone here knows Arc and wants to see it in the Shootout, I suggest they look into the faq: http://shootout.alioth.debian.org/u32q/faq.php http://shootout.alioth.debian.org/u32q/faq.php
- avibryant 17y agoJust a guess, but: Squeak only uses text files as an export format, and that format is more verbose than file-based source code would normally be. If you only counted the code and not the metadata, it would probably do better.
- igouy 17y agoNot much - Smalltalk code is written in sentences not squiggles. The productivity comes from reading and re-using the code that has already been written rather than duplicating stuff.
- stonemetal 17y agoAlso note that in the benchmark it is a gzip of the source so it is more about how well your source code compresses.
- russell 17y agoIt would be a useful to include the language name where it is not obvious. For example, he mentions Haskel, but I cant tell which one it is.
- silentbicycle 17y agoIt's "ghc", in the chart.
- Confusion 17y agoShows and compares graphs plotting performance vs. code size, based on the programming language benchmarks at http://shootout.alioth.debian.org/ http://shootout.alioth.debian.org/. Note that 'dependability' here means 'the consistency of the performance or code size'. Few solid conclusions can be drawn: small performance benchmarks are hardly indicative for both the code size and the performance of most actual projects in a language.
- petercooper 17y agoThe references to Ruby aren't entirely what they seem in here. They seem to be relating to Ruby on the older shootouts, rather than the recent ones. That is, the uber-slow Ruby 1.8, rather than the Perl-beating and PHP-matching Ruby 1.9 which is more closely related to "yarv" on this list.
- DavidMcLaughlin 17y ago> Perl-beating and PHP-matching Which is interesting considering Perl beats PHP on both axis in the linked article.
- bad_user 17y agoMaybe that's because Ruby 1.9 isn't being used in production yet. I tried using it but many gems are still incompatible with it.
- SwellJoe 17y agorather than the Perl-beating and PHP-matching Ruby 1.9 Maybe I just can't read, but Ruby 1.9 appears to still be slower than Perl on more than half of the tests. I wouldn't call that "Perl-beating".
- draegtun 17y agoIts great to see Ruby so much faster with 1.9 because 1.8 speed was such a stumbling block in data intensive stuff that I pushed towards it. However don't count your chickens just yet with regards to Ruby 1.9 being "Perl-beating" ;-) Don't you think its odd that Perl 5.10.0 sits only just above Ruby 1.8.7 in the current alioth shootout? Quick look at the tests and I can see a problem with a known 5.10.0 speed issue bug. Fixing this in a quick local test here makes all the difference ;-)
- igouy 17y ago> Fixing this in a quick local test here makes all the difference ;-) It makes all the difference to that one benchmark - not to the overall situation.
- silentbicycle 17y agoVery good graphs. Aside from X (time) and Y (code size), the center of the star implies "typical" performance, and the relative spread of the star shows specific data points. Ruby, for example, shoots out from the bottom right - to me, this says, "This is quite expressive, and if you know what you're doing, you can get reasonably good performance". It doesn't show memory usage or sample size, however - Rebol probably has significantly fewer samples than Python, for example, but it only looks like half as many. It's also interesting how the LuaJIT star has about the same shape as the Lua star, but with 1/3 the width (which is consistent with my experience); Psyco relates similarly to Python. Surprisingly, the Lua star's center appears to be at the same X as GHC's, but lower down, and LuaJIT's overlaps OCaml's. While I can accept well-written Lua being approximately as expressive as Haskell (depending on the problem), I don't think it's in quiiiite the same ballpark as OCaml, speed-wise. The most interesting shape, to me, is Io's: While the default performance seems pretty poor (the right edge meaning 8x as slow as the left edge), there are bands shooting all the way to the left, without adding any real height. I don't have any experience with Io (I'm curious, but have never had success porting it to OpenBSD/amd64); can anybody here comment on this?
- magv 17y ago> The most interesting shape, to me, is Io's There seems to be a mistake. If you compare Io to JRuby [1], you'll find that Io is always the slower one, even though the picture in the article suggests otherwise. [1] http://shootout.alioth.debian.org/gp4/benchmark.php?test=all&lang=io&lang2=jruby http://shootout.alioth.debian.org/gp4/benchmark.php?test=all...
- silentbicycle 17y agoAh, ok. Also, it seems to be missing about half the benchmarks.
- by 17y ago"Ruby, for example, shoots out from the bottom right - to me, this says, ... if you know what you're doing, you can get reasonably good performance" Sorry, but it does not say you can get reasonably good performance, it says this version of Ruby is slow, one of the slowest languages in the Shootout. And the graph itself says nothing about the skill level required to obtain the exhibited performance.
- bayareaguy 17y agoLanguages which include functional features such as lambda, map, and tail call optimization are highlighted in green. I found one minor inconsistency: Lua has all these but is still colored blueish.
- chops 17y agoIt doesn't have a native map command, but that's a rather trivial implemention. But I agree, it should be colored green in the "functional programming" section.
- silentbicycle 17y agoLua isn't usually associated with the functional languages. Besides mainly being promoted for embedded / extension use, I think the single biggest reason for this is that, while it has tail-call elimination, closures, first class functions, etc., you still have to say "return x" at the end of functions. I do a lot of FP-style stuff in Lua, and while the language doesn't actively resist it the way Python does, it's not the main direction its design flows. It seems like a small thing syntactically, but languages that return the result of their last expression by default (I think of them as "expression" languages), and that require an explicit block for side-effects (begin in Scheme and OCaml, progn in Common Lisp, etc.), tend to get idiomatically used in a functional manner, while languages that default to a series of statements with side-effects and require an explicit return at the end ("statement" languages) typically do not. While you can use several of the latter languages for functional programming, only the former are functional "by default". Saying "function(x) return x + 1 end" is a tiny bit more trouble than "fun x -> x + 1" or "(lambda (x) (+ x 1))", so while lambdas still get used where they're the best fit (as arguments for map or filter, for example), they're less common overall. Defaults influence a language's overall style at least as much as what's theoretically possible. Python's concept of what's "pythonic" seems to be the most explicit acknowledgement of this. Incidentally, I would love to see a language that crossed Lua with the ML family: a small, portable, clean implementation that interfaces easily with C, but with type inference, ML's top-notch module system, and (ideally) Haskell-style type-classes. OCaml is such a good language that I'm willing to tolerate its warts, but I'd strongly prefer a minimal ML-like language. (FWIW, I haven't used SML, though that's just because I do most of my programming on OpenBSD/amd64 and the smlnj port is i386 only.) When standalone, Lua is best suited to small/medium projects. Its historical emphasis on embedding in an existing project has made a module system for standalone use a bit of an afterthought.* As a project gets larger, static checking and stabilizing the module interfaces becomes far more important, and this is one area the ML languages really excel. * For a good summary, see "Almost Good Enough to Scale: A Lua Mail Handler and Spam Filter" (http://www.lua.org/wshop08.html#ramsey http://www.lua.org/wshop08.html#ramsey)
- nazgulnarsil 17y agodependability mostly is a stand in for a program making more assumptions for you no? In a language that makes a lot of assumptions its easier to create something where assumptions conflict somewhat invisibly.
- unbannable 17y agoFor this sort of comparison, mostly-functional multi-paradigm languages like SBCL and Ocaml should be run through the benchmarks twice; once with imperative code, once in the functional idiom. The often-repeated complaint (of which I'm skeptical) about functional languages is that they're only efficient when ugly imperative features are used, and it would be good to confirm or dispel that suspicion.
- silentbicycle 17y agoIt's more complicated than "functional is slower than imperative". Functional data structures are sometimes less immediately efficient, but add persistence (since they're immutable, they share subsections of the data structure, and/or have access to snapshots of past versions of the structure being modified). Whether this gives you needed features or just means extra work and cache misses depends on what you're doing. If you have a functional data structure and you want to add "undo" or backtracking, you're already most of the way there, while adding this to an imperative data structure means extra work to keep track of state changes. _Purely Functional Data Structures_ by Chris Okasaki is, by far, the best collected resource, if you want to read further. (His thesis (http://www.cs.cmu.edu/~rwh/theses/okasaki.pdf http://www.cs.cmu.edu/~rwh/theses/okasaki.pdf) is the kernel of the book.) Also, the fourth chapter of Developing Applications with Objective Caml (English translation: http://caml.inria.fr/pub/docs/oreilly-book/ http://caml.inria.fr/pub/docs/oreilly-book/) discusses functional and imperative styles' strengths and weaknesses. Reducing everything to runtime benchmarks obscures how, in practice, the trade-off is often more like "it was a huge pain in the ass to get working without bugs, but it's 2% faster" versus "it took twenty minutes and was correct on the first try". (Also, whether or not you use it in production, OCaml rules for prototyping complex data structures, in the same way Erlang does concurrency and C++ does linking errors.)
- javert 17y agoThis is one of the best things I've seen on the Internet in a while. :-) Definitely worth taking a look.
- ellyagg 17y agoInteresting. Looks like Yegge was right in falling in love with ocaml awhile back.
- tlrobinson 17y agoThe benchmarks used for the article was from 2005. Make sure to check out the 2009 versions: http://gmarceau.qc.ca/blog/uploaded_images/size-vs-speed-vs-depandability-paradim-2009.png http://gmarceau.qc.ca/blog/uploaded_images/size-vs-speed-vs-... V8 has propelled JavaScript into the coveted bottom-left corner, which is awesome. SquirrelFish Extreme should be right up there with it, if it were included.
- SwellJoe 17y agoMore reason to believe JavaScript is going to be huge on the server-side. As concise and (nearly as) powerful as Python, Perl, and Ruby, but getting faster at a much more rapid clip...and already required for the front-end, so everybody on the team is already familiar with it. Library coverage will take another couple of years to reach a reasonable level, but I'd bet on it happening.
- kinghajj 17y agoI agree. Once I learned how truly powerful Javascript really is, once you get past the horrible DOM, I knew it could one day become a general-purpose scripting language and not just a web one. As soon as someone makes a good command-line Javascript interpreter, web apps will come as soon as someone creates a CGI extension to that interpreter.
- tlrobinson 17y agoI'm working on it: http://jackjs.org/ http://jackjs.org/ http://narwhaljs.org/ http://narwhaljs.org/ https://wiki.mozilla.org/ServerJS https://wiki.mozilla.org/ServerJS
- silentbicycle 17y agoHave you ever looked at Lua?
- deleted 17y ago[deleted]
- 17y ago
- JulianMorrison 17y agoI wonder why mlton doesn't have a higher profile. Is it hard to use, or something?
- jongraehl 17y agoI looked at SML and there seemed to be a few warts compared to Ocaml: http://www.mpi-sws.org/~rossberg/sml-vs-ocaml.html http://www.mpi-sws.org/~rossberg/sml-vs-ocaml.html mlton may not allow separate compilation (it does whole-program optimization at least), and it's pretty slow even on small stuff, but if you were really using just the Standard ML language, you could use a faster compiler for development. I'd probably want to use some extensions, and I wasn't sure if the intersection of mlton's extensions with some other implementation could satisfy me.
- silentbicycle 17y agomlton seems to be used for optimized compilation for releases, while general development is typically done in another SML environment, such as smlnj. I don't have really concrete experience with SML, just OCaml, which also has its share of warts. It's a fantastic language in certain niches, though; generally, the sort of niches for which dynamic / "scripting" languages are usually a poor fit. (I wouldn't write a serious compiler in Python, for example.) They're very complementary. I think the ML family has a lot of advantages, but the community is pretty closely tied to academia (much like Haskell, but without the buzz). While that in itself is probably neither good nor bad, in this case a lot of the terminology used is especially math-y; There are many features ML has that programmers would find tremendously useful, except they're usually discussed using Greek letters, which is insular. I got a copy of The Definition of Standard ML from Amazon, hoping to get further insight into the language's design, and found it completely impenetrable. (Then again, I'm not a grad student.) Newer languages seem to be using type inference (finally!), but as far as I know, nobody has really ran with the design of its module system.
- ck113 17y agoI thought this was the most telling line of the article: "Ultimately the first factor of performance is the maturity of the implementation." That supports a common conviction held by fans of functional programming: if all of the years of arduous optimization that have been poured into GCC had instead been poured into (say) GHC, then Haskell would be even faster today than C is. That is, to many people functional programming languages seem to have more potential for performance than lower-level procedural languages, since they give the compiler so much more to work with, and in the long run a compiler can optimize much better than a programmer. But so much more work has been put into the C-style compilers that it's hard to make a fair comparison. It's still hard, but this experiment seems to give some solace to the FP camp.
- jongraehl 17y agoHaskell may potentially be compiled to within epsilon of C when using strict evaluation + mutable state + unboxed types (look at the shootout implementations; the majority are in this style), but lazy evaluation and functional data structures have a real cost. It's not merely that the ghc developers have been insufficiently clever and diligent.