20 ms·

Comparing the Same Project in Rust, Haskell, C++, Python, Scala and OCaml

783 points by trishume 7y ago

HelloFellowDevs 7y ago
Love reading your posts Tristan, a question about the choice of Rust. Was it specifically based on the previous experience you and your groupmates drew on or were these languages assigned to students? Was there any point in time where you and your team were trying to optimize your compiler?
- trishume 7y ago
  We got to choose. Everyone on our team had some experience in Rust and OCaml, the two of us more experienced in Rust were pretty indifferent between OCaml and Rust, and our third teammate wanted to use the project to learn more Rust so that's what we went with.
mruts 7y ago
I wonder how Racket would compare. I’ve written some DSLs at an old job for Racket and found the libraries for language design almost unreasonably powerful (and maybe unreasonably complicated as well). The dynamic typing would also help in reducing the amount of code.
- trishume 7y ago
  Yah I figure it would be like Python but better if you were allowed to use the language design libraries. However the course specifically forbids standard library components that are designed for implementing languages so that you have to write the parser and things yourself.
  - deleted 7y ago
    [deleted]
  - neilv 7y ago
    But you could use Racket if you wrote your own parser generator macro as part of the project's code? And would you be allowed to use Racket's existing facilities for representing and implementing macros? (You might want to use the macro stuff for your AST, IR, and transformations. Though you could still win with normal data types, but the macro stuff can help you win even more.)
spullara 7y ago
I'm sad that no one wrote it in the subset of Java that it was going to compile.
- denormalfloat 7y ago
  It's more surprising because javac is itself written in Java. It's definitely up to the task (even if it's targeting x86 instead).
  - spullara 7y ago
    Maybe no one thought of the immense satisfaction they would get at the end when they compile their compiler to x86 code after compiling it with javac to bytecode to run it the first time.
- mruts 7y ago
  They probably wouldn't have been able to finish the project in time.
  - ncmncm 7y ago
    And a good professor would have flunked them for poor judgment.
    - saagarjha 7y ago
      I doubt I will ever be a professor, but I would be happy to pass any student who did this provided they did the task adequately. Maybe extra points for cheekiness.
    - trishume 7y ago
      My professor was definitely a good professor and no way they would have done that. It clearly would have been done as an added challenge for themselves for fun and there's no reason to penalize that.
      - 7y ago
boyter 7y ago
I really wish when counting code that people would use one of the more modern code counters, or at least use cloc. Tokei, loc, scc, polyglot, loccount or gocloc give a much better idea over wc because they account for comments and blank lines and in the case of scc and tokei strings. I had a similar experience in university. Class had to implement a modified Turing machine. We could use whatever language we wanted. On person did it in C++ and it was several hundred lines. Another in Java which was slightly smaller. I implemented mine in Python and it was small enough to print on a single piece of paper. I think it was something like 30 lines or so. I exploited break statements to make it terse but readable. It did mean I was marked down a bit for them but it was by far the shortest program in the class.
- trishume 7y ago
  A few groups did give me output from a modern line counter, often Tokei or loc. I mention in the post that the ratios between source lines, word count lines and bytes were pretty similar across all projects. If people want to get a sense of how that compares to their intuitions in source lines they can use the ratio from our Rust project to convert. I ended up using raw lines for comparison because for relative measurement it didn't matter for the above reason, and I knew everyone had the same `wc` and I could get them to send me the output of the `wc {glob}` version that lists lines and bytes of all files so that I could drill down into which parts were different.
  - boyter 7y ago
    Ah! I assumed you had access to the code. That makes more sense.
- OnlyOneCannolo 7y ago
  For anyone interested in this sort of thing, there's also Unified Code Count (UCC) [1]. It has a lot of interesting design goals like being open and explicit about the counting rules, which is really useful if you want to predict things like cost and reliability. [1] https://csse.usc.edu/ucc_new/wordpress/ https://csse.usc.edu/ucc_new/wordpress/
ufmace 7y ago
I haven't made it through the whole thing yet, but I do want to register a vote in favor of using Lines Of Code count as a rough measure of program complexity. I think it's a perfectly valid things to do, provided that it isn't used as an evaluation metric and nobody is gaming it, and everyone is a reasonably good programmer, not doing crazy things like trying to stuff a massive algorithm onto one line to be clever, or copy-pasting the same code in 20 places because they don't understand functions and classes.
- chimpburger 7y ago
  When comparing different languages it becomes more of a measure of the relative abstraction levels of languages. The exact same program will have hundreds (or thousands) of times more LOC in Assembly than in Python.
  - applecrazy 7y ago
    Not only that, the LOC heuristic depends on the verbosity of the language. Java, for example, is more verbose, than say, Python, even though you’re close in abstraction level.
    - gpm 7y ago
      And just syntax decision. if (x) { y } else { z } if x: y else: z Assuming x and y are 1 line but long enough expressions you don't want to use a trinary operator (available in both) the python is 2/3rds the length of the c because of bracketing. Obviously cherry picked, but I bet these differences add up.
      - saagarjha 7y ago
        Of course, you can omit the braces if there’s a single statement (though I personally never do this).
        7y ago
jpittis 7y ago
> I think the smaller differences are also large enough to rule out extraordinary claims, like the ones I’ve read that say writing a compiler in Haskell takes less than half the code of C++ by virtue of the language Specifically the "by virtue of the language" part: Seems to me like it's unreasonable to claim the languages are on equal footing because fancy parser libraries aren't allowed to be used for the project. The fancy parser libraries exist for certain languages specifically because the languages enable them to be written. (For example in Haskell: monadic libaries, libraries that take advantage of GADTs, etc.)
- trishume 7y ago
  I don't think monadic parser libraries have a real claim to be that difference. All the languages listed have excellent parsing libraries that make things similarly easy, if not by language power than by grammar DSL with embeddable code snippets. I think if any library could make a real difference for Haskell it's most likely to be http://hackage.haskell.org/package/lens http://hackage.haskell.org/package/lens, which a Haskeller friend of mine claims could likely make a lot of the AST traversal and rewriting much terser.
  - howenterprisey 7y ago
    Hoopl (data flow analysis) would also make a difference. I did a very similar project at my university in Haskell and Hoopl definitely saved us from writing quite a bit of code. We also used parser combinators in the frontend, which I think saved us time too.
  - anaphor 7y ago
    I've found PEGs (Parsing Expression Grammars) to make things extremely easy and terse. E.g. OMeta, Parsley, etc. My experience with using both PEGs and parser combinators is that there isn't a huge difference in the total number of lines of code. On the other hand though, the syntax of PEGs would be easier to understand for someone who is familiar with BNF style notation.
  - pwm 7y ago
    While I found your article informative and interesting I think it only works in the very specific context of this assignment. Disallowing powerful language features/libraries means it's not a level playing field and thus not a fair comparison. Some languages standard libraries are tiny some are huge. Some languages have lots of advanced features. Eg. GP mentioned GADTs with which one can write type safe/correct by construction ASTs. In other words programs passing specific tests in a specific context does not imply they are comparable in terms of general correctness/robustness/maintainability (as you noted this re caught edge cases).
kris-s 7y ago
Great writeup, I'm very curious about the performance between the compilers.
trimbo 7y ago
Where's the code? I scanned this article a few times and can't find a link to it.
- trishume 7y ago
  None of it is open source because it was done for a school course that uses the same project every term. The only use for having access to the code would be cheating on that project. The school gets mad when people publish the source for their school projects, especially when it isn't otherwise useful, that's also why my ray tracer isn't open source.
  - empath75 7y ago
    I would not agree that that is the only use of that code. I’ve found a lot of school projects on GitHub when googling how to do stuff as a professional programmer and have found them useful quite often.
  - wolfgangK 7y ago
    Being a teacher myself, I understand this policy. Too bad, though, I'd have dearly loved to read the python implementation to rewrite it in Clojure. I think the language could allow for something at least as concise be maybe cleaner.
sonnyblarney 7y ago
"Their project was 17,211 raw lines, 15k source lines, and 637kb" So a couple of students cranked out 15K lines of code, in basically a subset of their study time, for a single class? I don't doubt they did it, what I doubt is the quality of what they wrote. Because it's 'just a project' it doesn't have to be fully debugged ... But that is a vast amount of code to hand-write over a short period of time. At that level, I think it's basically just 'code like the wind' with not much consideration for whether it works, architecture etc. etc. - which in a way might put the conclusions of the 'mini study' at odds. At 15K LOC dashed out very quickly ... these are not products, they're comparing 'rapidly written out lines of code', which is something else entirely from working code. Also, given that everyone is in Uni, and likely may not have been exposed to proper idiomatic code for the language ... this presents another issue. Ideally we'd want to compare reasonably idiomatic code for one language, to another. Kudos to the author for this, but we should be aware of some caveats.
- trishume 7y ago
  Note that project is the outlier for how much code it contained. That team did have the most trouble getting things done on time and fixing all the bugs, so they passed the least tests, as one would expect given how much code their design choices required them to write. I think the causality goes from their design to the line count, not from the line count to worse design, they had to write lots of code to implement their more abstract design choices. I looked at their code and didn't see any obvious deficiencies in anything other than the design. Also note that UWaterloo's CS and SE programs are different than other universities. Everyone on all teams had at least two years of full time work experience at 6 internships. The people I talked to were also programming enthusiasts who read online a lot. The only teams that may not have written the most idiomatic code are the person who used Python alone, and the Haskell team because I've heard idiomatic high-end Haskell involves an insane amount of knowledge of abstractions like lens, way more than all the other languages. This compilers class is also well known for requiring lots of work, so students often arrange their schedules to have a low load from other classes while taking it, which is possible since the CS program requirements have a lot of flexibility.
  - 7y ago
akhilcacharya 7y ago
Loved this part and thought it was hysterical - > Since my team had all interned at Jane Street the other language we considered using was OCaml, we decided on Rust but I was curious about how OCaml might have turned out so I talked to someone else I knew had interned at Jane Street and they indeed did their compiler in OCaml with two other former Jane Street interns. Relatedly this is why I get annoyed at people that don’t believe 10x or Nx developers exist - they absolutely do!
- dudul 7y ago
  Can you expand a little? I fail to see the connection between the quote and your conclusion.
  - akhilcacharya 7y ago
    Jane Street only interviews and hires really exceptional engineers. A team of exceptional engineers in a single CS class, even if it is at a school like Waterloo, would imply that the 10x phenomenon isn't a myth but a somewhat mundane reality in certain circles.
- ncmncm 7y ago
  In the 90's, the Swedish Ericsson and German Siemens companies participated in a joint (crash) project. Each sent 250 engineers. After six months, they delivered. A friend counted up lines of code written by each engineer, at the end. Fully half the code in the final product was written by one person: N lines by him, N lines by the other 499 people. He was a lead programmer, who issued two-week assignments to other engineers; if one wasn't done come Friday, he would do it himself on Saturday. He doesn't consider himself especially fast, because he knows someone else who codes ten times as fast, and wears out two keyboards per year. This project was also interesting because they took blood samples, by which they got objective measurements of stress. Everyone's stress level increased right up to the deadline -- except his. And their stress levels did not start falling until many months after.
  - madez 7y ago
awwaiid 7y ago
If you're into this sort of thing, definitely check out the ICFP Programming Contest -- take next Friday off and join in on the fun! https://icfpcontest2019.github.io/ https://icfpcontest2019.github.io/
- saagarjha 7y ago
  > Any programming language(s) on any platform(s) may be used. How does this work? What’s stopping me from using some hyperpowered esolang I cooked up for the competition?
  - deleted 7y ago
    [deleted]
  - krick 7y ago
    Nothing, if it helps you with the challenge.
  - dbpatterson 7y ago
    That the task description is only posted at the start of the competition?
    - geofft 7y ago
      I mean, if you design an esolang for the competition task and implement an optimizing compiler that's faster than any general purpose language out there, that absolutely feels like something that should be rewarded by the spirit of the competition! So it's okay that it's allowed by the letter of the competition, too.
  - exsf0859 7y ago
    If I recall correctly, the 2001 ICFP contest was won by someone who happened to have done their thesis on the contest problem. :-)
- lukego 7y ago
  Has anybody made a study of the corpus of ICFP contest entries? Seems like an interesting dataset with many languages, many programmers competing multiple times, etc.
mamcx 7y ago
Is possible to see the code of this project? And is the same of http://thume.ca/2019/04/18/writing-a-compiler-in-rust/ http://thume.ca/2019/04/18/writing-a-compiler-in-rust/? P.D: I'm building a relational language and my ideas mimic closely "writing-a-compiler-in-rust" ie: using pratt parsers and similar, but not see much of how do that on rust...
ncmncm 7y ago
I find it curious that both the team that used C++, and the author, both appear to believe that sum types and related facilities are not usable or conveniently expressible in C++. I got that Boost Spirit, a powerful parsing library, was forbidden. Were all the Boost libraries similarly embargoed?
- trishume 7y ago
  There was a general prohibition on libraries that don't ship with the compiler, but you could ask the professor for an exception. Someone asked about Boost and a blanket exception for the non-parsing and scanning parts of Boost was granted. However the C++ team I talked to didn't use it. Also note that it's not only the sum types that are important, powerful pattern matching facilities are part of what makes them so valuable.
  - shaklee3 7y ago
    C++ has had sum types for years, so I'm not sure why that was mentioned. If you aren't familiar/experienced with the language, then the comparison is not fair overall.
    - jcora 7y ago
      Yeah it's had everything for years. I'd rather shoot myself than use those features outside of a modern language where they aren't just bolted on along a billion other things.
      - shaklee3 7y ago
        That's not a useful comment. Would you rather they just kill the language instead of improving it?
        jcora 7y ago
        Absolutely. Killing it would be the most useful improvement.
bfung 7y ago
Excellently written, great topic, and done w/o flaming/too much bias. I'm amazed as many older folks in the industry would not be able to have this level of content and maturity to write an informative article.
- WalterBright 7y ago
  It's difficult to write a fair comparison without being a fairly competent programmer in each of the languages. The trouble is, if a person is an expert C programmer and then translates it to Python that he's only modestly familiar with, the Python program will look like C. It won't be idiomatic Python. For example, my early Fortran programs looked like Basic. My early C programs looked like Fortran. My early C++ programs looked like C. And my early D code looked like C++. It takes much more than being able to write a program in X to be able to write one that makes proper use of the language.
  - galaxyLogic 7y ago
    That is all true. At the same time here the groups were allowed to use the language of their choice. Presumably they chose languages they felt they were competent in. Of course an expert of a given programming language can write much better in it than a novice. But a comparison like this is not necessarily about comparing top-programmers in every language, but average programmers, because we want to know results that are true "on average" . The author does note he "knew (they) were highly competent". So they were not exactly novices in their language of choice. Writing a compiler is not a task for novices in general.
    - anaphor 7y ago
      The article says that the instructor for the course cautioned against using Haskell because some people overestimated their competency with it. I think it is actually very likely that people would chose a programming language or system for reasons other than how competent they are with it. E.g. to seem "smart" because you wrote your compiler in Haskell, even though you actually have much more experience with Java or Python. FTA: > Another interesting thing to note is that at the start of every offering of the course the professor says that students can use any language that can run on the school servers, but issues a warning that teams using Haskell have the highest variance in mark of any language, with many teams using Haskell overestimating their ability and crashing and burning then getting a terrible mark, more than any other language, while some Haskell teams do quite well and get perfect like my friends.
galaxyLogic 7y ago
Prolog, and Definite-Clause-Grammars https://en.wikipedia.org/wiki/Definite_clause_grammar https://en.wikipedia.org/wiki/Definite_clause_grammar
nautilus12 7y ago
I can hear all the Haskell fanboys screaming that they didn't do Haskell right because a kleisli arrow didn't appear and ascend them into monadic heaven.
- aetherspawn 7y ago
  To be fair, I don’t think any Haskeller would actually write this project without attoparsec and maybe lens by choice. They’re basically base libraries for this sort of thing.
  - nybble41 7y ago
    Agreed. The rule about not using any library that doesn't ship with the compiler creates a heavy bias in favor of "batteries included" languages like Python. I understand not allowing parsing libraries, given the nature of the assignment, but if even widely used libraries like lens are off-limits the result is not going to resemble idiomatic Haskell. As a matter of basic fairness, if some library or built-in language feature is to be permitted for one implementation then analogous libraries should be permitted for all the other implementations regardless of whether they happen to ship with the compiler. The eval function leveraged heavily by the Python version should probably also have been off-limits, for the same reason that parsing libraries were prohibited. Using eval amounts to embedding an existing fully-developed parser and runtime environment into the project as a library.
    - dymk 7y ago
      I think a much more interesting comparison would be to ask an expert in each one of these languages to implement a "production" ready version of the project - so allowing popular, stable, community-accepted libraries to be used. Then we'd see what truly idiomatic solutions to this look like.
FullyFunctional 7y ago
That was a really interesting article. I completely agree with >>> Abstractions may make things easier to extend in the future, or guard against certain types of errors, but they need to be considered against the fact that you may end up with 3 times the amount of code to understand and refactor, 3 times the amount of possible locations for bugs and less time left to spend on testing and further development. Choosing when and how to abstract is key. Abstraction in a fashion extends the language so you are trading off a burden on the reader (who have to learn and trust the abstraction) for increased expressiveness. Don't overdo it. (And don't abstract idioms).
- mhh__ 7y ago
  I think thinking carefully about the abstractions is also key, ideally not while at the computer. Good abstractions should be obvious and not really requiring refactoring as per se.
- rjayatilleka 7y ago
  Could you clarify what you mean by "don't abstract idioms?"
  - FullyFunctional 7y ago
    One example (taken from the real world): addr = (addr + PAGESIZE - 1) & ~PAGESIZE; contrast with addr = P2ALIGN(addr); The former is idiomatic and immediately readable. For the latter I have to go lookup the definition of P2ALIGN (and make sure I got the one actually used as there may be multiple!), check that it does what I expect, and memorize it. The value of abstracting the idiom is debatable. If it _is_ used sufficiently often, then it might be worthwhile, but often it isn't.
galaxyLogic 7y ago
I think the big big result from this study is: "Python: half the size !". A dynamic language like Python is better here, 2x better. I assume similar results would apply to other dynamic languages like JavaScript, Lisp, Smalltalk, Groovy etc. This does not say that static typing should not be used but I think it shows unequivocally that there is a considerable extra development cost associated with static typing. You might say that surely that additional cost would be compensated in reducing the cost of maintenance later. Maybe but I'm not sure. Any development effort of significant size (like writing a compiler here) is a combination of writing new code and adapting and modifying code already written. "Maintenance" is part of development. This is quite a quantitative study which gives credence to the claims of advocates of dynamic languages.
- trishume 7y ago
  I agree it's definitely an interesting result and a point in favour of dynamic languages. A caveat is that I'm pretty sure my friend intentionally sacrificed code quality to do it, I don't think you'd find that project as readable and understandable as the others. Another caveat is that you have to be okay with your codebase being extremely magical and metaprogramming-heavy, which many industrial users of dynamic languages avoid. As I mention, I'm personally into statically typed languages mostly for the performance and correctness benefits. I think it's plausible that on larger projects with teams the correctness benefits save enough debugging time that overall implementation time is lower, but I'm less confident in this now than I was before this comparison.
  - galaxyLogic 7y ago
    I've read my share of cryptic JavaScript written by others and in that sense I agree that in multi-person long-term projects static typing no doubt will have its advantages. My hunch is however that what is often overlooked in development with statically typed languages is that it takes considerable time and effort to come up with the right set of types. Many examples are written showing how types almost magically make programs easier to understand. But when you read such an example what is not stated is how much effort it took to come up with just those types. One way of thinking about it is that type-definitions are really a "second program" you must write. They check upon the primary program and validate it. But that means you must write that second program as well. It's like building an unsinkable ship with two hulls one inside the other. The quality will be great but it does cost more.
stretchwithme 7y ago
It would be interesting to see experts in different technologies compete, in front of a live audience, to implement or change a variety of projects. It would be useful to see what a master can do with each tool or platform.
- galaxyLogic 7y ago
  It would be interesting and educational to see how they would do it, definitely. But when comparing languages it makes sense to compare how well average people perform with them. We can't always hire an expert they are not universally available.
  - stretchwithme 7y ago
    It would certainly help people decide what tobecome an expert in. And there must be some correlation with what an expert and an amateur can do. Of course, I can think of examples where that isn't true.
analog31 7y ago
I conducted a similar exercise, albeit on a much smaller scale, when I was "language shopping." I took a project that I had written in Visual Basic (my go-to language at the time), and re-wrote it in a variety of programming environments. The project was just complex enough to require maybe 100-200 lines of code at the end of the day, thanks to liberal use of libraries. Not huge, but not atypical for a "scientific" programmer who uses code as a way to solve problems rather than to create software for others to use. It's pretty representative of my life as a programmer. The exercise forced me to learn enough of each language to get a feel for it, and then I could look at the programs alongside one another to assess their strengths. I also imposed some rules, such as that the language had to run on multiple platforms, and be FOSS. In addition to comparing the languages, I was also implicitly comparing libraries and even access to online help. This was also my first real exposure to StackOverflow. I tested Javascript, Python, GNU Octave, and wxMaxima. Ultimately Python won out and is my language of choice today. This was just my own little exercise, and not worth publishing, but has made me a believer in learning multiple languages.
- new4thaccount 7y ago
  Give Julia a try if you haven't. If you're coming from Python and also have some Octave experience, it will be easy and worth a look. For scientific coding I usually use either Python or Julia and Fortran or Octave at home sometimes.
anaphor 7y ago
I think the right metric to measure here is not quantitative, but more qualitative. In other words, how easy it is to adapt the language to the problem you are trying to solve. So, in order to do that you need to have a good understanding of the principles each language is based on. Once you've got that, then you look at the resulting code and "measure" how easy it is to understand. As was already said, this requires having someone with a very good understanding of the underpinnings of each language, which is not really going to be reasonable for most people. The problem is that if you simply tried to port an idiomatic solution from e.g. C++ or Python into Haskell or Scala, or whatever, then you would probably end up with something very ugly, because you didn't adapt the language to your problem. You tried to force it to do something it wasn't necessarily designed to do.
- Buttons840 7y ago
  A contest to see how easily code could be broken would be interesting. "I purposely broke the code somewhere in these 20 lines. Find the bug and fix it." You can use the compiler, you can use the test suite. You cannot diff the code against the original working code.
huxingyi 7y ago
The actual effort of writing one line of code in the specific language doesn’t mention in the article. Write one line of rust is extremely difficult than one line of Python, or even C++.
zimablue 7y ago
I feel like what you're actually testing with this is "amongst the top percentile of programmers, what are the proclivities of people by choice of favourite language" Because the implementations vary so much, that's the source of a lot of the LOC-difference. They effectively delivered more or less (if you count more stages as "more" which I would as it will make it easier to reason about/debug, and count a typesystem as "more", since it gives you more guarantees). python - solves the problem brutally fast, some ugly shortcuts haskell - solves the problem quite slowly/delivered the most rust/c++ - intermediate scala - solved fast, took shortcuts ocaml - this is the one that surprised me I'd have expected it to be the shortest with python
- dean177 7y ago
  Why would you expect the ocaml one to be shorter?
tanin 7y ago
The Scala result confirms my bias :P I love Scala because it strikes a great balance between being succinct and offering type safety. I wish you would use Ruby instead of Python. Python is strangely inconsistent. For example, Python doesn't really offer a rich standard library; people have to resort to ugly solutions for a simple problem (here's an example: https://stackoverflow.com/questions/363944/python-idiom-to-return-first-item-or-none https://stackoverflow.com/questions/363944/python-idiom-to-r...) Ruby would be better at an example of how a dynamic language can be more succinct.
- trishume 7y ago
  Preaching to the choir, I personally use Ruby over Python :) But I had to source from friends in the class, and my group used Rust because I'm all about that performance and correctness nowadays.
- nhumrich 7y ago
  "python doesnt offer a rich standard library" Python is usually considered one of the richest standard libraries out there. Yes, the is no standard way to get this one edgecase, but every language has warts and things like that. It sounds like you are just looking for an excuse to hate on Python.
  - tanin 7y ago
    > but every language has warts and things like that Sure, but we are talking about the brevity of a language. The richness of the standard library directly impacts brevity. So, Ruby would have been a better representative (for brevity) of a high-level dynamic-typed language. > the is no standard way to get this one edgecase I don't think `.first` is an edge case. It is used fairly often. One example that I can think of right now is when you want to fetch the first row from MySQL. MySQL returns an array, and you would need `.first` to get the first row or null. > It sounds like you are just looking for an excuse to hate on Python Not at all. While I don't prefer Python, I recognize there's a downside to a richer standard library. The language becomes more complex; harder to learn. Richest is relative. If we consider Python against every other programming language, then, yes, it's ONE of the richests. If we consider Python vs. Ruby vs. Scala, I doubt Python would be considered as richer or richest. For example, Scala offers a lot of ways to process a list like foldLeft, foldRight, unzip, headOption, lastOption, flatMap, groupBy, and many methods around Map, Set, and etc. There are many examples where Ruby version would result in shorter code like `array.delete_if` and etc. Python probably doesn't offer many of these methods.
Scarbutt 7y ago
How much time was given for this task?
chenglou 7y ago
Hey trishume! It’s been a while (this is the Reason person). Great post; it echoes all the experiences I/we’ve personally had, especially the alternative Rust section: on the spectrum of how much intelligence one needs to understand fancy abstractions, all of us programmers, good and bad, are pretty much lumped to the lower end. I’ve seen plenty of skilled folks “decompensate” their expertise by reaching for overly complex abstractions. The definition of a “strong” programmer, as per the post and usually in real world, should probably be rectified to include not only “the person knows how to use this abstraction”, but also “the person knows when to stop”. In the same vein of idea, it’d be interesting to know how Go would fare in this project. Go’s philosophy is pretty much the opposite of a language that’s usually used for compiler research; but I’ve seen indicators that using it could be surprisingly effective (and in light of this post, maybe less surprising). More importantly, it’d be nice to know the perf characteristics of each project =)
- trishume 7y ago
  I expect Go would end up like C++ without header files or OCaml without sum types and pattern matching, falling somewhere in between them. Although if errors are handled properly instead of by panic then it might really hurt in the line count from all the `if err != nil { return; }` Also I commented somewhere else in this thread re perf comparison. Short answer is that since there's no incentive towards performance the signal would be swamped by differences in how much people avoid O(n^2) algorithms even if you don't need to.
- Matthias247 7y ago
  > In the same vein of idea, it’d be interesting to know how Go would fare in this project. I would say a bit worse than the mentioned alternatives, which all have better type systems and thereby e.g. make it easier represent and manipulate the trees that are everywhere in compilers. But most likely it's still fine, and the line count metric would be more influenced by the fact how experienced the author is than the language. Things will get worse in Go if one wants not write a lexer/parser manually but use tooling for that. Parser combinators and other tools benefit a lot from generics. On the other hand e.g. writing a network server with decent performance will be in Go a lot easier and more straightforward than in any of the other mentioned languages. While all those languages are general purpose programming languages they definitely have their strengths in different areas. Some are better for some tasks (e.g. compilers), other are better for others.
jwm4 7y ago
Prohibition was an extremist political movement. Widespread public rejection of it, in the form of stills, speakeasy and other means proved it to be a law without public sopport, resulting in its repeal within 10 years - an absolute public rejection of a limit unsought by the majority. OP pov has zero basis in fact.
- pkilgore 7y ago
  Am I.... Missing something?
libman 7y ago
Should have included Nim and D.
namelosw 7y ago
I'm surprised that OCaml takes 1.0-1.6x, given originally Rust compiler was prototyped in OCaml...
- dean177 7y ago
  It mentions that this is almost entirely due to interface files on ocaml’s part. They unfortunately introduce a fair amount of duplication.
  - vphantom 7y ago
    Luckily, they don't bring with them a lot of cognitive overhead for the developer, so their presence masks the expressiveness of OCaml in LOC stats IMHO. I use interface files to make public signatures explicit, abstract away some types and write thorough doc-comments. I'd be tempted to exclude them from such a comparison to better relate code size with programmer efficiency, especially when comparing to a language like Python. On the other hand, in languages like C we couldn't exclude header files because they include macros. (Although I suppose a fancier comparison could count macros and exclude function signatures.)
riquito 7y ago
Another big takeaway is the reusability of integration tests. Write once, verify implementations in any language. Static or dynamic typing doesn't matter that much if you can verify your software, and there are many different reasons to choose a particular programming language.
- mirekrusin 7y ago
  You need to look at the whole picture, time taken from day one to release, performance/correctness, then maintenance/onboarding, how easy is it to refactor/add features etc? It's virtually impossible to definatelly conclude in absolute terms that one language is superior than other. One of my best experiences was dovetailing that happened on a project that had to be c or c++ where I prototyped fully working system in ruby first, then rewritten it in c. Everything went almost too smoothly.
- butterthebuddha 7y ago
  Well, "[t]esting shows the presence, not the absence of bugs" (Djikstra).
foobar_ 7y ago
Rust seems so ugly ... makes Perl line noise look like paradise. I guess it's that time of the year ... when I have to learn an esoteric language to keep my sanity.
choeger 7y ago
First of all: amazing project. Kudos to your professor. Who is it, btw? Second: intriguing how important meta programming becomes once you nailed the problem domain.
wjw94 7y ago
My take-away from this article is that the university I graduated from was terrible
- remify 7y ago
  Same here ... I feel dumb now
  - kreetx 7y ago
    Now you both have "all the disadvantages for success" - go and make good use of this! :p
- davemp 7y ago
  I assume OP is in grad school. I took a few grad classes during my undergrad and the difference was night and day (teaching and cohort). So maybe your school was fine and you were just in the wrong classes. :)
  - trishume 7y ago
    Nope undergrad. UWaterloo just has a special co-op program that extends the degree and fits in six four-month internships in alternating school and work, for a total of two years of work experience by graduation.
Avi-D-coder 7y ago
Haskell without lens, text, vector, etc... is a bit like rust with only core not std. The haskell standard library is tiny. Libraries like lens are not optional. In practice you won't understand any open source Haskell without rudimentary understanding of lens. I get why parser libraries were banned, but excluding lens, vector, and text? I like Rust a lot, but haskell minus it's more advanced type system is just Rust plus GC. Lets not pretend this is a fair comparison of languages when it's primarily a comparison of standard libraries.
- sbergot 7y ago
  This is why I gave up on Haskell. Lens works as advertised, but is a pain to learn and to use in practice: the abstraction is tough to grasp and it is hard to form an intuition about it. The compilation errors are laughingly esoteric. The number of adhoc squwiggly operators is ridiculous. You also need to understand a lot of language extensions to get how the type checking works. To me it looks like an impressive proof of concept for a future programming language based around it. If I were to start a project with Haskell the use of lens would be explicitly forbidden.
  - dymk 7y ago
    It's about as esoteric as somebody learning C++ for the first time. And from that perspective, it's totally normal for errors or syntax to be weird looking for a long time. Most of us, including myself, are biased towards languages like Java, C, C++, Javascript, because those are what we learn first - and so our expectations of what errors (or syntax) look like are shaped by our early experiences. So I don't think it's fair to say that Haskell's compiler errors or quirks are fundamentally less intuitive than something that GCC/G++ spits out even on a sunny day. Just odd when we expect errors to look a particular way, but Haskell is playing a totally different (not exactly harder) game.
    - posterboy 7y ago
      We learn those first, that are not developed as a research platform that happens to have a little production use.
dennisgorelik 7y ago
That is an important conclusion: "design decisions make a much larger difference than the language". Software developers' expertise varies more than expressiveness of programming languages.
deng 7y ago
This is an interesting writeup, but I'd say the groups settling on different algorithms matters much more than the language used. It is obviously a pretty big difference whether you use a recursive descent or an LR parser.
julvo 7y ago
I like the idea of using this project to compare languages. However, to me it seems the comparison is more of a comparison between specific implementations rather than languages. The high variance in LOC between the two rust implementations makes me wonder if there's a similar variance for other languages and the samples we have lie somewhere between e.g. 0.5 and 2 times the average size for a given language. Therefore, the python implementation could just be a really compact one, even when compared to other python implementations. It could be interesting to ask the Professor if he'd be willing to collect some stats over the years.
zerr 7y ago
No AWK and Ada this time? :)
edem 7y ago
These comparisons are actively harmful. What the OP is doing is the same as if you were trying to determine the winner of a race by looking at the competitors at the start. You don't know their speed, their strengths etc. You can't determine how maintainable the code you produce this way, how it performs under real world usage, etc. Please stop doing these.
- jquery_dev 7y ago
  I very much disagree with your opinion. OP never tried to determine a winner, just wanted to share some insights into the project. If we go by what you say, we would never have any comparisons at all, as there are always unknowns.
iLemming 7y ago
It's too bad Clojure was not considered. I think it would be the most concise codebase and the most easiest to reason about.
PaulAJ 7y ago
I'm not clear whether the SLOC counts are for the application itself or including their test harnesses. It would be good to have a table of results with this data.
azov 7y ago
One big difference (which the article seems to mostly ignore) is that the shortest project was written by a single developer, while all others were a team effort. Having the whole project - design, architecture, all implementation details - in one head is not a trivial advantage. Even ignoring communication overhead, there might be subtle duplication of code simply because different people choose to do similar things in slightly different ways. While the ratio of Rust to Python code kind of matches my expectations, I wonder how much of it might be due to the difference in team structure vs the difference in chosen language.
- celrod 7y ago
  These were different teams of students taking a class. Variance in student effort/quality is generally high. There were two Rust teams. One had 3x the code and passed less tests than the other. This is our only reference for how much noise is owed to team rather than language; no other language was used by more than one team. Python did best (least amount of code, yet also the most features) less because of Python, but more because the best programmer was using it. Would the other students have been able to take advantage of the duck typing and metaprogramming in the same way, or ended up following different designs? I'd have my doubts about the second Rust team. Although that Python allowed these is a feature, but I think we're really just looking at noise.
  - gricjeh 7y ago
    And then again still, the “best programmer” sacrificed code quality in pursuit of quickly producing features. The article doesn’t define code quality measures but I’d still take the lines of code metric as either not a measure of value in itself or with a massive grain of salt.
    - firethief 7y ago
      That was the right tradeoff for the situation. I think any attempt to use this case study to draw conclusions about maintainability would be a mistake; that's just not what this is useful for.
tomerbd 7y ago
and how much time does it take to maintain this code?
pwpwp 7y ago
Is the code online somewhere?
carapace 7y ago
Ooo lolly! I want to do this in Prolog! The bit about "no parsing helpers even if they’re in the standard library" makes me wonder about using DCGs, but if disallowed re-implementation would be straightforward and add only a small constant to the code volume. I would like to see metrics on speed of compilation and size of resulting asm.
dpc_pw 7y ago
Interesting, but nothing to make any conclusions from. Too many variables to be of any use. Almost everything was different about these projects (different parsers, error handling, features, developer skill, team size).
karmakaze 7y ago
I've condensed the relative source code sizes and notes for quick ref: Rust_1 1.0 using recursive descent, visitor Haskell 1.0-1.6x depending on how you count for interesting reasons C++ 1.4x for mundane reasons Python 0.5x fancy metaprogramming, dynamic typing, single author Rust_2 3x different design decisions Scala 0.7x LR table generator, online Java grammar OCaml 1.0-1.6x depending on how you count, similar to Haskell > AST visitors and recursive descent [...] weren’t taught in the course
- karmakaze 7y ago
  Looking at this, my takeaways are different: Similar implementations in Rust(1) and C++ are roughly the same Similar implementations in Haskell and OCaml are roughly the same A similar implementation in Rust(2) to one in Haskell/OCaml is roughly 2x-3x Don't underestimate dynamic typing, metaprogramming, and individual effectiveness But for real world, long lived projects I still promote static typing with the exception of startup development (MVP/product validation, early iterations) of new product ideas.
deleted 7y ago
[deleted]

783 points by trishume 7y ago

Love reading your posts Tristan, a question about the choice of Rust. Was it specifically based on the previous experience you and your groupmates drew on or were these languages assigned to students? Was there any point in time where you and your team were trying to optimize your compiler?

trishume 7y ago
We got to choose. Everyone on our team had some experience in Rust and OCaml, the two of us more experienced in Rust were pretty indifferent between OCaml and Rust, and our third teammate wanted to use the project to learn more Rust so that's what we went with.

mruts 7y ago

I wonder how Racket would compare. I’ve written some DSLs at an old job for Racket and found the libraries for language design almost unreasonably powerful (and maybe unreasonably complicated as well). The dynamic typing would also help in reducing the amount of code.

trishume 7y ago
Yah I figure it would be like Python but better if you were allowed to use the language design libraries. However the course specifically forbids standard library components that are designed for implementing languages so that you have to write the parser and things yourself.
- deleted 7y ago
  [deleted]
- neilv 7y ago
  But you could use Racket if you wrote your own parser generator macro as part of the project's code? And would you be allowed to use Racket's existing facilities for representing and implementing macros? (You might want to use the macro stuff for your AST, IR, and transformations. Though you could still win with normal data types, but the macro stuff can help you win even more.)

spullara 7y ago

I'm sad that no one wrote it in the subset of Java that it was going to compile.

denormalfloat 7y ago
It's more surprising because javac is itself written in Java. It's definitely up to the task (even if it's targeting x86 instead).
- spullara 7y ago
  Maybe no one thought of the immense satisfaction they would get at the end when they compile their compiler to x86 code after compiling it with javac to bytecode to run it the first time.
mruts 7y ago
They probably wouldn't have been able to finish the project in time.
- ncmncm 7y ago
  And a good professor would have flunked them for poor judgment.
  - saagarjha 7y ago
    I doubt I will ever be a professor, but I would be happy to pass any student who did this provided they did the task adequately. Maybe extra points for cheekiness.
  - trishume 7y ago
    My professor was definitely a good professor and no way they would have done that. It clearly would have been done as an added challenge for themselves for fun and there's no reason to penalize that.
    - 7y ago

boyter 7y ago

I really wish when counting code that people would use one of the more modern code counters, or at least use cloc. Tokei, loc, scc, polyglot, loccount or gocloc give a much better idea over wc because they account for comments and blank lines and in the case of scc and tokei strings. I had a similar experience in university. Class had to implement a modified Turing machine. We could use whatever language we wanted. On person did it in C++ and it was several hundred lines. Another in Java which was slightly smaller. I implemented mine in Python and it was small enough to print on a single piece of paper. I think it was something like 30 lines or so. I exploited break statements to make it terse but readable. It did mean I was marked down a bit for them but it was by far the shortest program in the class.

trishume 7y ago
A few groups did give me output from a modern line counter, often Tokei or loc. I mention in the post that the ratios between source lines, word count lines and bytes were pretty similar across all projects. If people want to get a sense of how that compares to their intuitions in source lines they can use the ratio from our Rust project to convert. I ended up using raw lines for comparison because for relative measurement it didn't matter for the above reason, and I knew everyone had the same `wc` and I could get them to send me the output of the `wc {glob}` version that lists lines and bytes of all files so that I could drill down into which parts were different.
- boyter 7y ago
  Ah! I assumed you had access to the code. That makes more sense.
OnlyOneCannolo 7y ago
For anyone interested in this sort of thing, there's also Unified Code Count (UCC) [1]. It has a lot of interesting design goals like being open and explicit about the counting rules, which is really useful if you want to predict things like cost and reliability. [1] https://csse.usc.edu/ucc_new/wordpress/ https://csse.usc.edu/ucc_new/wordpress/

ufmace 7y ago

I haven't made it through the whole thing yet, but I do want to register a vote in favor of using Lines Of Code count as a rough measure of program complexity. I think it's a perfectly valid things to do, provided that it isn't used as an evaluation metric and nobody is gaming it, and everyone is a reasonably good programmer, not doing crazy things like trying to stuff a massive algorithm onto one line to be clever, or copy-pasting the same code in 20 places because they don't understand functions and classes.

chimpburger 7y ago
When comparing different languages it becomes more of a measure of the relative abstraction levels of languages. The exact same program will have hundreds (or thousands) of times more LOC in Assembly than in Python.
- applecrazy 7y ago
  Not only that, the LOC heuristic depends on the verbosity of the language. Java, for example, is more verbose, than say, Python, even though you’re close in abstraction level.
  - gpm 7y ago
    And just syntax decision. if (x) { y } else { z } if x: y else: z Assuming x and y are 1 line but long enough expressions you don't want to use a trinary operator (available in both) the python is 2/3rds the length of the c because of bracketing. Obviously cherry picked, but I bet these differences add up.
    - saagarjha 7y ago
      Of course, you can omit the braces if there’s a single statement (though I personally never do this).
      - 7y ago

jpittis 7y ago

> I think the smaller differences are also large enough to rule out extraordinary claims, like the ones I’ve read that say writing a compiler in Haskell takes less than half the code of C++ by virtue of the language Specifically the "by virtue of the language" part: Seems to me like it's unreasonable to claim the languages are on equal footing because fancy parser libraries aren't allowed to be used for the project. The fancy parser libraries exist for certain languages specifically because the languages enable them to be written. (For example in Haskell: monadic libaries, libraries that take advantage of GADTs, etc.)

trishume 7y ago
I don't think monadic parser libraries have a real claim to be that difference. All the languages listed have excellent parsing libraries that make things similarly easy, if not by language power than by grammar DSL with embeddable code snippets. I think if any library could make a real difference for Haskell it's most likely to be http://hackage.haskell.org/package/lens http://hackage.haskell.org/package/lens, which a Haskeller friend of mine claims could likely make a lot of the AST traversal and rewriting much terser.
- howenterprisey 7y ago
  Hoopl (data flow analysis) would also make a difference. I did a very similar project at my university in Haskell and Hoopl definitely saved us from writing quite a bit of code. We also used parser combinators in the frontend, which I think saved us time too.
- anaphor 7y ago
  I've found PEGs (Parsing Expression Grammars) to make things extremely easy and terse. E.g. OMeta, Parsley, etc. My experience with using both PEGs and parser combinators is that there isn't a huge difference in the total number of lines of code. On the other hand though, the syntax of PEGs would be easier to understand for someone who is familiar with BNF style notation.
- pwm 7y ago
  While I found your article informative and interesting I think it only works in the very specific context of this assignment. Disallowing powerful language features/libraries means it's not a level playing field and thus not a fair comparison. Some languages standard libraries are tiny some are huge. Some languages have lots of advanced features. Eg. GP mentioned GADTs with which one can write type safe/correct by construction ASTs. In other words programs passing specific tests in a specific context does not imply they are comparable in terms of general correctness/robustness/maintainability (as you noted this re caught edge cases).

kris-s 7y ago

Great writeup, I'm very curious about the performance between the compilers.

trimbo 7y ago

Where's the code? I scanned this article a few times and can't find a link to it.

trishume 7y ago
None of it is open source because it was done for a school course that uses the same project every term. The only use for having access to the code would be cheating on that project. The school gets mad when people publish the source for their school projects, especially when it isn't otherwise useful, that's also why my ray tracer isn't open source.
- empath75 7y ago
  I would not agree that that is the only use of that code. I’ve found a lot of school projects on GitHub when googling how to do stuff as a professional programmer and have found them useful quite often.
- wolfgangK 7y ago
  Being a teacher myself, I understand this policy. Too bad, though, I'd have dearly loved to read the python implementation to rewrite it in Clojure. I think the language could allow for something at least as concise be maybe cleaner.

sonnyblarney 7y ago

"Their project was 17,211 raw lines, 15k source lines, and 637kb" So a couple of students cranked out 15K lines of code, in basically a subset of their study time, for a single class? I don't doubt they did it, what I doubt is the quality of what they wrote. Because it's 'just a project' it doesn't have to be fully debugged ... But that is a vast amount of code to hand-write over a short period of time. At that level, I think it's basically just 'code like the wind' with not much consideration for whether it works, architecture etc. etc. - which in a way might put the conclusions of the 'mini study' at odds. At 15K LOC dashed out very quickly ... these are not products, they're comparing 'rapidly written out lines of code', which is something else entirely from working code. Also, given that everyone is in Uni, and likely may not have been exposed to proper idiomatic code for the language ... this presents another issue. Ideally we'd want to compare reasonably idiomatic code for one language, to another. Kudos to the author for this, but we should be aware of some caveats.

trishume 7y ago
Note that project is the outlier for how much code it contained. That team did have the most trouble getting things done on time and fixing all the bugs, so they passed the least tests, as one would expect given how much code their design choices required them to write. I think the causality goes from their design to the line count, not from the line count to worse design, they had to write lots of code to implement their more abstract design choices. I looked at their code and didn't see any obvious deficiencies in anything other than the design. Also note that UWaterloo's CS and SE programs are different than other universities. Everyone on all teams had at least two years of full time work experience at 6 internships. The people I talked to were also programming enthusiasts who read online a lot. The only teams that may not have written the most idiomatic code are the person who used Python alone, and the Haskell team because I've heard idiomatic high-end Haskell involves an insane amount of knowledge of abstractions like lens, way more than all the other languages. This compilers class is also well known for requiring lots of work, so students often arrange their schedules to have a low load from other classes while taking it, which is possible since the CS program requirements have a lot of flexibility.
- 7y ago

akhilcacharya 7y ago

Loved this part and thought it was hysterical - > Since my team had all interned at Jane Street the other language we considered using was OCaml, we decided on Rust but I was curious about how OCaml might have turned out so I talked to someone else I knew had interned at Jane Street and they indeed did their compiler in OCaml with two other former Jane Street interns. Relatedly this is why I get annoyed at people that don’t believe 10x or Nx developers exist - they absolutely do!

dudul 7y ago
Can you expand a little? I fail to see the connection between the quote and your conclusion.
- akhilcacharya 7y ago
  Jane Street only interviews and hires really exceptional engineers. A team of exceptional engineers in a single CS class, even if it is at a school like Waterloo, would imply that the 10x phenomenon isn't a myth but a somewhat mundane reality in certain circles.
ncmncm 7y ago
In the 90's, the Swedish Ericsson and German Siemens companies participated in a joint (crash) project. Each sent 250 engineers. After six months, they delivered. A friend counted up lines of code written by each engineer, at the end. Fully half the code in the final product was written by one person: N lines by him, N lines by the other 499 people. He was a lead programmer, who issued two-week assignments to other engineers; if one wasn't done come Friday, he would do it himself on Saturday. He doesn't consider himself especially fast, because he knows someone else who codes ten times as fast, and wears out two keyboards per year. This project was also interesting because they took blood samples, by which they got objective measurements of stress. Everyone's stress level increased right up to the deadline -- except his. And their stress levels did not start falling until many months after.
- madez 7y ago

awwaiid 7y ago

If you're into this sort of thing, definitely check out the ICFP Programming Contest -- take next Friday off and join in on the fun! https://icfpcontest2019.github.io/ https://icfpcontest2019.github.io/

saagarjha 7y ago
> Any programming language(s) on any platform(s) may be used. How does this work? What’s stopping me from using some hyperpowered esolang I cooked up for the competition?
- deleted 7y ago
  [deleted]
- krick 7y ago
  Nothing, if it helps you with the challenge.
- dbpatterson 7y ago
  That the task description is only posted at the start of the competition?
  - geofft 7y ago
    I mean, if you design an esolang for the competition task and implement an optimizing compiler that's faster than any general purpose language out there, that absolutely feels like something that should be rewarded by the spirit of the competition! So it's okay that it's allowed by the letter of the competition, too.
- exsf0859 7y ago
  If I recall correctly, the 2001 ICFP contest was won by someone who happened to have done their thesis on the contest problem. :-)
lukego 7y ago
Has anybody made a study of the corpus of ICFP contest entries? Seems like an interesting dataset with many languages, many programmers competing multiple times, etc.

mamcx 7y ago

Is possible to see the code of this project? And is the same of http://thume.ca/2019/04/18/writing-a-compiler-in-rust/ http://thume.ca/2019/04/18/writing-a-compiler-in-rust/? P.D: I'm building a relational language and my ideas mimic closely "writing-a-compiler-in-rust" ie: using pratt parsers and similar, but not see much of how do that on rust...

ncmncm 7y ago

I find it curious that both the team that used C++, and the author, both appear to believe that sum types and related facilities are not usable or conveniently expressible in C++. I got that Boost Spirit, a powerful parsing library, was forbidden. Were all the Boost libraries similarly embargoed?

trishume 7y ago
There was a general prohibition on libraries that don't ship with the compiler, but you could ask the professor for an exception. Someone asked about Boost and a blanket exception for the non-parsing and scanning parts of Boost was granted. However the C++ team I talked to didn't use it. Also note that it's not only the sum types that are important, powerful pattern matching facilities are part of what makes them so valuable.
- shaklee3 7y ago
  C++ has had sum types for years, so I'm not sure why that was mentioned. If you aren't familiar/experienced with the language, then the comparison is not fair overall.
  - jcora 7y ago
    Yeah it's had everything for years. I'd rather shoot myself than use those features outside of a modern language where they aren't just bolted on along a billion other things.
    - shaklee3 7y ago
      That's not a useful comment. Would you rather they just kill the language instead of improving it?
      - jcora 7y ago
        Absolutely. Killing it would be the most useful improvement.

bfung 7y ago

Excellently written, great topic, and done w/o flaming/too much bias. I'm amazed as many older folks in the industry would not be able to have this level of content and maturity to write an informative article.

WalterBright 7y ago
It's difficult to write a fair comparison without being a fairly competent programmer in each of the languages. The trouble is, if a person is an expert C programmer and then translates it to Python that he's only modestly familiar with, the Python program will look like C. It won't be idiomatic Python. For example, my early Fortran programs looked like Basic. My early C programs looked like Fortran. My early C++ programs looked like C. And my early D code looked like C++. It takes much more than being able to write a program in X to be able to write one that makes proper use of the language.
- galaxyLogic 7y ago
  That is all true. At the same time here the groups were allowed to use the language of their choice. Presumably they chose languages they felt they were competent in. Of course an expert of a given programming language can write much better in it than a novice. But a comparison like this is not necessarily about comparing top-programmers in every language, but average programmers, because we want to know results that are true "on average" . The author does note he "knew (they) were highly competent". So they were not exactly novices in their language of choice. Writing a compiler is not a task for novices in general.
  - anaphor 7y ago
    The article says that the instructor for the course cautioned against using Haskell because some people overestimated their competency with it. I think it is actually very likely that people would chose a programming language or system for reasons other than how competent they are with it. E.g. to seem "smart" because you wrote your compiler in Haskell, even though you actually have much more experience with Java or Python. FTA: > Another interesting thing to note is that at the start of every offering of the course the professor says that students can use any language that can run on the school servers, but issues a warning that teams using Haskell have the highest variance in mark of any language, with many teams using Haskell overestimating their ability and crashing and burning then getting a terrible mark, more than any other language, while some Haskell teams do quite well and get perfect like my friends.

galaxyLogic 7y ago

Prolog, and Definite-Clause-Grammars https://en.wikipedia.org/wiki/Definite_clause_grammar https://en.wikipedia.org/wiki/Definite_clause_grammar

nautilus12 7y ago

I can hear all the Haskell fanboys screaming that they didn't do Haskell right because a kleisli arrow didn't appear and ascend them into monadic heaven.

aetherspawn 7y ago
To be fair, I don’t think any Haskeller would actually write this project without attoparsec and maybe lens by choice. They’re basically base libraries for this sort of thing.
- nybble41 7y ago
  Agreed. The rule about not using any library that doesn't ship with the compiler creates a heavy bias in favor of "batteries included" languages like Python. I understand not allowing parsing libraries, given the nature of the assignment, but if even widely used libraries like lens are off-limits the result is not going to resemble idiomatic Haskell. As a matter of basic fairness, if some library or built-in language feature is to be permitted for one implementation then analogous libraries should be permitted for all the other implementations regardless of whether they happen to ship with the compiler. The eval function leveraged heavily by the Python version should probably also have been off-limits, for the same reason that parsing libraries were prohibited. Using eval amounts to embedding an existing fully-developed parser and runtime environment into the project as a library.
  - dymk 7y ago
    I think a much more interesting comparison would be to ask an expert in each one of these languages to implement a "production" ready version of the project - so allowing popular, stable, community-accepted libraries to be used. Then we'd see what truly idiomatic solutions to this look like.

FullyFunctional 7y ago

That was a really interesting article. I completely agree with >>> Abstractions may make things easier to extend in the future, or guard against certain types of errors, but they need to be considered against the fact that you may end up with 3 times the amount of code to understand and refactor, 3 times the amount of possible locations for bugs and less time left to spend on testing and further development. Choosing when and how to abstract is key. Abstraction in a fashion extends the language so you are trading off a burden on the reader (who have to learn and trust the abstraction) for increased expressiveness. Don't overdo it. (And don't abstract idioms).

mhh__ 7y ago
I think thinking carefully about the abstractions is also key, ideally not while at the computer. Good abstractions should be obvious and not really requiring refactoring as per se.
rjayatilleka 7y ago
Could you clarify what you mean by "don't abstract idioms?"
- FullyFunctional 7y ago
  One example (taken from the real world): addr = (addr + PAGESIZE - 1) & ~PAGESIZE; contrast with addr = P2ALIGN(addr); The former is idiomatic and immediately readable. For the latter I have to go lookup the definition of P2ALIGN (and make sure I got the one actually used as there may be multiple!), check that it does what I expect, and memorize it. The value of abstracting the idiom is debatable. If it _is_ used sufficiently often, then it might be worthwhile, but often it isn't.

galaxyLogic 7y ago

I think the big big result from this study is: "Python: half the size !". A dynamic language like Python is better here, 2x better. I assume similar results would apply to other dynamic languages like JavaScript, Lisp, Smalltalk, Groovy etc. This does not say that static typing should not be used but I think it shows unequivocally that there is a considerable extra development cost associated with static typing. You might say that surely that additional cost would be compensated in reducing the cost of maintenance later. Maybe but I'm not sure. Any development effort of significant size (like writing a compiler here) is a combination of writing new code and adapting and modifying code already written. "Maintenance" is part of development. This is quite a quantitative study which gives credence to the claims of advocates of dynamic languages.

trishume 7y ago
I agree it's definitely an interesting result and a point in favour of dynamic languages. A caveat is that I'm pretty sure my friend intentionally sacrificed code quality to do it, I don't think you'd find that project as readable and understandable as the others. Another caveat is that you have to be okay with your codebase being extremely magical and metaprogramming-heavy, which many industrial users of dynamic languages avoid. As I mention, I'm personally into statically typed languages mostly for the performance and correctness benefits. I think it's plausible that on larger projects with teams the correctness benefits save enough debugging time that overall implementation time is lower, but I'm less confident in this now than I was before this comparison.
- galaxyLogic 7y ago
  I've read my share of cryptic JavaScript written by others and in that sense I agree that in multi-person long-term projects static typing no doubt will have its advantages. My hunch is however that what is often overlooked in development with statically typed languages is that it takes considerable time and effort to come up with the right set of types. Many examples are written showing how types almost magically make programs easier to understand. But when you read such an example what is not stated is how much effort it took to come up with just those types. One way of thinking about it is that type-definitions are really a "second program" you must write. They check upon the primary program and validate it. But that means you must write that second program as well. It's like building an unsinkable ship with two hulls one inside the other. The quality will be great but it does cost more.

stretchwithme 7y ago

It would be interesting to see experts in different technologies compete, in front of a live audience, to implement or change a variety of projects. It would be useful to see what a master can do with each tool or platform.

galaxyLogic 7y ago
It would be interesting and educational to see how they would do it, definitely. But when comparing languages it makes sense to compare how well average people perform with them. We can't always hire an expert they are not universally available.
- stretchwithme 7y ago
  It would certainly help people decide what tobecome an expert in. And there must be some correlation with what an expert and an amateur can do. Of course, I can think of examples where that isn't true.

analog31 7y ago

I conducted a similar exercise, albeit on a much smaller scale, when I was "language shopping." I took a project that I had written in Visual Basic (my go-to language at the time), and re-wrote it in a variety of programming environments. The project was just complex enough to require maybe 100-200 lines of code at the end of the day, thanks to liberal use of libraries. Not huge, but not atypical for a "scientific" programmer who uses code as a way to solve problems rather than to create software for others to use. It's pretty representative of my life as a programmer. The exercise forced me to learn enough of each language to get a feel for it, and then I could look at the programs alongside one another to assess their strengths. I also imposed some rules, such as that the language had to run on multiple platforms, and be FOSS. In addition to comparing the languages, I was also implicitly comparing libraries and even access to online help. This was also my first real exposure to StackOverflow. I tested Javascript, Python, GNU Octave, and wxMaxima. Ultimately Python won out and is my language of choice today. This was just my own little exercise, and not worth publishing, but has made me a believer in learning multiple languages.

new4thaccount 7y ago
Give Julia a try if you haven't. If you're coming from Python and also have some Octave experience, it will be easy and worth a look. For scientific coding I usually use either Python or Julia and Fortran or Octave at home sometimes.

anaphor 7y ago

I think the right metric to measure here is not quantitative, but more qualitative. In other words, how easy it is to adapt the language to the problem you are trying to solve. So, in order to do that you need to have a good understanding of the principles each language is based on. Once you've got that, then you look at the resulting code and "measure" how easy it is to understand. As was already said, this requires having someone with a very good understanding of the underpinnings of each language, which is not really going to be reasonable for most people. The problem is that if you simply tried to port an idiomatic solution from e.g. C++ or Python into Haskell or Scala, or whatever, then you would probably end up with something very ugly, because you didn't adapt the language to your problem. You tried to force it to do something it wasn't necessarily designed to do.

Buttons840 7y ago
A contest to see how easily code could be broken would be interesting. "I purposely broke the code somewhere in these 20 lines. Find the bug and fix it." You can use the compiler, you can use the test suite. You cannot diff the code against the original working code.

huxingyi 7y ago

The actual effort of writing one line of code in the specific language doesn’t mention in the article. Write one line of rust is extremely difficult than one line of Python, or even C++.

zimablue 7y ago

I feel like what you're actually testing with this is "amongst the top percentile of programmers, what are the proclivities of people by choice of favourite language" Because the implementations vary so much, that's the source of a lot of the LOC-difference. They effectively delivered more or less (if you count more stages as "more" which I would as it will make it easier to reason about/debug, and count a typesystem as "more", since it gives you more guarantees). python - solves the problem brutally fast, some ugly shortcuts haskell - solves the problem quite slowly/delivered the most rust/c++ - intermediate scala - solved fast, took shortcuts ocaml - this is the one that surprised me I'd have expected it to be the shortest with python

dean177 7y ago
Why would you expect the ocaml one to be shorter?

tanin 7y ago

The Scala result confirms my bias :P I love Scala because it strikes a great balance between being succinct and offering type safety. I wish you would use Ruby instead of Python. Python is strangely inconsistent. For example, Python doesn't really offer a rich standard library; people have to resort to ugly solutions for a simple problem (here's an example: https://stackoverflow.com/questions/363944/python-idiom-to-return-first-item-or-none https://stackoverflow.com/questions/363944/python-idiom-to-r...) Ruby would be better at an example of how a dynamic language can be more succinct.

trishume 7y ago
Preaching to the choir, I personally use Ruby over Python :) But I had to source from friends in the class, and my group used Rust because I'm all about that performance and correctness nowadays.
nhumrich 7y ago
"python doesnt offer a rich standard library" Python is usually considered one of the richest standard libraries out there. Yes, the is no standard way to get this one edgecase, but every language has warts and things like that. It sounds like you are just looking for an excuse to hate on Python.
- tanin 7y ago
  > but every language has warts and things like that Sure, but we are talking about the brevity of a language. The richness of the standard library directly impacts brevity. So, Ruby would have been a better representative (for brevity) of a high-level dynamic-typed language. > the is no standard way to get this one edgecase I don't think `.first` is an edge case. It is used fairly often. One example that I can think of right now is when you want to fetch the first row from MySQL. MySQL returns an array, and you would need `.first` to get the first row or null. > It sounds like you are just looking for an excuse to hate on Python Not at all. While I don't prefer Python, I recognize there's a downside to a richer standard library. The language becomes more complex; harder to learn. Richest is relative. If we consider Python against every other programming language, then, yes, it's ONE of the richests. If we consider Python vs. Ruby vs. Scala, I doubt Python would be considered as richer or richest. For example, Scala offers a lot of ways to process a list like foldLeft, foldRight, unzip, headOption, lastOption, flatMap, groupBy, and many methods around Map, Set, and etc. There are many examples where Ruby version would result in shorter code like `array.delete_if` and etc. Python probably doesn't offer many of these methods.

Scarbutt 7y ago

How much time was given for this task?

chenglou 7y ago

Hey trishume! It’s been a while (this is the Reason person). Great post; it echoes all the experiences I/we’ve personally had, especially the alternative Rust section: on the spectrum of how much intelligence one needs to understand fancy abstractions, all of us programmers, good and bad, are pretty much lumped to the lower end. I’ve seen plenty of skilled folks “decompensate” their expertise by reaching for overly complex abstractions. The definition of a “strong” programmer, as per the post and usually in real world, should probably be rectified to include not only “the person knows how to use this abstraction”, but also “the person knows when to stop”. In the same vein of idea, it’d be interesting to know how Go would fare in this project. Go’s philosophy is pretty much the opposite of a language that’s usually used for compiler research; but I’ve seen indicators that using it could be surprisingly effective (and in light of this post, maybe less surprising). More importantly, it’d be nice to know the perf characteristics of each project =)

trishume 7y ago
I expect Go would end up like C++ without header files or OCaml without sum types and pattern matching, falling somewhere in between them. Although if errors are handled properly instead of by panic then it might really hurt in the line count from all the `if err != nil { return; }` Also I commented somewhere else in this thread re perf comparison. Short answer is that since there's no incentive towards performance the signal would be swamped by differences in how much people avoid O(n^2) algorithms even if you don't need to.
Matthias247 7y ago
> In the same vein of idea, it’d be interesting to know how Go would fare in this project. I would say a bit worse than the mentioned alternatives, which all have better type systems and thereby e.g. make it easier represent and manipulate the trees that are everywhere in compilers. But most likely it's still fine, and the line count metric would be more influenced by the fact how experienced the author is than the language. Things will get worse in Go if one wants not write a lexer/parser manually but use tooling for that. Parser combinators and other tools benefit a lot from generics. On the other hand e.g. writing a network server with decent performance will be in Go a lot easier and more straightforward than in any of the other mentioned languages. While all those languages are general purpose programming languages they definitely have their strengths in different areas. Some are better for some tasks (e.g. compilers), other are better for others.

jwm4 7y ago

Prohibition was an extremist political movement. Widespread public rejection of it, in the form of stills, speakeasy and other means proved it to be a law without public sopport, resulting in its repeal within 10 years - an absolute public rejection of a limit unsought by the majority. OP pov has zero basis in fact.

pkilgore 7y ago
Am I.... Missing something?

libman 7y ago

Should have included Nim and D.

namelosw 7y ago

I'm surprised that OCaml takes 1.0-1.6x, given originally Rust compiler was prototyped in OCaml...

dean177 7y ago
It mentions that this is almost entirely due to interface files on ocaml’s part. They unfortunately introduce a fair amount of duplication.
- vphantom 7y ago
  Luckily, they don't bring with them a lot of cognitive overhead for the developer, so their presence masks the expressiveness of OCaml in LOC stats IMHO. I use interface files to make public signatures explicit, abstract away some types and write thorough doc-comments. I'd be tempted to exclude them from such a comparison to better relate code size with programmer efficiency, especially when comparing to a language like Python. On the other hand, in languages like C we couldn't exclude header files because they include macros. (Although I suppose a fancier comparison could count macros and exclude function signatures.)

riquito 7y ago

Another big takeaway is the reusability of integration tests. Write once, verify implementations in any language. Static or dynamic typing doesn't matter that much if you can verify your software, and there are many different reasons to choose a particular programming language.

mirekrusin 7y ago
You need to look at the whole picture, time taken from day one to release, performance/correctness, then maintenance/onboarding, how easy is it to refactor/add features etc? It's virtually impossible to definatelly conclude in absolute terms that one language is superior than other. One of my best experiences was dovetailing that happened on a project that had to be c or c++ where I prototyped fully working system in ruby first, then rewritten it in c. Everything went almost too smoothly.
butterthebuddha 7y ago
Well, "[t]esting shows the presence, not the absence of bugs" (Djikstra).

foobar_ 7y ago

Rust seems so ugly ... makes Perl line noise look like paradise. I guess it's that time of the year ... when I have to learn an esoteric language to keep my sanity.

choeger 7y ago

First of all: amazing project. Kudos to your professor. Who is it, btw? Second: intriguing how important meta programming becomes once you nailed the problem domain.

wjw94 7y ago

My take-away from this article is that the university I graduated from was terrible

remify 7y ago
Same here ... I feel dumb now
- kreetx 7y ago
  Now you both have "all the disadvantages for success" - go and make good use of this! :p
davemp 7y ago
I assume OP is in grad school. I took a few grad classes during my undergrad and the difference was night and day (teaching and cohort). So maybe your school was fine and you were just in the wrong classes. :)
- trishume 7y ago
  Nope undergrad. UWaterloo just has a special co-op program that extends the degree and fits in six four-month internships in alternating school and work, for a total of two years of work experience by graduation.

Avi-D-coder 7y ago

Haskell without lens, text, vector, etc... is a bit like rust with only core not std. The haskell standard library is tiny. Libraries like lens are not optional. In practice you won't understand any open source Haskell without rudimentary understanding of lens. I get why parser libraries were banned, but excluding lens, vector, and text? I like Rust a lot, but haskell minus it's more advanced type system is just Rust plus GC. Lets not pretend this is a fair comparison of languages when it's primarily a comparison of standard libraries.

sbergot 7y ago
This is why I gave up on Haskell. Lens works as advertised, but is a pain to learn and to use in practice: the abstraction is tough to grasp and it is hard to form an intuition about it. The compilation errors are laughingly esoteric. The number of adhoc squwiggly operators is ridiculous. You also need to understand a lot of language extensions to get how the type checking works. To me it looks like an impressive proof of concept for a future programming language based around it. If I were to start a project with Haskell the use of lens would be explicitly forbidden.
- dymk 7y ago
  It's about as esoteric as somebody learning C++ for the first time. And from that perspective, it's totally normal for errors or syntax to be weird looking for a long time. Most of us, including myself, are biased towards languages like Java, C, C++, Javascript, because those are what we learn first - and so our expectations of what errors (or syntax) look like are shaped by our early experiences. So I don't think it's fair to say that Haskell's compiler errors or quirks are fundamentally less intuitive than something that GCC/G++ spits out even on a sunny day. Just odd when we expect errors to look a particular way, but Haskell is playing a totally different (not exactly harder) game.
  - posterboy 7y ago
    We learn those first, that are not developed as a research platform that happens to have a little production use.

dennisgorelik 7y ago

That is an important conclusion: "design decisions make a much larger difference than the language". Software developers' expertise varies more than expressiveness of programming languages.

deng 7y ago

This is an interesting writeup, but I'd say the groups settling on different algorithms matters much more than the language used. It is obviously a pretty big difference whether you use a recursive descent or an LR parser.

julvo 7y ago

I like the idea of using this project to compare languages. However, to me it seems the comparison is more of a comparison between specific implementations rather than languages. The high variance in LOC between the two rust implementations makes me wonder if there's a similar variance for other languages and the samples we have lie somewhere between e.g. 0.5 and 2 times the average size for a given language. Therefore, the python implementation could just be a really compact one, even when compared to other python implementations. It could be interesting to ask the Professor if he'd be willing to collect some stats over the years.

zerr 7y ago

No AWK and Ada this time? :)

edem 7y ago

These comparisons are actively harmful. What the OP is doing is the same as if you were trying to determine the winner of a race by looking at the competitors at the start. You don't know their speed, their strengths etc. You can't determine how maintainable the code you produce this way, how it performs under real world usage, etc. Please stop doing these.

jquery_dev 7y ago
I very much disagree with your opinion. OP never tried to determine a winner, just wanted to share some insights into the project. If we go by what you say, we would never have any comparisons at all, as there are always unknowns.

iLemming 7y ago

It's too bad Clojure was not considered. I think it would be the most concise codebase and the most easiest to reason about.

PaulAJ 7y ago

I'm not clear whether the SLOC counts are for the application itself or including their test harnesses. It would be good to have a table of results with this data.

azov 7y ago

One big difference (which the article seems to mostly ignore) is that the shortest project was written by a single developer, while all others were a team effort. Having the whole project - design, architecture, all implementation details - in one head is not a trivial advantage. Even ignoring communication overhead, there might be subtle duplication of code simply because different people choose to do similar things in slightly different ways. While the ratio of Rust to Python code kind of matches my expectations, I wonder how much of it might be due to the difference in team structure vs the difference in chosen language.

celrod 7y ago
These were different teams of students taking a class. Variance in student effort/quality is generally high. There were two Rust teams. One had 3x the code and passed less tests than the other. This is our only reference for how much noise is owed to team rather than language; no other language was used by more than one team. Python did best (least amount of code, yet also the most features) less because of Python, but more because the best programmer was using it. Would the other students have been able to take advantage of the duck typing and metaprogramming in the same way, or ended up following different designs? I'd have my doubts about the second Rust team. Although that Python allowed these is a feature, but I think we're really just looking at noise.
- gricjeh 7y ago
  And then again still, the “best programmer” sacrificed code quality in pursuit of quickly producing features. The article doesn’t define code quality measures but I’d still take the lines of code metric as either not a measure of value in itself or with a massive grain of salt.
  - firethief 7y ago
    That was the right tradeoff for the situation. I think any attempt to use this case study to draw conclusions about maintainability would be a mistake; that's just not what this is useful for.

tomerbd 7y ago

and how much time does it take to maintain this code?

pwpwp 7y ago

Is the code online somewhere?

carapace 7y ago

Ooo lolly! I want to do this in Prolog! The bit about "no parsing helpers even if they’re in the standard library" makes me wonder about using DCGs, but if disallowed re-implementation would be straightforward and add only a small constant to the code volume. I would like to see metrics on speed of compilation and size of resulting asm.

dpc_pw 7y ago

Interesting, but nothing to make any conclusions from. Too many variables to be of any use. Almost everything was different about these projects (different parsers, error handling, features, developer skill, team size).

karmakaze 7y ago

I've condensed the relative source code sizes and notes for quick ref: Rust_1 1.0 using recursive descent, visitor Haskell 1.0-1.6x depending on how you count for interesting reasons C++ 1.4x for mundane reasons Python 0.5x fancy metaprogramming, dynamic typing, single author Rust_2 3x different design decisions Scala 0.7x LR table generator, online Java grammar OCaml 1.0-1.6x depending on how you count, similar to Haskell > AST visitors and recursive descent [...] weren’t taught in the course

karmakaze 7y ago
Looking at this, my takeaways are different: Similar implementations in Rust(1) and C++ are roughly the same Similar implementations in Haskell and OCaml are roughly the same A similar implementation in Rust(2) to one in Haskell/OCaml is roughly 2x-3x Don't underestimate dynamic typing, metaprogramming, and individual effectiveness But for real world, long lived projects I still promote static typing with the exception of startup development (MVP/product validation, early iterations) of new product ideas.

deleted 7y ago

[deleted]