4 ms·

Memory Access Microbenchmark: Clojure, F#, Go, Julia, Lua, Racket, Ruby, Self

19 points by logicchains 13y ago

lqdc13 13y ago
The Python code is terrible. That's not how you code in Python. Also, this implementation seems to take more than 32 GB on my machine. Here is a refactor: https://gist.github.com/lqdc/9149772 https://gist.github.com/lqdc/9149772 Anyway, this problem should not be solved this way in python either way. I would have stored everything in a Pandas table and computed stuff from there.
- logicchains 13y ago
  Thanks! I'm rerunning it and updating the tables progressively every few hours as optimisations are suggested. Feel free to make a pull request. Ideally within a day or so the implementations will have all been optimised enough to make for fairer comparison. I understand that a Pandas table would be a better idea, but the purpose of this benchmark is comparing the speed of the raw languages.
  - lqdc13 13y ago
    Also there is integer overflow in the C code and maybe others because the sum doesn't fit in 64 bit integer.
    - logicchains 13y ago
      It mentions that later in the post. The implementations in the final two tables use bignums for fairness and to avoid overflow.
  - lqdc13 13y ago
    I updated with a pandas script as well. I think the strength of the languages have in part to do with their libraries. You have to pick the right tools for the right task. The pandas version runs 3x faster than regular python, but 17x slower than C.
    - logicchains 13y ago
      Great, I'll include that in the next run of the benchmark. Is it Python 3 or 2? (Does it work with Pypy) Also, if you're comparing it to C you should compare it to C3.c, from the final tables, which uses bignums like Python does automatically. *Edit: I tried your faster_py.py, but it didn't seem any faster than Pypy2.py in the repo (running both with Pypy). I haven't yet got the Pandas version to work due to library compatibility issues (I've got about five versions of Python installed; still working on it).
      - lqdc13 13y ago
        It's about 2x faster in regular CPython 2.7 on my machine. In pypy it is actually way slower than your original one. In many cases, pypy cannot be used, because it's not compatible with all the libraries.
        logicchains 13y ago
        Ah, right. I've included your version (converted to Python 3) as Python3_fst; it's around twice as fast as regular CPython 3.3.
mattchamb 13y ago
I had a play around with your F# implementation so its abit cleaner and uses abit more fsharp-ish style: https://gist.github.com/mattchamb/20019b22ae841ff5ce1b https://gist.github.com/mattchamb/20019b22ae841ff5ce1b Im not sure if it is any faster though; as each run varies quite widely from the previous one
- logicchains 13y ago
  Interesting, thanks. It might be faster than FS.fs but not FS2.fs, as I think: tradesArray.[i] <- { TradeId = idx; ClientId = idx; ... } would lead to the creation of a new object every time whereas: trades.[i].TradeId <- (int64 i) trades.[i].ClientId <- (int64 1) etc. doesn't.
  - mattchamb 13y ago
    Yeah, thats where I had some problems understanding the intention of the test. In my edited one, the trade object is immutable, so you could just reuse the same object without editing it. In FS.fs, a whole new set of objects is created/allocated on the heap for each test iteration; whereas in FS2.fs, pre-allocated memory is just being edited in place. The first approach seems much more likely to create GC pressure than the second approach. Similarly, there are a few differences between the implementation in the different languages. For example, in CS.cs, the entire trades array is being re-allocated in each iteration of the test, whereas in the F# tests, it is allocated up front. Changing the array allocation to once only in the C# implementation caused a large speedup (from about 5 secs down to about 1.5 secs on my machine). Speaking of the C# implementation, here is a slightly cleaned up version with the altered array allocation: https://gist.github.com/mattchamb/9152487 https://gist.github.com/mattchamb/9152487 As a side note, I changed it from using DateTime.Now to using the Stopwatch class. The reason for this is that DateTime.Now has limited timer resolution (about 10ms), which you can see in the remarks section of the msdn docs (http://msdn.microsoft.com/en-us/library/system.datetime.now(v=vs.110).aspx http://msdn.microsoft.com/en-us/library/system.datetime.now(...)
    - mattchamb 13y ago

19 points by logicchains 13y ago

The Python code is terrible. That's not how you code in Python. Also, this implementation seems to take more than 32 GB on my machine. Here is a refactor: https://gist.github.com/lqdc/9149772 https://gist.github.com/lqdc/9149772 Anyway, this problem should not be solved this way in python either way. I would have stored everything in a Pandas table and computed stuff from there.

logicchains 13y ago
Thanks! I'm rerunning it and updating the tables progressively every few hours as optimisations are suggested. Feel free to make a pull request. Ideally within a day or so the implementations will have all been optimised enough to make for fairer comparison. I understand that a Pandas table would be a better idea, but the purpose of this benchmark is comparing the speed of the raw languages.
- lqdc13 13y ago
  Also there is integer overflow in the C code and maybe others because the sum doesn't fit in 64 bit integer.
  - logicchains 13y ago
    It mentions that later in the post. The implementations in the final two tables use bignums for fairness and to avoid overflow.
- lqdc13 13y ago
  I updated with a pandas script as well. I think the strength of the languages have in part to do with their libraries. You have to pick the right tools for the right task. The pandas version runs 3x faster than regular python, but 17x slower than C.
  - logicchains 13y ago
    Great, I'll include that in the next run of the benchmark. Is it Python 3 or 2? (Does it work with Pypy) Also, if you're comparing it to C you should compare it to C3.c, from the final tables, which uses bignums like Python does automatically. *Edit: I tried your faster_py.py, but it didn't seem any faster than Pypy2.py in the repo (running both with Pypy). I haven't yet got the Pandas version to work due to library compatibility issues (I've got about five versions of Python installed; still working on it).
    - lqdc13 13y ago
      It's about 2x faster in regular CPython 2.7 on my machine. In pypy it is actually way slower than your original one. In many cases, pypy cannot be used, because it's not compatible with all the libraries.
      - logicchains 13y ago
        Ah, right. I've included your version (converted to Python 3) as Python3_fst; it's around twice as fast as regular CPython 3.3.

mattchamb 13y ago

I had a play around with your F# implementation so its abit cleaner and uses abit more fsharp-ish style: https://gist.github.com/mattchamb/20019b22ae841ff5ce1b https://gist.github.com/mattchamb/20019b22ae841ff5ce1b Im not sure if it is any faster though; as each run varies quite widely from the previous one

logicchains 13y ago
Interesting, thanks. It might be faster than FS.fs but not FS2.fs, as I think: tradesArray.[i] <- { TradeId = idx; ClientId = idx; ... } would lead to the creation of a new object every time whereas: trades.[i].TradeId <- (int64 i) trades.[i].ClientId <- (int64 1) etc. doesn't.
- mattchamb 13y ago
  Yeah, thats where I had some problems understanding the intention of the test. In my edited one, the trade object is immutable, so you could just reuse the same object without editing it. In FS.fs, a whole new set of objects is created/allocated on the heap for each test iteration; whereas in FS2.fs, pre-allocated memory is just being edited in place. The first approach seems much more likely to create GC pressure than the second approach. Similarly, there are a few differences between the implementation in the different languages. For example, in CS.cs, the entire trades array is being re-allocated in each iteration of the test, whereas in the F# tests, it is allocated up front. Changing the array allocation to once only in the C# implementation caused a large speedup (from about 5 secs down to about 1.5 secs on my machine). Speaking of the C# implementation, here is a slightly cleaned up version with the altered array allocation: https://gist.github.com/mattchamb/9152487 https://gist.github.com/mattchamb/9152487 As a side note, I changed it from using DateTime.Now to using the Stopwatch class. The reason for this is that DateTime.Now has limited timer resolution (about 10ms), which you can see in the remarks section of the msdn docs (http://msdn.microsoft.com/en-us/library/system.datetime.now(v=vs.110).aspx http://msdn.microsoft.com/en-us/library/system.datetime.now(...)
  - mattchamb 13y ago