13 ms·

Checking for my own “benchmark”, a gameboy emulator in several different languages[0]; it’s CPU-bound but across ~3k lines of code, so slightly more representat

Checking for my own “benchmark”, a gameboy emulator in several different languages[0]; it’s CPU-bound but across ~3k lines of code, so slightly more representative of real-world apps than a single-function tight-loop microbenchmark: zig: Emulated 600 frames in 0.24s (2521fps) rs: Emulated 600 frames in 0.37s (1626fps) cpp: Emulated 600 frames in 0.40s (1508fps) nim: Emulated 600 frames in 0.44s (1367fps) go: Emulated 600 frames in 1.75s (342fps) php: Emulated 600 frames in 23.74s (25fps) py: Emulated 600 frames in 26.16s (23fps) # PyPy py: Emulated 600 frames in 33.10s (18fps) # 3.11 py: Emulated 600 frames in 61.43s (9fps) # 3.10 Doubling the speed is pretty nice :D Still the slowest out of all implementations though :P [0] https://github.com/shish/rosettaboy https://github.com/shish/rosettaboy EDIT> updated the nim compiler flags to build in release mode like most other languages, thanks @plainOldText!

gw99 4y ago
Interesting. On the trifecta of money vs pain vs speed, Go seems to be a reasonable compromise.
- Thaxll 4y ago
  It's probably "slow" because of sdl and cgo not native Go code. A gb emulator doesn't do much actually, it's about fixed array, bit shifting, switch cases etc ... I ran a quick pprof and indeed it's spending a lot of time in cgo: Showing nodes accounting for 28720ms, 67.67% of 42440ms total Dropped 145 nodes (cum <= 212.20ms) Showing top 10 nodes out of 53 flat flat% sum% cum cum% 13080ms 30.82% 30.82% 16600ms 39.11% runtime.cgocall 4720ms 11.12% 41.94% 4750ms 11.19% main.(*RAM).get 2840ms 6.69% 48.63% 33070ms 77.92% main.(*GPU).tick 1970ms 4.64% 53.28% 3720ms 8.77% runtime.mallocgc 1450ms 3.42% 56.69% 1470ms 3.46% main.(*RAM).set 1160ms 2.73% 59.43% 41350ms 97.43% main.(*GameBoy).tick 1000ms 2.36% 61.78% 3160ms 7.45% runtime.exitsyscall 890ms 2.10% 63.88% 1610ms 3.79% main.(*CPU).tick_interrupts 820ms 1.93% 65.81% 850ms 2.00% runtime.casgstatus 790ms 1.86% 67.67% 5530ms 13.03% main.(*CPU).tick
  - Shish2k 4y ago
    > It's probably "slow" because of sdl and cgo not native Go code The other languages are also using SDL via their respective interacting-with-C interfaces - what makes Go special here?
    - Thaxll 4y ago
      Go FFI is notoriously slow. https://stackoverflow.com/questions/28272285/why-cgos-performance-is-so-slow-is-there-something-wrong-with-my-testing-code/28272990#28272990 https://stackoverflow.com/questions/28272285/why-cgos-perfor...
  - jbverschoor 4y ago
    Yup.. a 240x performance difference (zig-py) has very little to do with the language, vm, or whatever. As soon as I saw that, I dismissed the benchmark. By these standards a 10 year old cpu with a beefy GPU will beat any new cpu as well.
    - chlorion 4y ago
      What does a GPU have to do with any of this? I think you may be a little confused about something but I am not sure what.
    - logicchains 4y ago
      >Yup.. a 240x performance difference (zig-py) has very little to do with the language, vm, or whatever. It absolutely does. A simple for loop in the standard Python interpreter will literally take 100x longer than the same thing in a language like C/C++, try for yourself if you don't believe me. CPython is unbelievably slow.
      - jbverschoor 4y ago
        Very interesting... Simple loop (0 -> 320000000) and add 1 to a variable. I couldn't reproduce 100x (no optimization flags, otherwise it won't do anything) Apple clang version 14.0.0 (clang-1400.0.29.102) -> 0m0.347s ruby 3.1.2p20 -> 0m11.314s Python 3.8.12 -> 0m19.662s So ruby is almost 60% faster, but the C version is "only" 32x faster than ruby. 55x python. I thought the differences would be smaller these days
- friedman23 4y ago
  Maximize pain for mediocre speed and money?
- redox99 4y ago
  It depends really. In this case, a game emulator, you can only get away with it because it's for an ancient game console. But otherwise you definitely cannot afford 5x slowdown compared to CPP for a game.
- Shish2k 4y ago
  FWIW I personally found Rust the least-painful language, but that may well be confirming my pre-established biases :) - With Zig I kept running into compiler bugs, plus no package manager (I’ve vendored SDL and Clap into the source tree) - C++ I’d occasionally shoot myself in the foot in ways that other languages would have caught, plus no package manager (OS-level package management does an OK job, so long as you don’t mind using old versions, and faffing about with different operating systems acting very differently) - The pain from Rust was one time where the compiler wanted me to specify a lifetime, and I didn’t understand, so I just spammed lifetime specifiers in various places until it compiled. I’ve been using Rust for a couple of years now and I still don’t really understand lifetimes, but thankfully 99% of the time I can avoid them. - Nim was a relatively nice language but massively lacking in available libraries (like even parsing command line arguments took me a day just trying to find a library which worked) - Go is pretty nice, my main pain is the tolerable but constantly-annoying verboseness of error handling (`err := foo(); if err != nil {return err}` compared to rust’s `foo()?`) - PHP I just hate on a deep and personal level thanks to years of being a PHP4/5 developer. The language is actually mostly-ok-ish these days, but the standard library is still full of frustration like inconsistent parameter orders within a family of functions. - Python is all-round really nice to write, but the test suite takes like 20 minutes to run, which really messes with my flow-state
  - Thaxll 4y ago
    "Rust the least-painful language" " I’ve been using Rust for a couple of years now and I still don’t really understand lifetimes" Seems like a major pain point.
    - Shish2k 4y ago
      It would be if I ran into it regularly -- but after using the language for a variety of professional and personal projects for a couple of years, this is the only time I’ve actually needed to manually-specify lifetimes since the compiler normally figures it out for me :)
    - robocat 4y ago
      They are saying rust is painful. Just that they find the other languages even more painful.
- DeathArrow 4y ago
  Also C# is quite reasonable.
humanistbot 4y ago
I am shocked that PHP of all things has faster speed than python.
- bityard 4y ago
  It's not as surprising if you look at the primary use case of PHP. It takes a request, does a bunch of stuff, and returns the result to the web server. And it needs to do all of it before the user who clicked on the link gets bored and goes somewhere else. (Or the API client calling it times out...)
  - coldtea 4y ago
    How is that relevant? This is not about it being faster for "it's primary use case" (optimized for that, etc). It's generally faster than Python in doing the same things as Python does, unrelated to web too.
    - lmm 4y ago
      The point is that PHP's primary, original use case is latency-sensitive in a way that Python's isn't. So it's always been subject to more performance pressure.
- crote 4y ago
  I am not. PHP is traditionally used solely for websites. Some of those have grown rather large, to the point that having engineers optimize the language is cheaper than buying more servers. Python, on the other hand, is first and foremost a scripting language. When performance does matter, you often end up using a wrapper around a C library, like NumPy. This means there is relatively little money in optimizing the Python interpreter.
  - DeathArrow 4y ago
    There are websites and large scale software. Google uses Python for a lot of its products. It would seem sensible that since Facebook poured a lot of resources in optimizing PHP, Google would have done the same for Python. Also, Python is the first or second most popular language.
insanitybit 4y ago
Very cool idea for a benchmark, thanks for the numbers. Pretty readable code as well, nice work. Is your benchmark running headless? I feel like that could be a source of noise but idk.
- Shish2k 4y ago
  Yeah, all these benchmarks are measured in headless mode - all the calculations of what should be on-screen are done, and pixels are written into a buffer ready to be displayed, but the Window is never opened and the buffer isn’t blitted.
stavros 4y ago
Can you try PyPy as well?
- Shish2k 4y ago
  Apparently not :( (These benchmarks were done on an M1 MacBook Pro) $ brew install pypy3 pypy3: The x86_64 architecture is required for this software.
  - verst 4y ago
    $ brew install pyenv $ pyenv install pypy3.9-7.3.9 I like using PyEnv for managing my Python versions. It will natively compile Python builds and should be doing the same on M1 (which is what I'm using). `pyenv install --list` shows you what is available. EDIT: Not sure why they don't have newer versions of PyPy there (I don't use PyPy) but all it takes is a PR to here: https://github.com/pyenv/pyenv https://github.com/pyenv/pyenv
    - deleted 4y ago
      [deleted]
    - Shish2k 4y ago
      Thanks! Added to the list, it’s suspiciously only a little bit faster than CPython 3.11 though, which probably needs more investigation… ED> Looks like CPython prefers using a dict as a lookup table for opcodes (which is what this implementation does), while PyPy prefers having a long series of if-statements. Hmm.
      - hnov 4y ago
        That makes sense because pypy can JIT a ladder of if/else to a compare/jump conditional each whereas a dict lookup can be an order of magnitude more complex. If they're 8bit opcodes, maybe having a list based lookup table will perform similarly on py3000 and still optimize on pypy? Or maybe https://docs.python.org/3/library/array.html https://docs.python.org/3/library/array.html rather than list.
winter_blue 4y ago
It's amazing that Zig is faster than both Rust and C++. Kudos to the Zig team & Andrew Kelly! I wonder what optimizations Zig does that lets it generate machine code / LLVM bitcode faster than both C++ and Rust (which certainly have larger teams backing them), at least in the case of this Gameboy emulator project.
- cma 4y ago
  Does zig have computed goto? That's important for emulators and virtual machines and C/C++ don't have it. Projects will often go out of their way to have a MinGW/gcc built module for core loops with that (gcc has it as an extension) even if the main project is built with MSVC.
  - nerpderp82 4y ago
    If Rust wasn't able to elide bounds checks, that is most likely where the perf difference is from. Analyzing the output the assembly output from goldbolt or looking at the MIR can help. https://stefan-marr.de/2022/10/cost-of-safety-in-java/ https://stefan-marr.de/2022/10/cost-of-safety-in-java/ Unsafety buys you a little more performance, but the baseline should the safe version, not the unsafe version. It is like having to explain on a case by case basis why you aren't using lead pipes for this application. Always default to safety. The difference between safe and unsafe native code is usually single digit percentage points. Or weeks on a Moore scale.
    - cma 4y ago
      I think you replied to the wrong comment, mine was about zig vs C++.
      - nerpderp82 4y ago
        I am talking about the whole stack of benchmarks and why there is a spread in perf. I am replying to you and the top level comment, if only the convos were a graph and not a tree.
    - 4y ago
plainOldText 4y ago
Just a quick glance at you repo, and I'm noticing you're running Zig `zig build -fstage1 -Drelease-fast=true` and Rust `cargo run --release` with the release flags on. You should do the same for Nim `nimble build -d:release --opt:speed`; Go too.
- Shish2k 4y ago
  Updating nim’s compiler flags, seems to be ~4x faster now :D nim: Emulated 600 frames in 0.44s (1367fps) Do you happen to know the right flags for release-mode Go? Last time I checked (admittedly years ago) I thought they just had the one “reasonably fast and reasonably debuggable” build mode
  - Thaxll 4y ago
    There is no release flag for Go.
    - throwaway894345 4y ago
      To elaborate, Go is always in release mode because release compiles are about as fast as other languages' debug modes.
      - LukeShu 4y ago
        I'd phrase that as "release mode by default", not "always in release mode". There are various debug things you can turn on, such as the race detector, the memory sanitizer, or coverage tracking.
  - plainOldText 4y ago
    I do not, sorry. I'm sure some Go developer can chime in.
  - cb321 4y ago
    With Nim `nimble build -d:danger -d:lto --passC:-march=native` I just got 1920 frames/s while with your rust build only 1237 fps on the same machine (EDIT: and 1307 frames/s with C++.)
jetbalsa 4y ago
I find it /really/ funny that PHP is faster then python at this. as a PHP code hacker, I love that PHP keeps coming up faster then python in a ton of tasks
- throwaway894345 4y ago
  That's not much of a flex, Python is deliberately slow--the reasoning was that the implementation should be simple and they'd just expose the entire interpreter as the extension API, and then people would just write extensions in C when they needed the speed. Since vanilla Python is so slow, the entire ecosystem is highly dependent on C extensions for passable performance, and since the C extension API is virtually the whole interpreter, changes that would make the interpreter fast would typically break much of the ecosystem. Unfortunately, "just write the performance-sensitive bits in C" is pretty impractical because it only works when you're handing the C routine a relatively small amount of data relative to the amount of work to be done on that data (otherwise the costs of marshaling to C data structures will quickly eat up any gains from processing in C). And unfortunately, it turns out that a whole bunch of code works this way, so the whole bargain of "slow interpreter + easy C extensions" breaks down for a lot of real-world applications, but now we're locked into it.
  - wheelerof4te 4y ago
    CPython can be fast once you eliminate the main bottleneck: the interpreter itself. Processing loops adds a lot of overhead because the interpreter has to make the case jumps after each loop. Figuring out a way to minimize that overhead by using built-in data structures and stdlib library will speed up your code by an order of magnitude. Don't forget, the built-in types are already running in C.
    - throwaway894345 4y ago
      I mean, yes, but that's probably not viable in many real-world applications. Usually you have a pretty complex data model (some graph-like structure) with Python methods that traverse it. You can't easily push that into native Python structures (at least not with any significant performance gain), and while you can push the whole thing into Rust or some faster language, at that point all of the interesting stuff is happening in Rust so why use Python at all (why pay the significant costs of a hybrid application)?
pcwalton 4y ago
Looking at Rust vs. Zig, you have the FLAGS register in a bitfield for Zig but it's separate bools for Rust. This is probably making your Rust code slower because the CPU can't set multiple flags at once. I'm also wondering if all your #[inline(always)] is slowing things down.
- Shish2k 4y ago
  Yeah, I really like zig’s approach to bitfields, everything Just Works with no faffing about with bit-shifting and OR/AND'ing. I forget the exact reason I used separate bools for rust, but I remember spending a day trying to do something zig-like and failing… IIRC each of the #[inline] statements was tested and each made a noticable performance improvement. That’s especially true for things like RAM::get() - since the gameboy does I/O by having different chunks of RAM act differently (some address ranges are just RAM, some are hardware controls, some read data from the cartridge, etc) you can replace the hundred-line generic get() with a single instruction if you happen to know that you are looking up one hard-coded address, and that address has no special behaviour.
  - deleted 4y ago
    [deleted]
- Rusky 4y ago
  I ran the Rust and Zig implementations through a profiler and (at least on my machine, using Zig nightly 0.10.0-dev.4583+875e98a57) the vast majority of the time difference is from the number of calls to the GPU paint_tile_line function- there's some behavioral difference upstream in the emulator and the GPU modules are just not doing the same work. For the benchmark 600 frames, the Rust emulator calls it 2500049 times, but the Zig emulator only calls it 1808121 times. This is very roughly the ratio between the reported times. At least one source of this discrepancy is that the Zig emulator doesn't think any sprites are active, but that doesn't account for all of it, and I'm hitting some crashes trying to run it with the display, so I'm probably not going to investigate it further. (I also hit the same sort of strange Zig compiler unreliability as some other comments have mentioned, where I had to add some logging in seemingly random places to get the Zig emulator to run at all. And the argument parser just blatantly returns a dead pointer to the ROM path; when I first ran it it just gave "error: InvalidUtf8" until I tracked that down.)
talideon 4y ago
I think a bigger thing to point out is that it's within shouting distance of PyPy, and there's plenty more work that can be done to make it faster. Next up is a mini-JIT, which should help in places with tight loops.
FerociousTimes 4y ago
Can you do Node 19.0 and Ruby 3.1.x too please?
- Shish2k 4y ago
  TypeScript and Java are the two remaining languages where I feel I know them well enough to translate 3k lines of code in a weekend each -- I have approximately zero knowledge of Ruby though, and no motivation to learn it ^^; Pull requests are welcome though!
  - FerociousTimes 4y ago
    OK, I'll watch your repo and track the updates and see when you release the JS/TS port and then I'll a give it a try and port it to Ruby and send it your way. Best wishes
mgkimsal 4y ago
Your comment in PHP src here https://github.com/shish/rosettaboy/blob/master/php/run.sh https://github.com/shish/rosettaboy/blob/master/php/run.sh says # opcache in 8.1 gives a nice speedup (25s to 10s) Is the 23.7 seconds above using the 8.1 opcache?
wk_end 4y ago
I'd be super curious to run a profiler on those Python builds and see where it's spending most of its time.
- deleted 4y ago
  [deleted]
b_mc2 4y ago
Very cool benchmark. I don't think this will be a huge difference but could you try mypyc? I'm definitely curious how it compares. It uses mypy to perform type checks and claims "Existing code with type annotations is often 1.5x to 5x faster when compiled. Code tuned for mypyc can be 5x to 10x faster."[1] [1] https://mypyc.readthedocs.io/en/latest/introduction.html https://mypyc.readthedocs.io/en/latest/introduction.html
lr1970 4y ago
To make Rust release build run faster you should enable LTO (Link Time Optimization). In the Cargo.toml put [profile.release] lto = true
justinsaccount 4y ago
This might speed up the py impl a few % --- a/py/src/cpu.py +++ b/py/src/cpu.py @@ -260,7 +260,6 @@ class CPU: param = None cmd_str = cmd.name self.PC += 1 - self._debug_str = f"[{self.PC:04X}({ins:02X})]: {cmd_str}"
habibur 4y ago
Here's his latest numbers after optimizations. From the repo. zig: Emulated 600 frames in 0.23s (2880fps) rs: Emulated 600 frames in 0.35s (1691fps) cpp: Emulated 600 frames in 0.40s (1519fps) nim: Emulated 600 frames in 0.44s (1367fps) go: Emulated 600 frames in 1.78s (338fps) php: Emulated 600 frames in 9.28s (65fps) py: Emulated 600 frames in 33.10s (18fps) That's after Python cheated with native implementation of bitblt while all other implementations are doing it pixel by pixel, which is more correct.
- cb321 4y ago
  Tables like this routinely lead to over-conclusion. Merely adding -d:lto (which changes no semantics) to the nimble build line boosts fps by 1.27x for me on nim-devel gcc-12 Skylake, making it 1.4x faster than the Rust on the same machine. Applying BS scaling to the above table, Nim would score 1736, only slightly faster than the Rust. Who knows what little flag tweaks in cpp/rs/zig could similarly re-arrange the numbers? Now, why is -d:lto (or -d:release) not the default build mode? Well, because it takes time and (some) people already complain about compile times and workflow (sometimes). Trade offs abound almost as much as over-conclusion. ;-)
sandGorgon 4y ago
what about java graalvm ?
make3 4y ago
do you use numpy for the numerical python stuff

gw99 4y ago

Interesting. On the trifecta of money vs pain vs speed, Go seems to be a reasonable compromise.

Thaxll 4y ago
It's probably "slow" because of sdl and cgo not native Go code. A gb emulator doesn't do much actually, it's about fixed array, bit shifting, switch cases etc ... I ran a quick pprof and indeed it's spending a lot of time in cgo: Showing nodes accounting for 28720ms, 67.67% of 42440ms total Dropped 145 nodes (cum <= 212.20ms) Showing top 10 nodes out of 53 flat flat% sum% cum cum% 13080ms 30.82% 30.82% 16600ms 39.11% runtime.cgocall 4720ms 11.12% 41.94% 4750ms 11.19% main.(*RAM).get 2840ms 6.69% 48.63% 33070ms 77.92% main.(*GPU).tick 1970ms 4.64% 53.28% 3720ms 8.77% runtime.mallocgc 1450ms 3.42% 56.69% 1470ms 3.46% main.(*RAM).set 1160ms 2.73% 59.43% 41350ms 97.43% main.(*GameBoy).tick 1000ms 2.36% 61.78% 3160ms 7.45% runtime.exitsyscall 890ms 2.10% 63.88% 1610ms 3.79% main.(*CPU).tick_interrupts 820ms 1.93% 65.81% 850ms 2.00% runtime.casgstatus 790ms 1.86% 67.67% 5530ms 13.03% main.(*CPU).tick
- Shish2k 4y ago
  > It's probably "slow" because of sdl and cgo not native Go code The other languages are also using SDL via their respective interacting-with-C interfaces - what makes Go special here?
  - Thaxll 4y ago
    Go FFI is notoriously slow. https://stackoverflow.com/questions/28272285/why-cgos-performance-is-so-slow-is-there-something-wrong-with-my-testing-code/28272990#28272990 https://stackoverflow.com/questions/28272285/why-cgos-perfor...
- jbverschoor 4y ago
  Yup.. a 240x performance difference (zig-py) has very little to do with the language, vm, or whatever. As soon as I saw that, I dismissed the benchmark. By these standards a 10 year old cpu with a beefy GPU will beat any new cpu as well.
  - chlorion 4y ago
    What does a GPU have to do with any of this? I think you may be a little confused about something but I am not sure what.
  - logicchains 4y ago
    >Yup.. a 240x performance difference (zig-py) has very little to do with the language, vm, or whatever. It absolutely does. A simple for loop in the standard Python interpreter will literally take 100x longer than the same thing in a language like C/C++, try for yourself if you don't believe me. CPython is unbelievably slow.
    - jbverschoor 4y ago
      Very interesting... Simple loop (0 -> 320000000) and add 1 to a variable. I couldn't reproduce 100x (no optimization flags, otherwise it won't do anything) Apple clang version 14.0.0 (clang-1400.0.29.102) -> 0m0.347s ruby 3.1.2p20 -> 0m11.314s Python 3.8.12 -> 0m19.662s So ruby is almost 60% faster, but the C version is "only" 32x faster than ruby. 55x python. I thought the differences would be smaller these days
friedman23 4y ago
Maximize pain for mediocre speed and money?
redox99 4y ago
It depends really. In this case, a game emulator, you can only get away with it because it's for an ancient game console. But otherwise you definitely cannot afford 5x slowdown compared to CPP for a game.
Shish2k 4y ago
FWIW I personally found Rust the least-painful language, but that may well be confirming my pre-established biases :) - With Zig I kept running into compiler bugs, plus no package manager (I’ve vendored SDL and Clap into the source tree) - C++ I’d occasionally shoot myself in the foot in ways that other languages would have caught, plus no package manager (OS-level package management does an OK job, so long as you don’t mind using old versions, and faffing about with different operating systems acting very differently) - The pain from Rust was one time where the compiler wanted me to specify a lifetime, and I didn’t understand, so I just spammed lifetime specifiers in various places until it compiled. I’ve been using Rust for a couple of years now and I still don’t really understand lifetimes, but thankfully 99% of the time I can avoid them. - Nim was a relatively nice language but massively lacking in available libraries (like even parsing command line arguments took me a day just trying to find a library which worked) - Go is pretty nice, my main pain is the tolerable but constantly-annoying verboseness of error handling (`err := foo(); if err != nil {return err}` compared to rust’s `foo()?`) - PHP I just hate on a deep and personal level thanks to years of being a PHP4/5 developer. The language is actually mostly-ok-ish these days, but the standard library is still full of frustration like inconsistent parameter orders within a family of functions. - Python is all-round really nice to write, but the test suite takes like 20 minutes to run, which really messes with my flow-state
- Thaxll 4y ago
  "Rust the least-painful language" " I’ve been using Rust for a couple of years now and I still don’t really understand lifetimes" Seems like a major pain point.
  - Shish2k 4y ago
    It would be if I ran into it regularly -- but after using the language for a variety of professional and personal projects for a couple of years, this is the only time I’ve actually needed to manually-specify lifetimes since the compiler normally figures it out for me :)
  - robocat 4y ago
    They are saying rust is painful. Just that they find the other languages even more painful.
DeathArrow 4y ago
Also C# is quite reasonable.

humanistbot 4y ago

I am shocked that PHP of all things has faster speed than python.

bityard 4y ago
It's not as surprising if you look at the primary use case of PHP. It takes a request, does a bunch of stuff, and returns the result to the web server. And it needs to do all of it before the user who clicked on the link gets bored and goes somewhere else. (Or the API client calling it times out...)
- coldtea 4y ago
  How is that relevant? This is not about it being faster for "it's primary use case" (optimized for that, etc). It's generally faster than Python in doing the same things as Python does, unrelated to web too.
  - lmm 4y ago
    The point is that PHP's primary, original use case is latency-sensitive in a way that Python's isn't. So it's always been subject to more performance pressure.
crote 4y ago
I am not. PHP is traditionally used solely for websites. Some of those have grown rather large, to the point that having engineers optimize the language is cheaper than buying more servers. Python, on the other hand, is first and foremost a scripting language. When performance does matter, you often end up using a wrapper around a C library, like NumPy. This means there is relatively little money in optimizing the Python interpreter.
- DeathArrow 4y ago
  There are websites and large scale software. Google uses Python for a lot of its products. It would seem sensible that since Facebook poured a lot of resources in optimizing PHP, Google would have done the same for Python. Also, Python is the first or second most popular language.

insanitybit 4y ago

Very cool idea for a benchmark, thanks for the numbers. Pretty readable code as well, nice work. Is your benchmark running headless? I feel like that could be a source of noise but idk.

Shish2k 4y ago
Yeah, all these benchmarks are measured in headless mode - all the calculations of what should be on-screen are done, and pixels are written into a buffer ready to be displayed, but the Window is never opened and the buffer isn’t blitted.

stavros 4y ago

Can you try PyPy as well?

Shish2k 4y ago
Apparently not :( (These benchmarks were done on an M1 MacBook Pro) $ brew install pypy3 pypy3: The x86_64 architecture is required for this software.
- verst 4y ago
  $ brew install pyenv $ pyenv install pypy3.9-7.3.9 I like using PyEnv for managing my Python versions. It will natively compile Python builds and should be doing the same on M1 (which is what I'm using). `pyenv install --list` shows you what is available. EDIT: Not sure why they don't have newer versions of PyPy there (I don't use PyPy) but all it takes is a PR to here: https://github.com/pyenv/pyenv https://github.com/pyenv/pyenv
  - deleted 4y ago
    [deleted]
  - Shish2k 4y ago
    Thanks! Added to the list, it’s suspiciously only a little bit faster than CPython 3.11 though, which probably needs more investigation… ED> Looks like CPython prefers using a dict as a lookup table for opcodes (which is what this implementation does), while PyPy prefers having a long series of if-statements. Hmm.
    - hnov 4y ago
      That makes sense because pypy can JIT a ladder of if/else to a compare/jump conditional each whereas a dict lookup can be an order of magnitude more complex. If they're 8bit opcodes, maybe having a list based lookup table will perform similarly on py3000 and still optimize on pypy? Or maybe https://docs.python.org/3/library/array.html https://docs.python.org/3/library/array.html rather than list.

winter_blue 4y ago

It's amazing that Zig is faster than both Rust and C++. Kudos to the Zig team & Andrew Kelly! I wonder what optimizations Zig does that lets it generate machine code / LLVM bitcode faster than both C++ and Rust (which certainly have larger teams backing them), at least in the case of this Gameboy emulator project.

cma 4y ago
Does zig have computed goto? That's important for emulators and virtual machines and C/C++ don't have it. Projects will often go out of their way to have a MinGW/gcc built module for core loops with that (gcc has it as an extension) even if the main project is built with MSVC.
- nerpderp82 4y ago
  If Rust wasn't able to elide bounds checks, that is most likely where the perf difference is from. Analyzing the output the assembly output from goldbolt or looking at the MIR can help. https://stefan-marr.de/2022/10/cost-of-safety-in-java/ https://stefan-marr.de/2022/10/cost-of-safety-in-java/ Unsafety buys you a little more performance, but the baseline should the safe version, not the unsafe version. It is like having to explain on a case by case basis why you aren't using lead pipes for this application. Always default to safety. The difference between safe and unsafe native code is usually single digit percentage points. Or weeks on a Moore scale.
  - cma 4y ago
    I think you replied to the wrong comment, mine was about zig vs C++.
    - nerpderp82 4y ago
      I am talking about the whole stack of benchmarks and why there is a spread in perf. I am replying to you and the top level comment, if only the convos were a graph and not a tree.
  - 4y ago

plainOldText 4y ago

Just a quick glance at you repo, and I'm noticing you're running Zig `zig build -fstage1 -Drelease-fast=true` and Rust `cargo run --release` with the release flags on. You should do the same for Nim `nimble build -d:release --opt:speed`; Go too.

Shish2k 4y ago
Updating nim’s compiler flags, seems to be ~4x faster now :D nim: Emulated 600 frames in 0.44s (1367fps) Do you happen to know the right flags for release-mode Go? Last time I checked (admittedly years ago) I thought they just had the one “reasonably fast and reasonably debuggable” build mode
- Thaxll 4y ago
  There is no release flag for Go.
  - throwaway894345 4y ago
    To elaborate, Go is always in release mode because release compiles are about as fast as other languages' debug modes.
    - LukeShu 4y ago
      I'd phrase that as "release mode by default", not "always in release mode". There are various debug things you can turn on, such as the race detector, the memory sanitizer, or coverage tracking.
- plainOldText 4y ago
  I do not, sorry. I'm sure some Go developer can chime in.
- cb321 4y ago
  With Nim `nimble build -d:danger -d:lto --passC:-march=native` I just got 1920 frames/s while with your rust build only 1237 fps on the same machine (EDIT: and 1307 frames/s with C++.)

jetbalsa 4y ago

I find it /really/ funny that PHP is faster then python at this. as a PHP code hacker, I love that PHP keeps coming up faster then python in a ton of tasks

throwaway894345 4y ago
That's not much of a flex, Python is deliberately slow--the reasoning was that the implementation should be simple and they'd just expose the entire interpreter as the extension API, and then people would just write extensions in C when they needed the speed. Since vanilla Python is so slow, the entire ecosystem is highly dependent on C extensions for passable performance, and since the C extension API is virtually the whole interpreter, changes that would make the interpreter fast would typically break much of the ecosystem. Unfortunately, "just write the performance-sensitive bits in C" is pretty impractical because it only works when you're handing the C routine a relatively small amount of data relative to the amount of work to be done on that data (otherwise the costs of marshaling to C data structures will quickly eat up any gains from processing in C). And unfortunately, it turns out that a whole bunch of code works this way, so the whole bargain of "slow interpreter + easy C extensions" breaks down for a lot of real-world applications, but now we're locked into it.
- wheelerof4te 4y ago
  CPython can be fast once you eliminate the main bottleneck: the interpreter itself. Processing loops adds a lot of overhead because the interpreter has to make the case jumps after each loop. Figuring out a way to minimize that overhead by using built-in data structures and stdlib library will speed up your code by an order of magnitude. Don't forget, the built-in types are already running in C.
  - throwaway894345 4y ago
    I mean, yes, but that's probably not viable in many real-world applications. Usually you have a pretty complex data model (some graph-like structure) with Python methods that traverse it. You can't easily push that into native Python structures (at least not with any significant performance gain), and while you can push the whole thing into Rust or some faster language, at that point all of the interesting stuff is happening in Rust so why use Python at all (why pay the significant costs of a hybrid application)?

pcwalton 4y ago

Looking at Rust vs. Zig, you have the FLAGS register in a bitfield for Zig but it's separate bools for Rust. This is probably making your Rust code slower because the CPU can't set multiple flags at once. I'm also wondering if all your #[inline(always)] is slowing things down.

Shish2k 4y ago
Yeah, I really like zig’s approach to bitfields, everything Just Works with no faffing about with bit-shifting and OR/AND'ing. I forget the exact reason I used separate bools for rust, but I remember spending a day trying to do something zig-like and failing… IIRC each of the #[inline] statements was tested and each made a noticable performance improvement. That’s especially true for things like RAM::get() - since the gameboy does I/O by having different chunks of RAM act differently (some address ranges are just RAM, some are hardware controls, some read data from the cartridge, etc) you can replace the hundred-line generic get() with a single instruction if you happen to know that you are looking up one hard-coded address, and that address has no special behaviour.
- deleted 4y ago
  [deleted]
Rusky 4y ago
I ran the Rust and Zig implementations through a profiler and (at least on my machine, using Zig nightly 0.10.0-dev.4583+875e98a57) the vast majority of the time difference is from the number of calls to the GPU paint_tile_line function- there's some behavioral difference upstream in the emulator and the GPU modules are just not doing the same work. For the benchmark 600 frames, the Rust emulator calls it 2500049 times, but the Zig emulator only calls it 1808121 times. This is very roughly the ratio between the reported times. At least one source of this discrepancy is that the Zig emulator doesn't think any sprites are active, but that doesn't account for all of it, and I'm hitting some crashes trying to run it with the display, so I'm probably not going to investigate it further. (I also hit the same sort of strange Zig compiler unreliability as some other comments have mentioned, where I had to add some logging in seemingly random places to get the Zig emulator to run at all. And the argument parser just blatantly returns a dead pointer to the ROM path; when I first ran it it just gave "error: InvalidUtf8" until I tracked that down.)

talideon 4y ago

I think a bigger thing to point out is that it's within shouting distance of PyPy, and there's plenty more work that can be done to make it faster. Next up is a mini-JIT, which should help in places with tight loops.

FerociousTimes 4y ago

Can you do Node 19.0 and Ruby 3.1.x too please?

Shish2k 4y ago
TypeScript and Java are the two remaining languages where I feel I know them well enough to translate 3k lines of code in a weekend each -- I have approximately zero knowledge of Ruby though, and no motivation to learn it ^^; Pull requests are welcome though!
- FerociousTimes 4y ago
  OK, I'll watch your repo and track the updates and see when you release the JS/TS port and then I'll a give it a try and port it to Ruby and send it your way. Best wishes

mgkimsal 4y ago

Your comment in PHP src here https://github.com/shish/rosettaboy/blob/master/php/run.sh https://github.com/shish/rosettaboy/blob/master/php/run.sh says # opcache in 8.1 gives a nice speedup (25s to 10s) Is the 23.7 seconds above using the 8.1 opcache?

wk_end 4y ago

I'd be super curious to run a profiler on those Python builds and see where it's spending most of its time.

deleted 4y ago
[deleted]

b_mc2 4y ago

Very cool benchmark. I don't think this will be a huge difference but could you try mypyc? I'm definitely curious how it compares. It uses mypy to perform type checks and claims "Existing code with type annotations is often 1.5x to 5x faster when compiled. Code tuned for mypyc can be 5x to 10x faster."[1] [1] https://mypyc.readthedocs.io/en/latest/introduction.html https://mypyc.readthedocs.io/en/latest/introduction.html

lr1970 4y ago

To make Rust release build run faster you should enable LTO (Link Time Optimization). In the Cargo.toml put [profile.release] lto = true

justinsaccount 4y ago

This might speed up the py impl a few % --- a/py/src/cpu.py +++ b/py/src/cpu.py @@ -260,7 +260,6 @@ class CPU: param = None cmd_str = cmd.name self.PC += 1 - self._debug_str = f"[{self.PC:04X}({ins:02X})]: {cmd_str}"

habibur 4y ago

Here's his latest numbers after optimizations. From the repo. zig: Emulated 600 frames in 0.23s (2880fps) rs: Emulated 600 frames in 0.35s (1691fps) cpp: Emulated 600 frames in 0.40s (1519fps) nim: Emulated 600 frames in 0.44s (1367fps) go: Emulated 600 frames in 1.78s (338fps) php: Emulated 600 frames in 9.28s (65fps) py: Emulated 600 frames in 33.10s (18fps) That's after Python cheated with native implementation of bitblt while all other implementations are doing it pixel by pixel, which is more correct.

cb321 4y ago
Tables like this routinely lead to over-conclusion. Merely adding -d:lto (which changes no semantics) to the nimble build line boosts fps by 1.27x for me on nim-devel gcc-12 Skylake, making it 1.4x faster than the Rust on the same machine. Applying BS scaling to the above table, Nim would score 1736, only slightly faster than the Rust. Who knows what little flag tweaks in cpp/rs/zig could similarly re-arrange the numbers? Now, why is -d:lto (or -d:release) not the default build mode? Well, because it takes time and (some) people already complain about compile times and workflow (sometimes). Trade offs abound almost as much as over-conclusion. ;-)

sandGorgon 4y ago

what about java graalvm ?

make3 4y ago

do you use numpy for the numerical python stuff