4 ms·

This looks like a fun project, but claiming that it “beats luajit” is very premature based on the benchmarks listed. The README lists only an allocation stress

This looks like a fun project, but claiming that it “beats luajit” is very premature based on the benchmarks listed. The README lists only an allocation stress test (fast allocs are good, to be sure) and recursive Fibonacci (always good fun to try but doesn’t resemble real usage). Looks like it doesn’t have a JIT compiler, and without that I’d put a lot of money against it ever being as fast as LuaJIT, V8, the JVM, etc etc in the general case. Edit to add: on reading more closely, there’s more benchmark-gaming going on here than I realised. On the second benchmark, it says: As you can see, minivm (no JIT) is a hair slower than Node JS (JIT) but beats luajit --joff by a fair margin (no JIT). They already noted that Node has a longer startup time, so that obviously dominates in a benchmark that’s only a few milliseconds long; and disabling the JIT in LuaJIT doesn’t tell us much.

CalChris 5y ago
Fibonacci is the poster child of hotspots. A JIT should be crushing an interpreter at Fibonacci as it is simple to generate good code for. Yes, it's just a microbenchmark; so you shouldn't make SPEC type conclusions from it. But you shouldn't dismiss it either.
isaacimagine 5y ago
Personally, I find it pretty impressive that it performs as well as these runtimes despite not having a JIT compiler. I'm pretty sure Shaw's written more benchmarks, but as the README explains, it's really hard to tell what the performance characteristics of a language are without writing a larger application. So far the largest applications written with MiniVM are Paka[0], a self-hosted language similar to Lua that targets MiniVM; os49[1], an operating system built on Paka/MiniVM in the spirit of lisp machines; and xori[2], an online playground for the language. Edit: to address your edit, the the Tree benchmark starts small, but grows exponentially (the graph is logarithmic). The final runs of the benchmark take about 15 seconds to run, and are run 10 times per language, at which point startup time is no longer a large factor. The fib test compares both luajit with the JIT on and JIT off: as MiniVM is not JIT'd, I think that this is a fair comparison to make. [0]: https://github.com/FastVM/paka https://github.com/FastVM/paka [1]: https://github.com/ShawSumma/os49 https://github.com/ShawSumma/os49 [2]: https://fastvm.github.io/xori https://fastvm.github.io/xori
- iainmerrick 5y ago
  Sure, it’s a nice piece of work! I just don’t think you should claim it’s very fast unless you can show that it actually is very fast. The current benchmarks look like toy benchmarks. (I don’t mean that in a derogatory way -- they’re exactly the kind of tiny programs you need to build and test while bringing up a new VM.)
  - mst 5y ago
    The disclaimer at the top of the benchmark section - https://github.com/FastVM/minivm#benchmarks https://github.com/FastVM/minivm#benchmarks - seemed pretty reasonable to me. It's very clear that microbenchmarks don't tell the whole story, and that whole application benchmarks to give more real results will be desirable later. I mean, I agree these benchmarks don't mean that much except for being a useful way to verify that the core code is working and reasonably performant - but they're up front about that fact so calling it 'benchmark-gaming' seems a trifle unfair.
4984 5y ago
MiniVM will JIT eventually. Right now the biggest barrier is the VM snapshots. Currently MiniVM its own stack and heap and can snapshot them from any point in the program. One thing to note about MiniVM is that it has very strong types. V8 has many steps to go through when it needs to perform something like addition (weak typing). MiniVM supports only 4 basic types. MiniVM's reference frontend Paka is fully self hosted, It is currently the only large MiniVM program.
- samatman 5y ago
  That sounds like a solid basis for adding asymmetric coroutines as well, is that something you're thinking about adding? I like the philosophy and gumption shown by this project but coroutines aren't a feature I'd lightly give up. Digging what I see so far!
  - 4984 5y ago
    MiniVM uses coroutines already, every 1000 branches the vm will return to the scheduler. The Build system can accept CFLAGS+=-DVM_BRANCH_DEFER which makes MiniVM able to run part of bytecode up-to N instructions. It is not exposed on the insruction level yet tho. Would be quite easy to add.
    - exikyut 5y ago
      It might be interesting to be able to tell the VM to resume into code for exactly N instructions or branches. I can see instrumentation/debugging tooling using this with N=1 for example.
    - samatman 5y ago
      Ah thank you, that's very promising! It was the kind of thing I wanted to hear, I write Lua every day and four data types didn't appear to support 'thread'. I might just kick the tires on this over the weekend.
    - rgrmrts 5y ago
      That’s cool. Have you played around with that number? I’m curious how you pick that constant and what the trade offs are

CalChris 5y ago

Fibonacci is the poster child of hotspots. A JIT should be crushing an interpreter at Fibonacci as it is simple to generate good code for. Yes, it's just a microbenchmark; so you shouldn't make SPEC type conclusions from it. But you shouldn't dismiss it either.

isaacimagine 5y ago

Personally, I find it pretty impressive that it performs as well as these runtimes despite not having a JIT compiler. I'm pretty sure Shaw's written more benchmarks, but as the README explains, it's really hard to tell what the performance characteristics of a language are without writing a larger application. So far the largest applications written with MiniVM are Paka[0], a self-hosted language similar to Lua that targets MiniVM; os49[1], an operating system built on Paka/MiniVM in the spirit of lisp machines; and xori[2], an online playground for the language. Edit: to address your edit, the the Tree benchmark starts small, but grows exponentially (the graph is logarithmic). The final runs of the benchmark take about 15 seconds to run, and are run 10 times per language, at which point startup time is no longer a large factor. The fib test compares both luajit with the JIT on and JIT off: as MiniVM is not JIT'd, I think that this is a fair comparison to make. [0]: https://github.com/FastVM/paka https://github.com/FastVM/paka [1]: https://github.com/ShawSumma/os49 https://github.com/ShawSumma/os49 [2]: https://fastvm.github.io/xori https://fastvm.github.io/xori

iainmerrick 5y ago
Sure, it’s a nice piece of work! I just don’t think you should claim it’s very fast unless you can show that it actually is very fast. The current benchmarks look like toy benchmarks. (I don’t mean that in a derogatory way -- they’re exactly the kind of tiny programs you need to build and test while bringing up a new VM.)
- mst 5y ago
  The disclaimer at the top of the benchmark section - https://github.com/FastVM/minivm#benchmarks https://github.com/FastVM/minivm#benchmarks - seemed pretty reasonable to me. It's very clear that microbenchmarks don't tell the whole story, and that whole application benchmarks to give more real results will be desirable later. I mean, I agree these benchmarks don't mean that much except for being a useful way to verify that the core code is working and reasonably performant - but they're up front about that fact so calling it 'benchmark-gaming' seems a trifle unfair.

4984 5y ago

MiniVM will JIT eventually. Right now the biggest barrier is the VM snapshots. Currently MiniVM its own stack and heap and can snapshot them from any point in the program. One thing to note about MiniVM is that it has very strong types. V8 has many steps to go through when it needs to perform something like addition (weak typing). MiniVM supports only 4 basic types. MiniVM's reference frontend Paka is fully self hosted, It is currently the only large MiniVM program.

samatman 5y ago
That sounds like a solid basis for adding asymmetric coroutines as well, is that something you're thinking about adding? I like the philosophy and gumption shown by this project but coroutines aren't a feature I'd lightly give up. Digging what I see so far!
- 4984 5y ago
  MiniVM uses coroutines already, every 1000 branches the vm will return to the scheduler. The Build system can accept CFLAGS+=-DVM_BRANCH_DEFER which makes MiniVM able to run part of bytecode up-to N instructions. It is not exposed on the insruction level yet tho. Would be quite easy to add.
  - exikyut 5y ago
    It might be interesting to be able to tell the VM to resume into code for exactly N instructions or branches. I can see instrumentation/debugging tooling using this with N=1 for example.
  - samatman 5y ago
    Ah thank you, that's very promising! It was the kind of thing I wanted to hear, I write Lua every day and four data types didn't appear to support 'thread'. I might just kick the tires on this over the weekend.
  - rgrmrts 5y ago
    That’s cool. Have you played around with that number? I’m curious how you pick that constant and what the trade offs are