9 ms·
Completely unscientific, but if these outputs are any indication, this is going to be great news for Rubby users in the future... $ time ruby -e "puts 'hello
by Pewpewarrows 14y ago
Completely unscientific, but if these outputs are any indication, this is going to be great news for Rubby users in the future...
$ time ruby -e "puts 'hello world'"
hello world
real 0m0.184s
user 0m0.079s
sys 0m0.092s
$ time ~/Downloads/topaz/bin/topaz -e "puts 'hello world'"
hello world
real 0m0.007s
user 0m0.002s
sys 0m0.004s
- deleted 14y ago[deleted]
- kingkilr 14y agoTopaz doesn't run rails yet (as far as I know, I didn't even dare to try!), so I doubt you'll find any benchmarks ;) There is one benchmark in the bench/ directory of the repository you can try though!
- fusiongyro 14y agoInstead of asking, you could read the linked article, which itself says it isn't complete enough to run Rails yet.
- eranation 14y agoImpressive, how does it compare to JRuby?
- andrewaylett 14y agoThat's probably not a fair test to use on JRuby, as the JVM is notoriously slow to start.
- JoshTriplett 14y agoThat doesn't make it unfair to JRuby, it just means JRuby will probably lose that benchmark. :)
- nirvdrum 14y agoIf the goal is to benchmark Ruby execution time, it's unfair. The slow startup time is a valid concern when considering short-lived sessions, but it's kind of meaningless when trying to benchmark a Ruby implementation.
- fleitz 14y agoThere are no 'fair' benchmarks, all benchmarks should be biased to the problem you're actually solving running your actual workload. If you can't replay your workload at multiples of real volume then you should probably work on doing that before benchmarking as it helps you out with the real problem of verifying your infrastructure. In general a benchmark is probably the worst metric you could ever use for deciding on an implementation. Unless the profit margin of your business is razor thin and dependant eeking out every last drop of performance, and even then most of those gains will be from extremely small sections of code that are probably best written in assembler by a programming God, and you should investigate FPGAs, ASICs, and other high performance solutions. If your benchmark (infrastructure) involves a database (or anything that uses disks) that's probably going to be the problem long before the speed of your language / language implementation.
- nirvdrum 14y agoI don't even know where to begin with this. In all comparisons, you should remove confounding variables. Yes, you should benchmark something you actually care about, otherwise what's the point? That doesn't mean all other variables are immediately null and void. That's why I said said if your goal is to measure ruby execution time, you should remove startup time. As for the practice of benchmarking in general, you're partially right. Micro-benchmarks are usually useless because they don't map to real work load. But profiling and speeding up small portions that are used heavily can have drastic improvements that in isolation seem small -- the death by a thousand cuts problem. Not all improvements come from isolated instances with very slow performance profiles. This fallacy about DB access and not needing to optimize really needs to go away though. Even if 50% of your app is spent hitting DB, you have opportunity to speed up the other 50% and it's likely far easier. Ruby in particular is ripe for improvements on the CPU side. I managed to reduce my entire test suite time by 30% by speeding up psych. I managed to cut the number of servers I need in EC2 in half by switching from MRI, Pasenger, and resque to JRuby, TorqueBox, and Sidekiq. And I've managed to speed up my page rendering time anywhere from 8 - 40x by switching from haml to slim. None of these changes required modifications to my DB, none required me to write assembly, none required me to switch to custom-built hardware, and each helped reduce the expenses for my bootstrapped startup, while improving the overall experience for my customers.
- deleted 14y ago[deleted]
- deleted 14y ago[deleted]
- fijal 14y agothe idea was to implement enough of "hard stuff". So it should not change. but hello world is not a good idea (this one also might actually change due to library loading, but please don't benchmark it like that)
- jtanderson 14y agoMakes sense. I wasn't claiming any "absolute" benchmark, of course, just pointing out that real benchmarks should wait until the implementation is more complete.
- fijal 14y agoin other words - if you can run it today, you should believe numbers. if you cannot run it, then well, you cannot.
- stcredzero 14y ago> the lack of many core features probably helps out a lot atm. I don't understand why the lack of features would affect the time of "hello world."
- ephemeralgomi 14y agoLoading the standard library takes time. With a smaller (underimplemented) standard library, you can get to the user's code much more quickly.
- stcredzero 14y agoPerl 5 had an in-memory footprint of about 768k. VisualWorks Smalltalk had a standard one of 12MB, yet loaded and started it faster than Perl did with 768k. If just your standard library takes very long at all to load, something needs attention.
- seivan 14y agoTry comparing with Rubinius
- Cowen 14y agoWhenever I see frontpages for these kinds of projects like "a faster X" or "X written in Blub", the first thing I want to see on the frontpage is how this new project compares to X in terms of quality and performance. Even specious benchmarks would help more than zero benchmarks. I wish more frontpages for these kinds of projects would do that.
- pyre 14y agoIf they put that on their frontpage, there would be at least 20 posts on here bashing them for it because they didn't get it right (or just accusing them of outright lying/incompetence).
- callum85 14y ago"Even specious benchmarks would help more than zero benchmarks." I disagree. Zero benchmarks is definitely better than specious benchmarks.
- Cowen 14y agoTo clarify, I was trying to use "specious" as a synonym of "flawed." I thought this was the common usage, but apparently not. As to your point, obviously no one should be making any decisions off of flawed benchmarks, but flawed benchmarks (not so far as outright lies, just flawed) at least give me an objective justification to investigate further. Even some flawed benchmarks could help turn the initial tide of responses like "this is X written in Blub, it's bound to be better!" or "this is a faster X! Now everything will be twice as fast!" They're silly examples, but it seems like every time a new technology comes out, these are the kinds of knee-jerk, overly-optimistic reactions people tend to have.
- kenko 14y ago" To clarify, I was trying to use "specious" as a synonym of "flawed." I thought this was the common usage, but apparently not." "Specious" does mean, more or less, flawed. Zero benchmarks are better than flawed benchmarks.
- draegtun 14y agoTo scratch a quick interesting thought that came into my head I just had a look at Cardinal, which is a Ruby implementation running on the Parrot VM - https://github.com/parrot/cardinal https://github.com/parrot/cardinal So downloaded & built Cardinal (which went seamlessly however I did have Parrot already installed) then I did same benchmarks alongside ruby1.8 here: $ time ruby -e "puts 'hello world'" hello world real 0m0.130s user 0m0.049s sys 0m0.071s $ time parrot-cardinal -e "puts 'hello world'" hello world real 0m0.057s user 0m0.037s sys 0m0.019s Very interesting because I thought Cardinal was supposed to be slow! I think more diverse benchmarks are required. And when time permitting I might add Topaz & ruby1.9 into the mix.
- grn 14y agoIn order to eliminate the potential bias of startup time I run similar test in many iterations. Here's what I got: $ time ruby -e "10000.times { puts 'hello world' }" > /dev/null real 0m0.102s user 0m0.096s sys 0m0.005s and $ time ./topaz -e "10000.times { puts 'hello world' }" > /dev/null real 0m0.098s user 0m0.071s sys 0m0.026s Any idea why I don't see such big difference?
- emillon 14y agoThe overhead of starting the interpreter must be longer that executing the loop.
- fijal 14y agotopaz probably has a pretty slow IO (for bad reasons, it's an RPython problem)
- fernandezpablo 14y agowhat is your 'ruby'?
- onedognight 14y agoGood question. How many shells is rbenv/rvm executing? Do you get similar results with an absolute path?
- grn 14y agoruby that comes with Debian Squeeze. $ ruby --version ruby 1.8.7 (2010-08-16 patchlevel 302) [i486-linux]
- duaneb 14y agoBecause the majority of the cycles ran are probably in 'puts', which is implemented in C if I were to guess.
- rbehrends 14y agoBecause a puts loop is I/O bound, not CPU bound.
- raphael_kimmig 14y agoThere is a neural net example benchmark in the topaz git repo. Don't know how representative that example is, but at least startup time shouldn't be dominating the results... $ ruby -v ruby 1.9.3p194 (2012-04-20 revision 35410) [x86_64-darwin11.3.0] $ ruby bench_neural_net.rb ruby bench_neural_net.rb 17,74s user 0,02s system 99% cpu 17,771 total $ bin/topaz bench_neural_net.rb bin/topaz bench_neural_net.rb 3,43s user 0,03s system 99% cpu 3,466 total
- ErikRogneby 14y agoshould be able to run any number of ruby examples from here: http://benchmarksgame.alioth.debian.org/u32/ruby.php http://benchmarksgame.alioth.debian.org/u32/ruby.php
- snissn 14y ago$ time ruby -e "puts 'hello world'" The program 'ruby' can be found in the following packages: * ruby1.8 * ruby1.9.1 Ask your administrator to install one of them real 0m0.060s user 0m0.040s sys 0m0.016s
- axomhacker 14y agoDon't know if you are trolling. But if this is genuine: this is rbenv telling you that you have multiple rubies installed. 1. Pick one (just for this session): $ rbenv shell ruby1.9.1 2. And then run the example. By the way 1.9.1 is really old already, 1.9.3 has a lot more bug fixes.
- alrs 14y agoruby1.9.1 on Debian isn't ruby 1.9.1. The Ruby language changed between 1.9 and 1.9.1, so a new package name had to be created. If 1.9.1 was just called "1.9" it would break all of the packages in Debian that depend on whatever language features were different between 1.9 and 1.9.1. "ruby1.9.1" in Debian 7.0 provides version 1.9.3.194.
- axomhacker 14y agoThe OP's output looked more like rbenv's output though, right? How did you figure that to be debian's message?
- haven 14y agoExpanding on the concept of entirely unscientific benchmarks, I benched some simple prime number math with Topaz, Ruby, JRuby and RBX. Looks promising for Topaz! https://gist.github.com/havenwood/4724778 https://gist.github.com/havenwood/4724778
- deleted 14y ago[deleted]
- pserwylo 14y agoYou should check out a tool I heard about recently called "ministat". An example of how to use it to benchmark multiple runs, and then compute the statistics from each run are available here: http://anholt.net/compare-perf/ http://anholt.net/compare-perf/ Example output: +------------------------------------------------------------------------------+ | + x | | + x | | + + x x x | | + ++ +++ + x xxx xx x | |++ ++++++++++++++++++ x x x xxx xxx xxxxx | |++ ++++++++++++++++++ +++ + ++ xxxxxxxxxx xxxxxxxxxxxxxxxx xx x xx| | |______MA______| |________A________| | +------------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 57 45.62364 46.437353 45.93506 45.951554 0.19060973 + 57 44.785579 45.534727 45.042576 45.056702 0.16634531 Difference at 95.0% confidence -0.894852 +/- 0.0656777 -1.94738% +/- 0.142928% (Student's t, pooled s = 0.178889)
- charliesome 14y agoHow the hell is your Ruby that slow to start? $ time ruby -e "puts 'hello world'" hello world real 0m0.011s user 0m0.008s sys 0m0.003s
- d33pika 14y agoThe first time I ran: time ruby -e "puts 'hello world'" hello world real 0m0.221s user 0m0.005s sys 0m0.006s subsequent times: time ruby -e "puts 'hello world'" hello world real 0m0.008s user 0m0.005s sys 0m0.003s So, I guess he ran ruby first followed by topaz and ended up with those results