3 ms·

There is rubyspec, but it doesn't cover all possible paths, just the ones that the developers deemed important enough to test. What I am curious to know is _wh

by dkubb 10y ago

There is rubyspec, but it doesn't cover all possible paths, just the ones that the developers deemed important enough to test. What I am curious to know is _what_ does ruby use besides rubyspec? Do they run tests against the top 100, 1000, etc rubygems before releasing? I know other communities like Rust (and even Perl) do this kind of regression testing. While this won't catch all possible regressions it seems like a reasonable starting point to uncover issues before a release.

chrisseaton 10y ago
> Do they run tests against the top 100, 1000, etc rubygems before releasing? I don't believe they do. They only recently started testing against ruby spec. I work on an alternative implementation, and we are planning to test against literally all the gems in RubyGems ourselves. We test the top 100 or so at the moment.
- dkubb 10y ago
  If I might make a suggestion, I'd recommend doing some kind of tracing when you run the tests so you can have a mapping of what parts of your code are exercised by which gems/tests. Then when you make changes to your code you can run the tests that relate to that part of the system first, thus giving you more confidence that your change didn't cause a regression. This should be much more efficient than simply doing a full gem run (although I'd recommend that too, since it could uncover hidden dependencies).
riffraff 10y ago
ruby has always had its own set of tests, rubyspec is a very recent development.
- Someone 10y ago
  Very recent? 2006, according to Wikipedia (https://en.wikipedia.org/wiki/RubySpec https://en.wikipedia.org/wiki/RubySpec). That makes it about half the age of ruby. Worse, Wikipedia claims "The RubySpec project was discontinued at the end of 2014 due to a lack of uptake from mainstream ruby developers"
  - riffraff 10y ago
    I don't understand what you're arguing against, given your own words say it didn't have uptake (in MRI) till 2014. The project has been revived and adopted officially by MRI only in 2015. http://eregon.github.io/rubyspec/2015/07/29/rubyspec-is-reborn.html http://eregon.github.io/rubyspec/2015/07/29/rubyspec-is-rebo...
znpy 10y ago
There has been quite a controversy about rubyspec afaik: the tldr is that the de-facto ruby spec is basically 'whatever piece of software capable of running rails'. This is because the ruby devs change stuff around very often and don't keep an official spec of the language, to the extent that the creator of rubyspec had given up. This might have changed in the meantime, dunno. https://github.com/rubinius/rubinius-archive/blob/cf54187d421275eec7d2db0abd5d4c059755b577/_posts/2014-12-31-matz-s-ruby-developers-don-t-use-rubyspec.markdown https://github.com/rubinius/rubinius-archive/blob/cf54187d42...
- shellac 10y ago
  To be honest the issues with rubyspec in the past were sociological more than anything else. It was a right pain making contributions, and certainly not a welcoming community. Specs have been part of ruby core for quite a while now, thanks to the efforts of other members of the community.
- riffraff 10y ago
  it has indeed changed, the project has been reignited as "The Ruby Spec Suite". It still doesn't cover everything, but it's used and contributed to by multiple implementations, and looks healthy https://github.com/ruby/spec/graphs/contributors https://github.com/ruby/spec/graphs/contributors

by dkubb 10y ago

chrisseaton 10y ago

> Do they run tests against the top 100, 1000, etc rubygems before releasing? I don't believe they do. They only recently started testing against ruby spec. I work on an alternative implementation, and we are planning to test against literally all the gems in RubyGems ourselves. We test the top 100 or so at the moment.

dkubb 10y ago
If I might make a suggestion, I'd recommend doing some kind of tracing when you run the tests so you can have a mapping of what parts of your code are exercised by which gems/tests. Then when you make changes to your code you can run the tests that relate to that part of the system first, thus giving you more confidence that your change didn't cause a regression. This should be much more efficient than simply doing a full gem run (although I'd recommend that too, since it could uncover hidden dependencies).

riffraff 10y ago

ruby has always had its own set of tests, rubyspec is a very recent development.

Someone 10y ago
Very recent? 2006, according to Wikipedia (https://en.wikipedia.org/wiki/RubySpec https://en.wikipedia.org/wiki/RubySpec). That makes it about half the age of ruby. Worse, Wikipedia claims "The RubySpec project was discontinued at the end of 2014 due to a lack of uptake from mainstream ruby developers"
- riffraff 10y ago
  I don't understand what you're arguing against, given your own words say it didn't have uptake (in MRI) till 2014. The project has been revived and adopted officially by MRI only in 2015. http://eregon.github.io/rubyspec/2015/07/29/rubyspec-is-reborn.html http://eregon.github.io/rubyspec/2015/07/29/rubyspec-is-rebo...

znpy 10y ago

There has been quite a controversy about rubyspec afaik: the tldr is that the de-facto ruby spec is basically 'whatever piece of software capable of running rails'. This is because the ruby devs change stuff around very often and don't keep an official spec of the language, to the extent that the creator of rubyspec had given up. This might have changed in the meantime, dunno. https://github.com/rubinius/rubinius-archive/blob/cf54187d421275eec7d2db0abd5d4c059755b577/_posts/2014-12-31-matz-s-ruby-developers-don-t-use-rubyspec.markdown https://github.com/rubinius/rubinius-archive/blob/cf54187d42...

shellac 10y ago
To be honest the issues with rubyspec in the past were sociological more than anything else. It was a right pain making contributions, and certainly not a welcoming community. Specs have been part of ruby core for quite a while now, thanks to the efforts of other members of the community.
riffraff 10y ago
it has indeed changed, the project has been reignited as "The Ruby Spec Suite". It still doesn't cover everything, but it's used and contributed to by multiple implementations, and looks healthy https://github.com/ruby/spec/graphs/contributors https://github.com/ruby/spec/graphs/contributors