9 ms·

Show HN: PBT – A property-based testing library for Ruby

196 points by ohbarye 2y ago

Hello HN, I introduce a property-based testing tool for Ruby. Ruby's ease of test-writing and rich ecosystem are widely acclaimed. However, property-based testing is not as widely used as in other languages such as Haskell or Elixir, and I think this is because there is no de facto testing tool. This gem currently not only has the basic functionality of stateless property-based testing but also has the following features: - Support verbose mode that allows you to see its shrinking procedure and algorithms. - Support several concurrency/parallel executions of each test. As of now, Ractor/Thread/Process are available. (The default is sequential, considering benchmark results and actual use cases) Happy hacking!

lijok 2y ago
Amazing work ! Happy to see any effort towards PBT.
- ohbarye 2y ago
  Thank you. I hope the gem will make your testing life happier!
cortesoft 2y ago
Can this be used alongside rspec? Like have these be tests that are included in the rspec test run, and have the results alongside your other rspec tests?
- ohbarye 2y ago
  The answer to both questions is yes. This gem doesn't aim to reimplement a testing framework. It can also be considered just an assertion. You can use this alongside any testing framework like RSpec, minitest, and etc.
jack_riminton 2y ago
Interesting! I wonder how this could be applied to a Rails project (since that's what ruby is most used for). For example how would it know to test a wide selection of possible values for different variables if they're not explicitly set up?
- Alifatisk 2y ago
  > Rails [...] that's what ruby is most used for I wish that wasn't the case, I use Ruby heavily for gluing different systems together.
  - mikepurvis 2y ago
    Outside of language level preferences, is there something about the ecosystem around Ruby that makes it particularly well suited to this? I would have thought Python was the glue language winner just because there are already bindings to the entire universe available for it.
    - deleted 2y ago
      [deleted]
    - portpecos 2y ago
      Ruby just simply isn’t the glue language winner because of the heavy emphasis on rails. From a systems perspective, I had to switch to python because it has pyroute2, which supports rtnl, devlink, ethtool and more. I would have thought ruby had a full-fledged netlink library right now considering the stability of chef and puppet. But all I could find was this from 8 years ago: https://github.com/BytemarkHosting/netlinkrb https://github.com/BytemarkHosting/netlinkrb I started off with ruby for systems glue but now I have a mix of python and ruby. I wish it was all ruby but the lack of updated “glue gems” and the prevalence of updated “glue eggs” means python really is the “glue language winner”.
      - Alifatisk 2y ago
        > all I could find was this from 8 years ago: https://github.com/BytemarkHosting/netlinkrb https://github.com/BytemarkHosting/netlinkrb Yeah, that's so typical. I've almost become used to this, seeing useful gems being very old. To me, I see that as either abandoned (which means I have to fork it and polish it) or the gem is considered complete.
      - sam2426679 2y ago
mberning 2y ago
Very cool. Wonder how much time this adds to a test suite as you incorporate it more. Might need to have some different test targets to keep things fast.
- ohbarye 2y ago
  Thank you. In my opinion, PBT should be used in combination with example-based testing. Besides, since example-based testing cases account for the majority of tests, I think it's rare that PBT's execution time is dominant. As for the combination and usages, refs: https://medium.com/criteo-engineering/introduction-to-property-based-testing-f5236229d237 https://medium.com/criteo-engineering/introduction-to-proper...
parentheses 2y ago
Very specific question: when tests fail due to a single example, you suggest creating a specific assertion with the given seed. I imagine the seed is used to generate data and depending on the order of your generators, it produces different results. For example, in: PBT.assert(seed = ..) do PBT.property(PBT.integer, PBT.string) ... end Would changing the order of parameters to `property` change the actual test case?
- tybug 2y ago
  (not OP but I would be surprised if the answer wasn't) yes, because you're changing the order in which the random draws are interpreted. But this isn't a problem in practice because you generally aren't changing the generator in the middle of debugging a failure.
  - parentheses 2y ago
    So, this means refactoring becomes potentially difficult. While the gem is still a great accomplishment and very useful, I'd have to engineer my way around this issue before using it with things like a Rails Model which could have changing shape. @OP: I wonder if the README (and possibly runner) should suggest writing a test-case that doesn't rely on PBT when the user wants to preserve a case for future testing. The issue here is that if you're saving a singular example and it represents a weird corner case, it's totally conceivable that a small change will result in an invisible change to that test case. Another idea: it'd be great if the test could simply take examples that are failing and add them to a `failing_examples.rb` or some such. I know I'd use a feature like this quite a bit.
    - deleted 2y ago
      [deleted]
    - ohbarye 2y ago
      I think there are usually three actions a programmer can take when PBT fails. - Create a test case that doesn't depend on PBT as you suggest. - Fix the production code being tested since its failure is an unexpected bug. - Fix the PBT itself. This means that the programmer has had wrong assumption for the test target. I think it's difficult for the tool to know which choice is the best on a failure. But if there's any good idea, I'd like to incorporate it. :)
bloopernova 2y ago
Can anyone recommend an introduction to PBT for someone who knows nothing about it? Edited to add: D'oh! There's an explanation in the linked repo.
- arialdomartini 2y ago
  You might appreciate this little series of posts, specifically thought to provide a smooth introduction: https://arialdomartini.github.io/property-testing https://arialdomartini.github.io/property-testing
- ohbarye 2y ago
  It's very basic content though, here's the slides of my presentation at RubyKaigi 2024. This also includes the PBT gems's explanations. https://speakerdeck.com/ohbarye/unlocking-potential-of-property-based-testing-with-ractor https://speakerdeck.com/ohbarye/unlocking-potential-of-prope... Speaking of my personal story, I used the fast-check documentation and Fred Herbert's book (their links are in the README) to study.
Lammy 2y ago
Excited to see libraries using Ractors as somebody who has also fallen in love with them ♡
- ohbarye 2y ago
  I can really empathize. Hope the Ruby ecosystem gets more compatible with Ractor.
hdjrkaofbdjwm 2y ago
Great work, looks really clean! Check out Minithesis [1] and the paper explaining internal shrinking [2]. It will strongly improve your library's shrinking capabilities and prevent users from having to write their own shrinkers for custom objects in most cases. [1] https://github.com/DRMacIver/minithesis https://github.com/DRMacIver/minithesis [2] https://drops.dagstuhl.de/entities/document/10.4230/LIPIcs.ECOOP.2020.13 https://drops.dagstuhl.de/entities/document/10.4230/LIPIcs.E...
- ohbarye 2y ago
  Wow, I didn't know about them. Thanks for letting me know!
Lio 2y ago
Lovely stuff, thank you. It would be cool if we could eventually make use of RBS or inline Sorbet or something better than both to get the types for property testing for free.
- ohbarye 2y ago
  Good point. There's already such a project to run PBT based on RBS. I'm also keeping an eye on this. https://github.com/ksss/raap https://github.com/ksss/raap
slackstation 2y ago
They missed the opportunity to name this library 'Double Shot'.
- ohbarye 2y ago
  I love seeing people suggest clever names I couldn't have thought of.

196 points by ohbarye 2y ago

lijok 2y ago

Amazing work ! Happy to see any effort towards PBT.

ohbarye 2y ago
Thank you. I hope the gem will make your testing life happier!

cortesoft 2y ago

Can this be used alongside rspec? Like have these be tests that are included in the rspec test run, and have the results alongside your other rspec tests?

ohbarye 2y ago
The answer to both questions is yes. This gem doesn't aim to reimplement a testing framework. It can also be considered just an assertion. You can use this alongside any testing framework like RSpec, minitest, and etc.

jack_riminton 2y ago

Interesting! I wonder how this could be applied to a Rails project (since that's what ruby is most used for). For example how would it know to test a wide selection of possible values for different variables if they're not explicitly set up?

Alifatisk 2y ago
> Rails [...] that's what ruby is most used for I wish that wasn't the case, I use Ruby heavily for gluing different systems together.
- mikepurvis 2y ago
  Outside of language level preferences, is there something about the ecosystem around Ruby that makes it particularly well suited to this? I would have thought Python was the glue language winner just because there are already bindings to the entire universe available for it.
  - deleted 2y ago
    [deleted]
  - portpecos 2y ago
    Ruby just simply isn’t the glue language winner because of the heavy emphasis on rails. From a systems perspective, I had to switch to python because it has pyroute2, which supports rtnl, devlink, ethtool and more. I would have thought ruby had a full-fledged netlink library right now considering the stability of chef and puppet. But all I could find was this from 8 years ago: https://github.com/BytemarkHosting/netlinkrb https://github.com/BytemarkHosting/netlinkrb I started off with ruby for systems glue but now I have a mix of python and ruby. I wish it was all ruby but the lack of updated “glue gems” and the prevalence of updated “glue eggs” means python really is the “glue language winner”.
    - Alifatisk 2y ago
      > all I could find was this from 8 years ago: https://github.com/BytemarkHosting/netlinkrb https://github.com/BytemarkHosting/netlinkrb Yeah, that's so typical. I've almost become used to this, seeing useful gems being very old. To me, I see that as either abandoned (which means I have to fork it and polish it) or the gem is considered complete.
    - sam2426679 2y ago

mberning 2y ago

Very cool. Wonder how much time this adds to a test suite as you incorporate it more. Might need to have some different test targets to keep things fast.

ohbarye 2y ago
Thank you. In my opinion, PBT should be used in combination with example-based testing. Besides, since example-based testing cases account for the majority of tests, I think it's rare that PBT's execution time is dominant. As for the combination and usages, refs: https://medium.com/criteo-engineering/introduction-to-property-based-testing-f5236229d237 https://medium.com/criteo-engineering/introduction-to-proper...

parentheses 2y ago

Very specific question: when tests fail due to a single example, you suggest creating a specific assertion with the given seed. I imagine the seed is used to generate data and depending on the order of your generators, it produces different results. For example, in: PBT.assert(seed = ..) do PBT.property(PBT.integer, PBT.string) ... end Would changing the order of parameters to `property` change the actual test case?

tybug 2y ago
(not OP but I would be surprised if the answer wasn't) yes, because you're changing the order in which the random draws are interpreted. But this isn't a problem in practice because you generally aren't changing the generator in the middle of debugging a failure.
- parentheses 2y ago
  So, this means refactoring becomes potentially difficult. While the gem is still a great accomplishment and very useful, I'd have to engineer my way around this issue before using it with things like a Rails Model which could have changing shape. @OP: I wonder if the README (and possibly runner) should suggest writing a test-case that doesn't rely on PBT when the user wants to preserve a case for future testing. The issue here is that if you're saving a singular example and it represents a weird corner case, it's totally conceivable that a small change will result in an invisible change to that test case. Another idea: it'd be great if the test could simply take examples that are failing and add them to a `failing_examples.rb` or some such. I know I'd use a feature like this quite a bit.
  - deleted 2y ago
    [deleted]
  - ohbarye 2y ago
    I think there are usually three actions a programmer can take when PBT fails. - Create a test case that doesn't depend on PBT as you suggest. - Fix the production code being tested since its failure is an unexpected bug. - Fix the PBT itself. This means that the programmer has had wrong assumption for the test target. I think it's difficult for the tool to know which choice is the best on a failure. But if there's any good idea, I'd like to incorporate it. :)

bloopernova 2y ago

Can anyone recommend an introduction to PBT for someone who knows nothing about it? Edited to add: D'oh! There's an explanation in the linked repo.

arialdomartini 2y ago
You might appreciate this little series of posts, specifically thought to provide a smooth introduction: https://arialdomartini.github.io/property-testing https://arialdomartini.github.io/property-testing
ohbarye 2y ago
It's very basic content though, here's the slides of my presentation at RubyKaigi 2024. This also includes the PBT gems's explanations. https://speakerdeck.com/ohbarye/unlocking-potential-of-property-based-testing-with-ractor https://speakerdeck.com/ohbarye/unlocking-potential-of-prope... Speaking of my personal story, I used the fast-check documentation and Fred Herbert's book (their links are in the README) to study.

Lammy 2y ago

Excited to see libraries using Ractors as somebody who has also fallen in love with them ♡

ohbarye 2y ago
I can really empathize. Hope the Ruby ecosystem gets more compatible with Ractor.

hdjrkaofbdjwm 2y ago

Great work, looks really clean! Check out Minithesis [1] and the paper explaining internal shrinking [2]. It will strongly improve your library's shrinking capabilities and prevent users from having to write their own shrinkers for custom objects in most cases. [1] https://github.com/DRMacIver/minithesis https://github.com/DRMacIver/minithesis [2] https://drops.dagstuhl.de/entities/document/10.4230/LIPIcs.ECOOP.2020.13 https://drops.dagstuhl.de/entities/document/10.4230/LIPIcs.E...

ohbarye 2y ago
Wow, I didn't know about them. Thanks for letting me know!

Lio 2y ago

Lovely stuff, thank you. It would be cool if we could eventually make use of RBS or inline Sorbet or something better than both to get the types for property testing for free.

ohbarye 2y ago
Good point. There's already such a project to run PBT based on RBS. I'm also keeping an eye on this. https://github.com/ksss/raap https://github.com/ksss/raap

slackstation 2y ago

They missed the opportunity to name this library 'Double Shot'.

ohbarye 2y ago
I love seeing people suggest clever names I couldn't have thought of.