3 ms·

Example 1 bothers me. It says > This test has just made it impossible to introduce another active project without breaking it, even if the scope was not actual

by RHSeeger 10mo ago

Example 1 bothers me. It says > This test has just made it impossible to introduce another active project without breaking it, even if the scope was not actually broken. Add a new variant of an active project for an unrelated test and now you have to also update this test. And then goes on to test that the known active projects are indeed included in what the call to Project.active returns. However, that doesn't test that "active scope returns active projects". Rather, it tests that - active scope returns _at least some of the_ active projects, and And it does not test that - active scope returns _all_ of the active projects - active scope does not return non-active projects Which, admittedly, is only different because the original statement is ambiguous. But the difference is that the test will pass if it returns non-active projects, too; which probably is not the expected behavior. I prefer to set things up so that my test fixtures (test data) are created as close to the test as possible, and then test it in the way the article is saying is wrong (in some cases)... ie, test that the call to Project.active returns _only_ those projects that should be active. Another option would be to have 3 different tests that test all those things, but the second one (_all_ of the active projects) is going to fail if the text fixture changes to include more active projects.

radanskoric 10mo ago
Author here. Thanks for writing up your thoughts on this! The "doesn't include non-active projects objections is easy", please check the Example 1 test again, there's a line for that: ``` refute_includes active_projects, projects(:inactive) ``` Hm, if you missed it, perhaps I should have emphasised this part more, maybe add a blank line before it ... Regarding the fact that the test does not check that the scope returns "all" active projects, that's a bit more complex to address but let me let tell you how I'm thinking about it: The point of tests is to validate expected behaviours and prevent regressions (i.e. breaking old behaviour when introducing new features). It is impossible for tests to do this 100%. E.g. even if you test that the scope returns all active projects present in the fixtures that doesn't guarantee that the scope always returns all active projects for any possible list of active projects. If you want 100% validation your only choice is to turn to formal proof methods but that's whole different topic. You could always add more active project examples. When you write a test that is checking that "Active projects A,B and C" are returned that is the same test as if your fixtures contained ONLY active projects A,B and C and then you tested that all of them are returned. In either case it is up to you to make sure that the projects are representative. So by rewriting the test to check: 1. These example projects are included. 2. These other example projects are excluded. You can write a test that is equally powerful as if you restricted your fixtures just to those example projects and then made an absolute comparison. You're not loosing any testing power. Expect you're making the test easier to maintain. Does that make sense? Let me know which part is still confusing and I'll try to rephrase the explanation.
- RHSeeger 10mo ago
  I want to start by saying that I agree with what you're trying to accomplish here. And I agree with some of the ways you go about it. I'm trying to find the right words to covey what I mean here, but... the best I can come with is... what I'm saying here isn't "you're wrong because", it's "what you're doing seems to miss some situations; here's what I do that helps for those". > The "doesn't include non-active projects objections is easy", please check the Example 1 test again, there's a line for that: You're correct; I totally missed that. > In either case it is up to you to make sure that the projects are representative. That's fair, but that's also the point you're trying to address / make more robust by how you're trying to write tests (what the article is about). Specifically - The article is about: How to make sure you're tests are robust against test fixtures changing - That comment says: It's up to you to make sure your test fixtures don't change in a way that breaks your tests > You can write a test that is equally powerful as if you restricted your fixtures just to those example projects and then made an absolute comparison. You're not loosing any testing power. Expect you're making the test easier to maintain. By restricting your fixtures to just the projects (that are relevant to the test), you're making _the tests_ easier to maintain; not just the one test but the test harness as a whole. What I mean is that you're reducing "action at a distance". When you modify the data for your test, you don't need to worry about what other tests, somewhere else, might also be impacted. Plus you do gain testing power, because you can test more things. For example, you can confirm it returns _every_ active project. All that being said, what I'm talking about relies on creating the test data local to the tests. And doing that has a cost (time, generally). So there's a tradeoff there.
  - radanskoric 10mo ago
    I think I'm getting what you mean and I almost completely agree with you, let me address one part, the only part where I don't agree: > Plus you do gain testing power, because you can test more things. For example, you can confirm it returns _every_ active project. Imagine this: 1. You start with some fixtures. You crafted the fixtures and you're happy that the fixtures are good for the test you're about to write. 2. You write a test where you assert the EXACT collection that is returned. This is, as you say, a test that "confirms the scope returns _every_ active project". 3. You now rewrite the test so that it checks that the collection includes ALL active projects and excludes all inactive projects. Do you agree that nothing changed when you went from 2 to 3? As long as you don't change the fixtures, those 2 version of the test will behave exactly the same: if one passes so will the other and if one fails so will the other. As long as fixtures don't change they have exactly the same testing power. If you agree on that, now imagine that you added another project to the fixtures. Has the testing power of the tests changed just because fixtures have been changed?
jon-wood 10mo ago
Strongly agree with that. Its slower, but I will always prefer building actual database records as either part of the test or in the test context rather than relying on some predefined fixtures. That makes the test behaviour clearer, and it means you don't have a bunch of unrelated tests failing because someone changed a fixture to accommodate a new test.

radanskoric 10mo ago

Author here. Thanks for writing up your thoughts on this! The "doesn't include non-active projects objections is easy", please check the Example 1 test again, there's a line for that: ``` refute_includes active_projects, projects(:inactive) ``` Hm, if you missed it, perhaps I should have emphasised this part more, maybe add a blank line before it ... Regarding the fact that the test does not check that the scope returns "all" active projects, that's a bit more complex to address but let me let tell you how I'm thinking about it: The point of tests is to validate expected behaviours and prevent regressions (i.e. breaking old behaviour when introducing new features). It is impossible for tests to do this 100%. E.g. even if you test that the scope returns all active projects present in the fixtures that doesn't guarantee that the scope always returns all active projects for any possible list of active projects. If you want 100% validation your only choice is to turn to formal proof methods but that's whole different topic. You could always add more active project examples. When you write a test that is checking that "Active projects A,B and C" are returned that is the same test as if your fixtures contained ONLY active projects A,B and C and then you tested that all of them are returned. In either case it is up to you to make sure that the projects are representative. So by rewriting the test to check: 1. These example projects are included. 2. These other example projects are excluded. You can write a test that is equally powerful as if you restricted your fixtures just to those example projects and then made an absolute comparison. You're not loosing any testing power. Expect you're making the test easier to maintain. Does that make sense? Let me know which part is still confusing and I'll try to rephrase the explanation.

RHSeeger 10mo ago
I want to start by saying that I agree with what you're trying to accomplish here. And I agree with some of the ways you go about it. I'm trying to find the right words to covey what I mean here, but... the best I can come with is... what I'm saying here isn't "you're wrong because", it's "what you're doing seems to miss some situations; here's what I do that helps for those". > The "doesn't include non-active projects objections is easy", please check the Example 1 test again, there's a line for that: You're correct; I totally missed that. > In either case it is up to you to make sure that the projects are representative. That's fair, but that's also the point you're trying to address / make more robust by how you're trying to write tests (what the article is about). Specifically - The article is about: How to make sure you're tests are robust against test fixtures changing - That comment says: It's up to you to make sure your test fixtures don't change in a way that breaks your tests > You can write a test that is equally powerful as if you restricted your fixtures just to those example projects and then made an absolute comparison. You're not loosing any testing power. Expect you're making the test easier to maintain. By restricting your fixtures to just the projects (that are relevant to the test), you're making _the tests_ easier to maintain; not just the one test but the test harness as a whole. What I mean is that you're reducing "action at a distance". When you modify the data for your test, you don't need to worry about what other tests, somewhere else, might also be impacted. Plus you do gain testing power, because you can test more things. For example, you can confirm it returns _every_ active project. All that being said, what I'm talking about relies on creating the test data local to the tests. And doing that has a cost (time, generally). So there's a tradeoff there.
- radanskoric 10mo ago
  I think I'm getting what you mean and I almost completely agree with you, let me address one part, the only part where I don't agree: > Plus you do gain testing power, because you can test more things. For example, you can confirm it returns _every_ active project. Imagine this: 1. You start with some fixtures. You crafted the fixtures and you're happy that the fixtures are good for the test you're about to write. 2. You write a test where you assert the EXACT collection that is returned. This is, as you say, a test that "confirms the scope returns _every_ active project". 3. You now rewrite the test so that it checks that the collection includes ALL active projects and excludes all inactive projects. Do you agree that nothing changed when you went from 2 to 3? As long as you don't change the fixtures, those 2 version of the test will behave exactly the same: if one passes so will the other and if one fails so will the other. As long as fixtures don't change they have exactly the same testing power. If you agree on that, now imagine that you added another project to the fixtures. Has the testing power of the tests changed just because fixtures have been changed?

jon-wood 10mo ago

Strongly agree with that. Its slower, but I will always prefer building actual database records as either part of the test or in the test context rather than relying on some predefined fixtures. That makes the test behaviour clearer, and it means you don't have a bunch of unrelated tests failing because someone changed a fixture to accommodate a new test.