5 ms·

Another huge problem with TDD is a lot of practitioners do not create negative tests. A negative test verifies that something the system shouldn't do is not be

Another huge problem with TDD is a lot of practitioners do not create negative tests. A negative test verifies that something the system shouldn't do is not being done. For example, if you shouldn't allow unauthenticated users unauthenticated users to delete something, there should be a test to verify that authenticated users can't delete something. But if you only add tests that will fail and then make succeed via implementation, you would probably never think to make those kind of test. You certainly can create negative tests with TDD, but most docs on TDD never discuss it.

jay_kyburz 5y ago
There should be a test looking for duplicate phrases. (I'm laughing with you not at you :))
deleted 5y ago
[deleted]
dqv 5y ago
This is how I’ve been making sure multitenancy in my app doesn’t leak data. It shouldn’t leak but how can I be sure? Run tests that try to access data on one tenant from another tenant and fail if the data is accessible.
- eb0la 5y ago
  Most of the time we make tests when something is in our radar. I guess you have use case in your spec that tells you that's the expected behaviour, or at least some clever developer figured out it should be done that way. But it is not always so "easy": Sometimes you don't have the domain knowledge, or the spec doesn't say a word about negative cases...
- onion2k 5y ago
  Run tests that try to access data on one tenant from another tenant and fail if the data is accessible. That tells you that the method your tests use to try to access data don't work, not that it's impossible to access data across tenants. If there's a leak you haven't tested for then your app is still insecure. It'd be better to think of a design that actually makes it impossible (assuming that's even possible for your scenario).
  - brabel 5y ago
    Exactly. Sometimes, good design is still the only real way of making sure something works or is secure. It's a bit like using a type system (though that's by all means not the only way) to ensure invariants.
    - necovek 5y ago
      Agreed: tests only test for discrete "points" of the input space (even with things like hypothesis testing and randomized input testing), and in general, in pure TDD, it is sufficient to test for one each of the class of points (eg. only one negative case to ensure negative part of the code is triggered). But we should not forget we are all working against legacy code bases and libraries outside our control, and sometimes, tests might be the best available tool for achieving something other than their primary purpose, which I'd happily accept as a fun hack.
  - dqv 5y ago
    I think it's well-designed, but I would definitely want someone to go over it and look for any obvious issues if I made it public. I wanted it to be annoying as possible to do something outside of the tenant context and it is definitely annoying. Absent an explicit flag to disable tenancy for a specific operation, the operation will fail, raising an exception that the tenant id was not set. This is mostly enforced at the database interface right now i.e. the interface ensures that a tenant id is present in the query. Then in the database itself, null values for the tenant id aren't allowed. I also have a test that runs over every table and ensures there is a tenant id column except for those that are explicitly listed in the test as not needing a tenant id. I also have it set up to not expose the tenant id to the user at all, instead opting for a "public id" with limited utility and only used where needed.
rmetzler 5y ago
This is not about TDD, this is about automated tests.
jrochkind1 5y ago
You can certainly sometimes write a test that fails first to have a test that an unauthenticated user can't delete something. You write the test before have implemented the authentication system. Either generally entirely, or just in the "delete something" subsystem. You write a test that says when unauthenticated user tries to delete something, an error of the appropriate kind happens. The test fails. Then you implement the authentication system to make the test pass. I have written this exact kind of test. I do not universally do "TDD" though, I probably write a failing test before I've written code less than 25% of the time. I do write what you call "negative" tests, sometimes TDD sometimes not -- I think I don't write "negative" tests TDD any less often than any other kind of test. I may not write "negative" tests enough, but it's not because of TDD (since I don't mostly write tests first!). So I'm not here being a universal TDD adherent. I guess I'm just being boring and unhelpful "it depends, and I just use my intuition built on years of developing." (I think this kind of intuition generally grows after years of developing on the same platform, so people switching tech all teh time doesn't help. But that' anotehr topic). I am aware "you just need to have skilled developers using their judgement" isn't particularly helpful. The attraction of TDD is the idea that you don't need that, a novice can follow this process and still achieve reliabile and robust results, efficiently. I don't really agree... but also think it's important to know about and have experienced TDD so you will have it in your toolbox to apply sometimes. I don't think "negative tests" is a particularly useful category for deciding when -- I have definitely done TDD usefully with "negative tests", you can totally write a failing test for something the system shoudln't do... in many circumstances. I agree that TDD in the hands of an inexperienced developer isn't a substitute for stepping back and thinking about design/architecture, you still need to think about design/architecture. But also that TDD has definitely sometimes helped me think about design/architecture -- sometimes in a really profoundly useful way. The very unhelpful "it depends".
projektfu 5y ago
If you would design it why wouldn't you test for it? If I were writing authentication and authorization I would test that unauthenticated and unauthorized activities produced the correct errors.
necovek 5y ago
While I've seen it happen, it's usually an honest mistake by a developer. What makes you feel it's "a lot of practitioners"?
AtlasBarfed 5y ago
The amount of code paths that even simply code can traverse can become vastly large quickly. Negative testing is often like shooting in the dark. Negative tests from what I've seen accrue by encountering unexpected results from code in the wild and writing tests to make sure they are handled. They are a result of code being used, not by proactive or predictive design.

jay_kyburz 5y ago

There should be a test looking for duplicate phrases. (I'm laughing with you not at you :))

deleted 5y ago

[deleted]

dqv 5y ago

This is how I’ve been making sure multitenancy in my app doesn’t leak data. It shouldn’t leak but how can I be sure? Run tests that try to access data on one tenant from another tenant and fail if the data is accessible.

eb0la 5y ago
Most of the time we make tests when something is in our radar. I guess you have use case in your spec that tells you that's the expected behaviour, or at least some clever developer figured out it should be done that way. But it is not always so "easy": Sometimes you don't have the domain knowledge, or the spec doesn't say a word about negative cases...
onion2k 5y ago
Run tests that try to access data on one tenant from another tenant and fail if the data is accessible. That tells you that the method your tests use to try to access data don't work, not that it's impossible to access data across tenants. If there's a leak you haven't tested for then your app is still insecure. It'd be better to think of a design that actually makes it impossible (assuming that's even possible for your scenario).
- brabel 5y ago
  Exactly. Sometimes, good design is still the only real way of making sure something works or is secure. It's a bit like using a type system (though that's by all means not the only way) to ensure invariants.
  - necovek 5y ago
    Agreed: tests only test for discrete "points" of the input space (even with things like hypothesis testing and randomized input testing), and in general, in pure TDD, it is sufficient to test for one each of the class of points (eg. only one negative case to ensure negative part of the code is triggered). But we should not forget we are all working against legacy code bases and libraries outside our control, and sometimes, tests might be the best available tool for achieving something other than their primary purpose, which I'd happily accept as a fun hack.
- dqv 5y ago
  I think it's well-designed, but I would definitely want someone to go over it and look for any obvious issues if I made it public. I wanted it to be annoying as possible to do something outside of the tenant context and it is definitely annoying. Absent an explicit flag to disable tenancy for a specific operation, the operation will fail, raising an exception that the tenant id was not set. This is mostly enforced at the database interface right now i.e. the interface ensures that a tenant id is present in the query. Then in the database itself, null values for the tenant id aren't allowed. I also have a test that runs over every table and ensures there is a tenant id column except for those that are explicitly listed in the test as not needing a tenant id. I also have it set up to not expose the tenant id to the user at all, instead opting for a "public id" with limited utility and only used where needed.

rmetzler 5y ago

This is not about TDD, this is about automated tests.

jrochkind1 5y ago

You can certainly sometimes write a test that fails first to have a test that an unauthenticated user can't delete something. You write the test before have implemented the authentication system. Either generally entirely, or just in the "delete something" subsystem. You write a test that says when unauthenticated user tries to delete something, an error of the appropriate kind happens. The test fails. Then you implement the authentication system to make the test pass. I have written this exact kind of test. I do not universally do "TDD" though, I probably write a failing test before I've written code less than 25% of the time. I do write what you call "negative" tests, sometimes TDD sometimes not -- I think I don't write "negative" tests TDD any less often than any other kind of test. I may not write "negative" tests enough, but it's not because of TDD (since I don't mostly write tests first!). So I'm not here being a universal TDD adherent. I guess I'm just being boring and unhelpful "it depends, and I just use my intuition built on years of developing." (I think this kind of intuition generally grows after years of developing on the same platform, so people switching tech all teh time doesn't help. But that' anotehr topic). I am aware "you just need to have skilled developers using their judgement" isn't particularly helpful. The attraction of TDD is the idea that you don't need that, a novice can follow this process and still achieve reliabile and robust results, efficiently. I don't really agree... but also think it's important to know about and have experienced TDD so you will have it in your toolbox to apply sometimes. I don't think "negative tests" is a particularly useful category for deciding when -- I have definitely done TDD usefully with "negative tests", you can totally write a failing test for something the system shoudln't do... in many circumstances. I agree that TDD in the hands of an inexperienced developer isn't a substitute for stepping back and thinking about design/architecture, you still need to think about design/architecture. But also that TDD has definitely sometimes helped me think about design/architecture -- sometimes in a really profoundly useful way. The very unhelpful "it depends".

projektfu 5y ago

If you would design it why wouldn't you test for it? If I were writing authentication and authorization I would test that unauthenticated and unauthorized activities produced the correct errors.

necovek 5y ago

While I've seen it happen, it's usually an honest mistake by a developer. What makes you feel it's "a lot of practitioners"?

AtlasBarfed 5y ago

The amount of code paths that even simply code can traverse can become vastly large quickly. Negative testing is often like shooting in the dark. Negative tests from what I've seen accrue by encountering unexpected results from code in the wild and writing tests to make sure they are handled. They are a result of code being used, not by proactive or predictive design.