3 ms·

I'm not even a developer so I know very little about unit tests but I would assume that in something with an authentication system you would have a unit test th

I'm not even a developer so I know very little about unit tests but I would assume that in something with an authentication system you would have a unit test that verifies that known invalid credentials don't permit access.

api 11y ago
This one just validated that good credentials worked and bad ones didn't. It passed of course. Good to ship.
Symbiote 11y ago
Verifying that invalid credentials are rejected by the part that checks credentials is one thing, and I'd expect this to be an obvious test case. But to > verifies that known invalid credentials don't permit access you need a second test: "use invalid credentials, then try to download file, expect failure", i.e. a sequence of actions in a single test. That test is evidently missing from my heating system's web interface, which has exactly this problem. There's a box to type in a PIN code, and it lets you proceed / retry correctly. But it's not necessary to bother with this, you can change the temperature without authenticating if you know the URL.
mikeash 11y ago
It's extremely common to write unit tests that only test success cases but not error cases. It's pretty much a special case of confirmation bias, as illustrated by the classic "2-4-6 task" (implemented here as 2-4-8, but it's the same idea): http://www.nytimes.com/interactive/2015/07/03/upshot/a-quick-puzzle-to-test-your-problem-solving.html?_r=0 http://www.nytimes.com/interactive/2015/07/03/upshot/a-quick... You're totally right that you should have a unit test that verifies failure with invalid credentials, not just success with valid credentials. But a lot of programmers are not as smart as you are here!
- gardano 11y ago
  I guess my worry is that I don't know what I don't know (regarding security), but I'm pretty sure there are use-cases that haven't even been defined yet (insert favourite Relmsfeld quote here).
  - mikeash 11y ago
    Security is hard, no doubt, and even careful tests may not turn up all problems. Not testing your negative cases at all is such a basic failure, and an extremely common one too.
s_kilk 11y ago
You'd think so, but no. I actually uncovered a bug in a JWT library once when I wrote a test for my application to verify that an invalid token would be rejected. Let's just say there was no corresponding test in the libraries test suite.

api 11y ago

This one just validated that good credentials worked and bad ones didn't. It passed of course. Good to ship.

Symbiote 11y ago

Verifying that invalid credentials are rejected by the part that checks credentials is one thing, and I'd expect this to be an obvious test case. But to > verifies that known invalid credentials don't permit access you need a second test: "use invalid credentials, then try to download file, expect failure", i.e. a sequence of actions in a single test. That test is evidently missing from my heating system's web interface, which has exactly this problem. There's a box to type in a PIN code, and it lets you proceed / retry correctly. But it's not necessary to bother with this, you can change the temperature without authenticating if you know the URL.

mikeash 11y ago

It's extremely common to write unit tests that only test success cases but not error cases. It's pretty much a special case of confirmation bias, as illustrated by the classic "2-4-6 task" (implemented here as 2-4-8, but it's the same idea): http://www.nytimes.com/interactive/2015/07/03/upshot/a-quick-puzzle-to-test-your-problem-solving.html?_r=0 http://www.nytimes.com/interactive/2015/07/03/upshot/a-quick... You're totally right that you should have a unit test that verifies failure with invalid credentials, not just success with valid credentials. But a lot of programmers are not as smart as you are here!

gardano 11y ago
I guess my worry is that I don't know what I don't know (regarding security), but I'm pretty sure there are use-cases that haven't even been defined yet (insert favourite Relmsfeld quote here).
- mikeash 11y ago
  Security is hard, no doubt, and even careful tests may not turn up all problems. Not testing your negative cases at all is such a basic failure, and an extremely common one too.

s_kilk 11y ago

You'd think so, but no. I actually uncovered a bug in a JWT library once when I wrote a test for my application to verify that an invalid token would be rejected. Let's just say there was no corresponding test in the libraries test suite.