3 ms·

Most embedded engineer types know nothing about and never think about security. One example I saw once was an FTP server where the auth commands worked but were

by api 11y ago

Most embedded engineer types know nothing about and never think about security. One example I saw once was an FTP server where the auth commands worked but were irrelevant. All commands always worked. It passed the unit tests therefore it was good.

gardano 11y ago
I started taking a deep-dive look at iOS HomeKit, and was alarmed at the security implications that would be on my shoulders. I'm not an embedded engineer type, so the security burden I saw being hefted upon my shoulders felt much more onerous than the typical REST type API I've been accustomed to. At this point, I cannot imagine writing an app using IoT while at the same time feeling confident in understanding every bad security use-case. Perhaps `IoT Security Expert` is an up-and-coming career path, but it's not a job description for the faint of heart nor for the one-off developer to be able to approach with any sense of confidence.
jlgaddis 11y ago
I'm not even a developer so I know very little about unit tests but I would assume that in something with an authentication system you would have a unit test that verifies that known invalid credentials don't permit access.
- api 11y ago
  This one just validated that good credentials worked and bad ones didn't. It passed of course. Good to ship.
- Symbiote 11y ago
  Verifying that invalid credentials are rejected by the part that checks credentials is one thing, and I'd expect this to be an obvious test case. But to > verifies that known invalid credentials don't permit access you need a second test: "use invalid credentials, then try to download file, expect failure", i.e. a sequence of actions in a single test. That test is evidently missing from my heating system's web interface, which has exactly this problem. There's a box to type in a PIN code, and it lets you proceed / retry correctly. But it's not necessary to bother with this, you can change the temperature without authenticating if you know the URL.
- mikeash 11y ago
  It's extremely common to write unit tests that only test success cases but not error cases. It's pretty much a special case of confirmation bias, as illustrated by the classic "2-4-6 task" (implemented here as 2-4-8, but it's the same idea): http://www.nytimes.com/interactive/2015/07/03/upshot/a-quick-puzzle-to-test-your-problem-solving.html?_r=0 http://www.nytimes.com/interactive/2015/07/03/upshot/a-quick... You're totally right that you should have a unit test that verifies failure with invalid credentials, not just success with valid credentials. But a lot of programmers are not as smart as you are here!
  - gardano 11y ago
    I guess my worry is that I don't know what I don't know (regarding security), but I'm pretty sure there are use-cases that haven't even been defined yet (insert favourite Relmsfeld quote here).
    - mikeash 11y ago
      Security is hard, no doubt, and even careful tests may not turn up all problems. Not testing your negative cases at all is such a basic failure, and an extremely common one too.
- s_kilk 11y ago
  You'd think so, but no. I actually uncovered a bug in a JWT library once when I wrote a test for my application to verify that an invalid token would be rejected. Let's just say there was no corresponding test in the libraries test suite.

by api 11y ago

gardano 11y ago

I started taking a deep-dive look at iOS HomeKit, and was alarmed at the security implications that would be on my shoulders. I'm not an embedded engineer type, so the security burden I saw being hefted upon my shoulders felt much more onerous than the typical REST type API I've been accustomed to. At this point, I cannot imagine writing an app using IoT while at the same time feeling confident in understanding every bad security use-case. Perhaps `IoT Security Expert` is an up-and-coming career path, but it's not a job description for the faint of heart nor for the one-off developer to be able to approach with any sense of confidence.

jlgaddis 11y ago

I'm not even a developer so I know very little about unit tests but I would assume that in something with an authentication system you would have a unit test that verifies that known invalid credentials don't permit access.

api 11y ago
This one just validated that good credentials worked and bad ones didn't. It passed of course. Good to ship.
Symbiote 11y ago
Verifying that invalid credentials are rejected by the part that checks credentials is one thing, and I'd expect this to be an obvious test case. But to > verifies that known invalid credentials don't permit access you need a second test: "use invalid credentials, then try to download file, expect failure", i.e. a sequence of actions in a single test. That test is evidently missing from my heating system's web interface, which has exactly this problem. There's a box to type in a PIN code, and it lets you proceed / retry correctly. But it's not necessary to bother with this, you can change the temperature without authenticating if you know the URL.
mikeash 11y ago
It's extremely common to write unit tests that only test success cases but not error cases. It's pretty much a special case of confirmation bias, as illustrated by the classic "2-4-6 task" (implemented here as 2-4-8, but it's the same idea): http://www.nytimes.com/interactive/2015/07/03/upshot/a-quick-puzzle-to-test-your-problem-solving.html?_r=0 http://www.nytimes.com/interactive/2015/07/03/upshot/a-quick... You're totally right that you should have a unit test that verifies failure with invalid credentials, not just success with valid credentials. But a lot of programmers are not as smart as you are here!
- gardano 11y ago
  I guess my worry is that I don't know what I don't know (regarding security), but I'm pretty sure there are use-cases that haven't even been defined yet (insert favourite Relmsfeld quote here).
  - mikeash 11y ago
    Security is hard, no doubt, and even careful tests may not turn up all problems. Not testing your negative cases at all is such a basic failure, and an extremely common one too.
s_kilk 11y ago
You'd think so, but no. I actually uncovered a bug in a JWT library once when I wrote a test for my application to verify that an invalid token would be rejected. Let's just say there was no corresponding test in the libraries test suite.