6 ms·

> But we don't write tests to check if our code works. I write tests to check if my code works. And tests that document how the code is supposed to work curren

> But we don't write tests to check if our code works. I write tests to check if my code works. And tests that document how the code is supposed to work currently are usually enough to prevent code from breaking in the future. Anything related to privacy or security should be fully tested. But for the typical startup, I'd posit that beyond that test coverage should be more closely related to the number of users and level of usage rather than to the amount of code.

e40 10y ago
The number of times my tests have 1) found bugs in my code, and 2) exposed problems with the API I was about to publish... uncountable.
hannob 10y ago
> Anything related to privacy or security should be fully tested. How often do you write code that's not related to privacy or security? As soon as you connect something to the Internet it's related to privacy and security. The only situation where privacy and security don't matter a whole lot is if your code runs on airgapped devices with very limited tasks.
- EpicEng 10y ago
  I rarely write code that connects to anything, so almost never? There's a lot of code that's not written by web devs.
  - hannob 10y ago
    > I rarely write code that connects to anything, so almost never? Sounds hard to believe. What kind of code would that be? > There's a lot of code that's not written by web devs. There's also a lot more than the web that has some form of connectivity with the Internet (even if it's not directly connected it may still parse data that comes from untrusted sources). There is a widespread belief among many that "security is important, but doesn't matter for me". The most extreme example is obviously IoT ("Who would want to hack my coffeemaker?"), but there's a lot more. The unfortunate truth is: There is hardly any code these days that is not security relevant.
    - nickpsecurity 10y ago
      "Sounds hard to believe. What kind of code would that be?" You do realize there's a bajillion non-networked apps in existence? Word processors, excel, editors/IDE's, system tools (esp monitoring/backup), media players that don't download stuff, compression libraries, MATLAB-style tools for numeric analysis, and so on.
      - hannob 10y ago
        > You do realize there's a bajillion non-networked apps in existence? Word processors, excel, editors/IDE's, system tools (esp monitoring/backup), media players that don't download stuff, compression libraries, MATLAB-style tools for numeric analysis, and so on. All of them parse potentially untrusted inputs. They don't have to be directly network connected to be a security risk. Just pick the first example: A word processor. It is not a security risk only if you can guarantee that you'll only ever open documents that you created yourself. If you ever use it to open documents you got from someone else it needs to take security into consideration.
      - mwexler 10y ago
        Sadly, almost everything you mention involve opening files, and files are often on networks. Standard modal file|open on windows allows http:// http:// access to files on hosted servers, for example. Suddenly, all of these are potentially networked in the hands of users using the code for unintended purposes or in unexpected ways.
        Jtsummers 10y ago
        But you don't write those dialogs yourself. You write code that calls the system file open dialog. Then, presumably, you do some check on the file for sanity checking (is it the right type? does it parse correctly? does the internal size # equal the value the OS is claiming?). If those checks fail, you report an error. Otherwise, you process it. The source doesn't matter at that point. And if there's a glitch in the dialog for handling HTTP or similar sources, then that's on the system developers to correct, and on you to report if you discover it.
- tlrobinson 10y ago
  Most client-side web app code doesn't have much bearing on privacy/security, aside from avoiding XSS and a few other pitfalls.
- Alex3917 10y ago
  Right, I meant that in the context of things that tests are designed to catch. E.g. almost no one writes tests to look for unsafe eval or whatever. A good example is that I care a lot about making sure our search endpoint doesn't return private user data. But beyond that, I'd rather just know that the endpoint returns a 200 and let someone tell us if it's broken rather than have an extra three hundred lines of code to see if it's returning the correct results. If we get a ton of users then that will probably change, but for now the cost of writing and maintaining those extra tests wouldn't be worth the benefit.
- mannykannot 10y ago
  > The only situation where privacy and security don't matter a whole lot is if your code runs on airgapped devices with very limited tasks. That used to be true for the software in cars, but it no longer is. The problems that result are not the fault of the original authors; that belongs to the people who decided to bridge the airgap without thinking through the consequences.
aequitas 10y ago
> And tests that document how the code is supposed to work... For me the most important word here is 'supposed'. All the time I read documentation describe how code works step by step (what each 'if' does, but spelled out more verbose). And test that only test that a function does by mocking out everything else. But I don't care reading what code does. I can see that by looking at the code. I want to know what the developer intended/expected the code to do, so I can validate it against what the code actually does. Most of the time assumptions are made with those expectations. And with those expectations you have a much better idea why a trivial refactor of a piece of logic could unearth a massive 'undocumented feature'.
mannykannot 10y ago
More generally, testing effort should be distributed according to a risk analysis. Even a one-off statistical report can have serious consequences if far-reaching decisions are made on its basis.

e40 10y ago

The number of times my tests have 1) found bugs in my code, and 2) exposed problems with the API I was about to publish... uncountable.

hannob 10y ago

> Anything related to privacy or security should be fully tested. How often do you write code that's not related to privacy or security? As soon as you connect something to the Internet it's related to privacy and security. The only situation where privacy and security don't matter a whole lot is if your code runs on airgapped devices with very limited tasks.

EpicEng 10y ago
I rarely write code that connects to anything, so almost never? There's a lot of code that's not written by web devs.
- hannob 10y ago
  > I rarely write code that connects to anything, so almost never? Sounds hard to believe. What kind of code would that be? > There's a lot of code that's not written by web devs. There's also a lot more than the web that has some form of connectivity with the Internet (even if it's not directly connected it may still parse data that comes from untrusted sources). There is a widespread belief among many that "security is important, but doesn't matter for me". The most extreme example is obviously IoT ("Who would want to hack my coffeemaker?"), but there's a lot more. The unfortunate truth is: There is hardly any code these days that is not security relevant.
  - nickpsecurity 10y ago
    "Sounds hard to believe. What kind of code would that be?" You do realize there's a bajillion non-networked apps in existence? Word processors, excel, editors/IDE's, system tools (esp monitoring/backup), media players that don't download stuff, compression libraries, MATLAB-style tools for numeric analysis, and so on.
    - hannob 10y ago
      > You do realize there's a bajillion non-networked apps in existence? Word processors, excel, editors/IDE's, system tools (esp monitoring/backup), media players that don't download stuff, compression libraries, MATLAB-style tools for numeric analysis, and so on. All of them parse potentially untrusted inputs. They don't have to be directly network connected to be a security risk. Just pick the first example: A word processor. It is not a security risk only if you can guarantee that you'll only ever open documents that you created yourself. If you ever use it to open documents you got from someone else it needs to take security into consideration.
    - mwexler 10y ago
      Sadly, almost everything you mention involve opening files, and files are often on networks. Standard modal file|open on windows allows http:// http:// access to files on hosted servers, for example. Suddenly, all of these are potentially networked in the hands of users using the code for unintended purposes or in unexpected ways.
      - Jtsummers 10y ago
        But you don't write those dialogs yourself. You write code that calls the system file open dialog. Then, presumably, you do some check on the file for sanity checking (is it the right type? does it parse correctly? does the internal size # equal the value the OS is claiming?). If those checks fail, you report an error. Otherwise, you process it. The source doesn't matter at that point. And if there's a glitch in the dialog for handling HTTP or similar sources, then that's on the system developers to correct, and on you to report if you discover it.
tlrobinson 10y ago
Most client-side web app code doesn't have much bearing on privacy/security, aside from avoiding XSS and a few other pitfalls.
Alex3917 10y ago
Right, I meant that in the context of things that tests are designed to catch. E.g. almost no one writes tests to look for unsafe eval or whatever. A good example is that I care a lot about making sure our search endpoint doesn't return private user data. But beyond that, I'd rather just know that the endpoint returns a 200 and let someone tell us if it's broken rather than have an extra three hundred lines of code to see if it's returning the correct results. If we get a ton of users then that will probably change, but for now the cost of writing and maintaining those extra tests wouldn't be worth the benefit.
mannykannot 10y ago
> The only situation where privacy and security don't matter a whole lot is if your code runs on airgapped devices with very limited tasks. That used to be true for the software in cars, but it no longer is. The problems that result are not the fault of the original authors; that belongs to the people who decided to bridge the airgap without thinking through the consequences.

aequitas 10y ago

> And tests that document how the code is supposed to work... For me the most important word here is 'supposed'. All the time I read documentation describe how code works step by step (what each 'if' does, but spelled out more verbose). And test that only test that a function does by mocking out everything else. But I don't care reading what code does. I can see that by looking at the code. I want to know what the developer intended/expected the code to do, so I can validate it against what the code actually does. Most of the time assumptions are made with those expectations. And with those expectations you have a much better idea why a trivial refactor of a piece of logic could unearth a massive 'undocumented feature'.

mannykannot 10y ago

More generally, testing effort should be distributed according to a risk analysis. Even a one-off statistical report can have serious consequences if far-reaching decisions are made on its basis.