6 ms·

I was hoping to see an explanation for why the pattern is prevalent, but the author does not know. "One of my fundamental rules of system design is when people

by ngneer 3y ago

I was hoping to see an explanation for why the pattern is prevalent, but the author does not know. "One of my fundamental rules of system design is when people keep doing it wrong, the people are right and your system or idea is wrong." I am not sure under what circumstances this is true, but for security it is definitely false. People keep forgetting to sanitize inputs, for example. Does that render input sanitization a broken idea?

guns 3y ago
> People keep forgetting to sanitize inputs, for example. Does that render input sanitization a broken idea? Possibly. You could prevent careless use of unsanitized inputs with a type system. Fail-safe designs should be more common.
ncallaway 3y ago
It’s actually a critical concept for security contexts. If everyone forgets to sanitize input, that means our current processes are fucking broken. I like to imagine if this excuse happened in aviation how it would sound. “Everyone forgets to put down landing gear before landing, so does that mean landing gear is a bad idea?” No, but it does mean the process for landing is catastrophically broken, and needs to be overhauled. If that happened twice, we’d implement strict rules around landing like checklists with standardized procedures to have a second person verifying that the process was followed and each task was accomplished, and defining a “sterile phase of flight” where talking about anything other than the landing process was disallowed. So, input sanitization isn’t a broken idea, but the fact that it keeps going wrong clearly demonstrates that out process and culture around security are catastrophically broken.
- ngneer 3y ago
  Presumably pilots know to do that? I absolutely agree that psychological acceptability is key for building secure systems, and that the human factor must not be neglected. I also agree that process and culture around security are deeply broken. I just do not agree with the idea that if all are doing it, there must be some deep truth beneath. Yes, people habits must not be ignored, but to claim that an idea that goes against those habits is wrong, as the author claims, is inaccurate.
  - JohnFen 3y ago
    > I just do not agree with the idea that if all are doing it, there must be some deep truth beneath. I think it does say that there's a deep truth underneath. That deep truth is that the "correct" way is not good enough. The way that it's not good enough may very well be that it doesn't mesh well with how people work or think. That aspect is as important as any other aspect of design.
    - ngneer 3y ago
      Interesting viewpoint. I can see why it indicates the "correct" way is not easy or foolproof enough, but this is all it says. Ease of use is certainly an important aspect of design. I just think "not good enough" and "wrong" are two different things.
      - JohnFen 3y ago
        Well, I think that if something is "not good enough" to the point where people actively avoid using it, that elevates it to "incorrect".
        deleted 3y ago
        [deleted]
  - lelanthran 3y ago
    > I just do not agree with the idea that if all are doing it, there must be some deep truth beneath. Hypothetically, lets say you design a system that, for every legitimate 1000 users accessing it, 999 fail to understand it enough to gain their legitimate access. Would you still be comfortable telling the people that paid for the system that it's not the system that's wrong, but the users?
    - ngneer 3y ago
      That is for sure a broken system. I would not feel comfortable telling users they are wrong, even if the numbers were 10 out of 1000 or less. But we might be talking about different things. I gave the example of developers building a hopefully secure system, not users using it. If your point is that developers are users of languages, and that 1 in 1000 succeeds in building a secure system, then indeed languages are broken. But are you really arguing against input sanitization?
Brian_K_White 3y ago
Yes. Or rather it means that requiring people to remember to do it is wrong. Maybe it's not possible to do any better (no magic language that can tell what a given string of bytes will eventually be used for) but that doesn't change the fact that the reason inputs don't get sanitized is because the system requires an unreliable component to be reliable. It's also a bad example because it's not really a choice (no magic language).
- ngneer 3y ago
  I agree, it is akin to requiring people to come up with high entropy passwords, another "brilliant" idea. However, the fact that there is no other choice and no magic language is what makes it a good example. Input sanitization is an oft-cited requirement because machines cannot (yet) do it for us. While it is not a great fix and people often get it wrong, it can be likened to a pilot checklist, to piggyback on another comment. No one says checklists are bad ideas, we all know humans tend to be forgetful.
  - Brian_K_White 3y ago
    I like the pilot checklist concept as an example of a way to address a problem when there can be no magic automatic way. It's a different way to address the problem of the unreliable component than just requiring it to be magically reliable, or replacing it with something else that's reliable (automation, ie magic compiler or runtime), by adding procedure and redundancy and cross verification so that the final output is reliable even though all the individual worker parts are not. It's just ECC in human scope. So I guess you're right that it is an example of the article's assertion that "If people always do something wrong, that is evidense that there is a system in place which results in people doing that thing wrong, and that system is wrong." The system is the overall norms of software development, and the wrongness is that it doesn't include something equivalent to a pilot checklist culture around the important bits, even though everyone knows that those bits are important, and knows that humans are unable to be reliable.
    - ngneer 3y ago
praisewhitey 3y ago
A system that requires users to sanitize inputs is indeed broken. System designers should make sure the system handles sanitation and not the end user.
jandrese 3y ago
This is a crucial failing in far too many security products. They see buggy instantiations of their API all over the place and start ranting about incompetent developers instead of asking themselves why otherwise smart people are struggling to use the API. The developers aren't all idiots, they are following your incomplete documentation and using the buggy and incomplete example code you provided. Or they are using the buggy example code they found online because you didn't provide any. If your API isn't useful without first taking a college course on the problem domain then that is a problem with your API and probably documentation. The whole point of an API is to encapsulate the complexity and provide the user a tool they can use without having to first learn everything about it.
- ngneer 3y ago
  There are a lot of poor quality products out there, and security products are no different. They too follow a distribution. I am not sure why the parent comment uses the second person. I too find security products to be a liability sometimes, and, if an API does not deliver, will try to switch it out.
lelanthran 3y ago
> I am not sure under what circumstances this is true, but for security it is definitely false. People keep forgetting to sanitize inputs, for example. Does that render input sanitization a broken idea? Well ... yes? If people are forgetting to do something that is required, then that something is explicitly needed. Better to make that "something" implicitly added in the process no matter what the user does[1]. Or make the process break if the user "forgets"[2]. All difficult things to be sure, but easier than expecting the user to remember which of the 100 different NON-DEFAULT sanitation packages to install, configure and use, for output to HTML, SQL, JSON, Logs, and more. If the language makes it easy to do, and there is very little blowback in terms of security[3], then users are gonna do it. [1] Auto-sanitise strings, obviously. I dunno how you'd actually do this though. [2] When running as a web-service, have Django/whatever-framework configure Python/whatever-language to emit warnings whenever string interpolation is used without any escape function. [3] For HTML, the exploitation of string interpolation are few and far between; they're so rare as to be lost in the noise. Hence, users don't use it. For SQL, injection was a real problem, with the risk of getting pwned on string interpolation being close to 100% on a good day (and actually 100% on a bad day), and so users actually used the mitigations there were.
- ngneer 3y ago
  I agree with you that an automatic default implementation is far better than leaving it to a human. I also agree with your examples on where and when it could be done. Lamentably, security is not free. Input sanitization is required because programs exhibit data-dependent behavior (which is what makes them useful), that can cause them to do weird things when working with weird data. We mostly tend to "forget" to ensure our programs exhibit "correct" behavior for all data, but sadly no one else can do it for us.
  - lelanthran 3y ago
    To be honest, I rarely forget about security, but I don't want to have to remember. What I want is to have my runtime/framework/language drop me into the pit of success (a popular phrase, might even have it's own Wikipedia page)
naet 3y ago
It does actually render "making people remember to sanitize every input" a poor system, if everyone is forgetting to do so. The input sanitization system could be improved by requiring sanitized inputs, reminding people that they need to sanitize inputs, automatically sanitizing inputs, etc. Clearly the current system is leading to a lot of insecurity.

by ngneer 3y ago

guns 3y ago

> People keep forgetting to sanitize inputs, for example. Does that render input sanitization a broken idea? Possibly. You could prevent careless use of unsanitized inputs with a type system. Fail-safe designs should be more common.

ncallaway 3y ago

It’s actually a critical concept for security contexts. If everyone forgets to sanitize input, that means our current processes are fucking broken. I like to imagine if this excuse happened in aviation how it would sound. “Everyone forgets to put down landing gear before landing, so does that mean landing gear is a bad idea?” No, but it does mean the process for landing is catastrophically broken, and needs to be overhauled. If that happened twice, we’d implement strict rules around landing like checklists with standardized procedures to have a second person verifying that the process was followed and each task was accomplished, and defining a “sterile phase of flight” where talking about anything other than the landing process was disallowed. So, input sanitization isn’t a broken idea, but the fact that it keeps going wrong clearly demonstrates that out process and culture around security are catastrophically broken.

ngneer 3y ago
Presumably pilots know to do that? I absolutely agree that psychological acceptability is key for building secure systems, and that the human factor must not be neglected. I also agree that process and culture around security are deeply broken. I just do not agree with the idea that if all are doing it, there must be some deep truth beneath. Yes, people habits must not be ignored, but to claim that an idea that goes against those habits is wrong, as the author claims, is inaccurate.
- JohnFen 3y ago
  > I just do not agree with the idea that if all are doing it, there must be some deep truth beneath. I think it does say that there's a deep truth underneath. That deep truth is that the "correct" way is not good enough. The way that it's not good enough may very well be that it doesn't mesh well with how people work or think. That aspect is as important as any other aspect of design.
  - ngneer 3y ago
    Interesting viewpoint. I can see why it indicates the "correct" way is not easy or foolproof enough, but this is all it says. Ease of use is certainly an important aspect of design. I just think "not good enough" and "wrong" are two different things.
    - JohnFen 3y ago
      Well, I think that if something is "not good enough" to the point where people actively avoid using it, that elevates it to "incorrect".
      - deleted 3y ago
        [deleted]
- lelanthran 3y ago
  > I just do not agree with the idea that if all are doing it, there must be some deep truth beneath. Hypothetically, lets say you design a system that, for every legitimate 1000 users accessing it, 999 fail to understand it enough to gain their legitimate access. Would you still be comfortable telling the people that paid for the system that it's not the system that's wrong, but the users?
  - ngneer 3y ago
    That is for sure a broken system. I would not feel comfortable telling users they are wrong, even if the numbers were 10 out of 1000 or less. But we might be talking about different things. I gave the example of developers building a hopefully secure system, not users using it. If your point is that developers are users of languages, and that 1 in 1000 succeeds in building a secure system, then indeed languages are broken. But are you really arguing against input sanitization?

Brian_K_White 3y ago

Yes. Or rather it means that requiring people to remember to do it is wrong. Maybe it's not possible to do any better (no magic language that can tell what a given string of bytes will eventually be used for) but that doesn't change the fact that the reason inputs don't get sanitized is because the system requires an unreliable component to be reliable. It's also a bad example because it's not really a choice (no magic language).

ngneer 3y ago
I agree, it is akin to requiring people to come up with high entropy passwords, another "brilliant" idea. However, the fact that there is no other choice and no magic language is what makes it a good example. Input sanitization is an oft-cited requirement because machines cannot (yet) do it for us. While it is not a great fix and people often get it wrong, it can be likened to a pilot checklist, to piggyback on another comment. No one says checklists are bad ideas, we all know humans tend to be forgetful.
- Brian_K_White 3y ago
  I like the pilot checklist concept as an example of a way to address a problem when there can be no magic automatic way. It's a different way to address the problem of the unreliable component than just requiring it to be magically reliable, or replacing it with something else that's reliable (automation, ie magic compiler or runtime), by adding procedure and redundancy and cross verification so that the final output is reliable even though all the individual worker parts are not. It's just ECC in human scope. So I guess you're right that it is an example of the article's assertion that "If people always do something wrong, that is evidense that there is a system in place which results in people doing that thing wrong, and that system is wrong." The system is the overall norms of software development, and the wrongness is that it doesn't include something equivalent to a pilot checklist culture around the important bits, even though everyone knows that those bits are important, and knows that humans are unable to be reliable.
  - ngneer 3y ago

praisewhitey 3y ago

A system that requires users to sanitize inputs is indeed broken. System designers should make sure the system handles sanitation and not the end user.

jandrese 3y ago

This is a crucial failing in far too many security products. They see buggy instantiations of their API all over the place and start ranting about incompetent developers instead of asking themselves why otherwise smart people are struggling to use the API. The developers aren't all idiots, they are following your incomplete documentation and using the buggy and incomplete example code you provided. Or they are using the buggy example code they found online because you didn't provide any. If your API isn't useful without first taking a college course on the problem domain then that is a problem with your API and probably documentation. The whole point of an API is to encapsulate the complexity and provide the user a tool they can use without having to first learn everything about it.

ngneer 3y ago
There are a lot of poor quality products out there, and security products are no different. They too follow a distribution. I am not sure why the parent comment uses the second person. I too find security products to be a liability sometimes, and, if an API does not deliver, will try to switch it out.

lelanthran 3y ago

> I am not sure under what circumstances this is true, but for security it is definitely false. People keep forgetting to sanitize inputs, for example. Does that render input sanitization a broken idea? Well ... yes? If people are forgetting to do something that is required, then that something is explicitly needed. Better to make that "something" implicitly added in the process no matter what the user does[1]. Or make the process break if the user "forgets"[2]. All difficult things to be sure, but easier than expecting the user to remember which of the 100 different NON-DEFAULT sanitation packages to install, configure and use, for output to HTML, SQL, JSON, Logs, and more. If the language makes it easy to do, and there is very little blowback in terms of security[3], then users are gonna do it. [1] Auto-sanitise strings, obviously. I dunno how you'd actually do this though. [2] When running as a web-service, have Django/whatever-framework configure Python/whatever-language to emit warnings whenever string interpolation is used without any escape function. [3] For HTML, the exploitation of string interpolation are few and far between; they're so rare as to be lost in the noise. Hence, users don't use it. For SQL, injection was a real problem, with the risk of getting pwned on string interpolation being close to 100% on a good day (and actually 100% on a bad day), and so users actually used the mitigations there were.

ngneer 3y ago
I agree with you that an automatic default implementation is far better than leaving it to a human. I also agree with your examples on where and when it could be done. Lamentably, security is not free. Input sanitization is required because programs exhibit data-dependent behavior (which is what makes them useful), that can cause them to do weird things when working with weird data. We mostly tend to "forget" to ensure our programs exhibit "correct" behavior for all data, but sadly no one else can do it for us.
- lelanthran 3y ago
  To be honest, I rarely forget about security, but I don't want to have to remember. What I want is to have my runtime/framework/language drop me into the pit of success (a popular phrase, might even have it's own Wikipedia page)

naet 3y ago

It does actually render "making people remember to sanitize every input" a poor system, if everyone is forgetting to do so. The input sanitization system could be improved by requiring sanitized inputs, reminding people that they need to sanitize inputs, automatically sanitizing inputs, etc. Clearly the current system is leading to a lot of insecurity.