10 ms·

I changed the word "shill" in an edit right after I posted as that was unfair/unhelpful and realized you might think it was directed at you. It was not. I welco

by lrvick 8y ago

I changed the word "shill" in an edit right after I posted as that was unfair/unhelpful and realized you might think it was directed at you. It was not. I welcome this type of debate personally. > Do you professionally audit software? I do as a matter of fact. I'll be honest it is normally much easier in closed products as I know what to look for. It is generally much harder to find flaws in popular open source systems as someone else has usally long beaten me to the low hanging fruit in critical codepaths. I have been burned and seen others burned so many times by closed software and teams that ship security regressions that I now personally use only open tools I can audit at some level, or can audit the auditing and reproducible build process. I have seen "security" companies cut corners too many times in order to feature farm to trust anything that won't let me see the source code. I have even professionally audited multiple systems on the spreadsheet myself, some of which I am aware of vulnerabilities for currently under embargo. So far you have cited examples which I not only didn't ignore but responded to in the form of counter examples. This is however turning into a open source vs closed source debate with subjective evidence, which in the end always boils down to if you have blind faith in a small group of people or not. In the spreadsheet I tried to only include things that could be fairly objective, but when we get into concepts of trust of binaries and their authors, it gets muddy to be sure, and we will end up choosing paths based on our own threat profiles and experience. This is why I tried to include -everything- in the spreadsheet I am aware of, so people with threat profiles different than mine can make informed choices.

tptacek 8y ago
You professionally audit secure messaging applications? For what kinds of vulnerabilities? I have turned down invitations to audit some of these applications because I didn't feel qualified to render an assessment (I've been doing professional software security assessment, of closed/open source applications, since 1996). Did you take money for those assessments? Which ones did you do? I'd like to take a closer look at some of them sometime.
- lrvick 8y ago
  I audit tools myself, my employers, or partners consider and if I find obvious flaws then I file bugs for those products and generally we either don't use them or limit their use. I feel qualified only to certify that a tool is obviously insecure, but I don't think -anyone- will ever be qualified to solo certify something as totally secure. (but sadly that is how clean audits are generally read). I have in the past hired 3-4 audit firms and each would find flaws the others did not, and that mine did not, but fail to spot flaws that mine did. No one has the full picture. The only relevant one not under embargo atm was my recent casual 2-3 hour audit of Lifesize which I found right away had a number of alarming issues the company would not address. Some of these may of been found to be non issues under further inspection, but there was more than enough to assert security was not a major focus of the platform. This cursory look was all it took for me to feel confident in ending any consideration of that product to protect the interests of the entity considering it. I reported all these to the company and there was no response for over 90 days and made my findings public after due warning. They are available here: https://gist.github.com/lrvick/6e600d8484cfb415d1e2b06e8b3452c2 https://gist.github.com/lrvick/6e600d8484cfb415d1e2b06e8b345... (reminder: this was only 2-3 hours of work and by all means grain of salt)
  - tptacek 8y ago
    I think you probably understand where I am coming from at this point. When I think about "audits" for secure messengers, I think about professional, specialized, contracted assessments with dedicated teams of experts (I'm a little biased, since that used to be my line of work). And my two objections to this discussion are: 1. I think LVH is right to point out that "audit" doesn't have much meaning in the document you've produced; it gloms together assessments of wildly differing depth and quality and condenses them all to a single pass/fail. 2. Secure messaging security is hard, much harder than secure transports (which you alluded to in the other subthread when you mentioned TLS). There are in fact not that many people in the world who can do a proper cryptographic assessment of a messenger at this point (not because it's prohibitively difficult; it should be well within the reach of everyone with a graduate degree in cryptography who enjoys coding --- rather, just because it's a specialized skill set that not many people have an opportunity to get good at). Like I said: I wouldn't say I'm qualified to perform such an assessment (LVH, different story). And all that "just" gets you the cryptography! If your messenger gets popular, the framework it's built on becomes one of the most important targets on the Internet! So I get itchy when people say things that amount to "I've audited things, I have an authoritative opinion about this stuff". Probably not? Maybe, but, like, if we were going to place a bet...? You can turn that logic around on me. But I'm not the one making an extraordinary claim. You are. So: what really bugs me is the idea that this is a problem domain you can reduce to a Wikipedia-style comparison chart. That kind of chart will get people hurt.

by lrvick 8y ago

tptacek 8y ago

You professionally audit secure messaging applications? For what kinds of vulnerabilities? I have turned down invitations to audit some of these applications because I didn't feel qualified to render an assessment (I've been doing professional software security assessment, of closed/open source applications, since 1996). Did you take money for those assessments? Which ones did you do? I'd like to take a closer look at some of them sometime.

lrvick 8y ago
I audit tools myself, my employers, or partners consider and if I find obvious flaws then I file bugs for those products and generally we either don't use them or limit their use. I feel qualified only to certify that a tool is obviously insecure, but I don't think -anyone- will ever be qualified to solo certify something as totally secure. (but sadly that is how clean audits are generally read). I have in the past hired 3-4 audit firms and each would find flaws the others did not, and that mine did not, but fail to spot flaws that mine did. No one has the full picture. The only relevant one not under embargo atm was my recent casual 2-3 hour audit of Lifesize which I found right away had a number of alarming issues the company would not address. Some of these may of been found to be non issues under further inspection, but there was more than enough to assert security was not a major focus of the platform. This cursory look was all it took for me to feel confident in ending any consideration of that product to protect the interests of the entity considering it. I reported all these to the company and there was no response for over 90 days and made my findings public after due warning. They are available here: https://gist.github.com/lrvick/6e600d8484cfb415d1e2b06e8b3452c2 https://gist.github.com/lrvick/6e600d8484cfb415d1e2b06e8b345... (reminder: this was only 2-3 hours of work and by all means grain of salt)
- tptacek 8y ago
  I think you probably understand where I am coming from at this point. When I think about "audits" for secure messengers, I think about professional, specialized, contracted assessments with dedicated teams of experts (I'm a little biased, since that used to be my line of work). And my two objections to this discussion are: 1. I think LVH is right to point out that "audit" doesn't have much meaning in the document you've produced; it gloms together assessments of wildly differing depth and quality and condenses them all to a single pass/fail. 2. Secure messaging security is hard, much harder than secure transports (which you alluded to in the other subthread when you mentioned TLS). There are in fact not that many people in the world who can do a proper cryptographic assessment of a messenger at this point (not because it's prohibitively difficult; it should be well within the reach of everyone with a graduate degree in cryptography who enjoys coding --- rather, just because it's a specialized skill set that not many people have an opportunity to get good at). Like I said: I wouldn't say I'm qualified to perform such an assessment (LVH, different story). And all that "just" gets you the cryptography! If your messenger gets popular, the framework it's built on becomes one of the most important targets on the Internet! So I get itchy when people say things that amount to "I've audited things, I have an authoritative opinion about this stuff". Probably not? Maybe, but, like, if we were going to place a bet...? You can turn that logic around on me. But I'm not the one making an extraordinary claim. You are. So: what really bugs me is the idea that this is a problem domain you can reduce to a Wikipedia-style comparison chart. That kind of chart will get people hurt.