5 ms·

I audit tools myself, my employers, or partners consider and if I find obvious flaws then I file bugs for those products and generally we either don't use them

by lrvick 8y ago

I audit tools myself, my employers, or partners consider and if I find obvious flaws then I file bugs for those products and generally we either don't use them or limit their use. I feel qualified only to certify that a tool is obviously insecure, but I don't think -anyone- will ever be qualified to solo certify something as totally secure. (but sadly that is how clean audits are generally read). I have in the past hired 3-4 audit firms and each would find flaws the others did not, and that mine did not, but fail to spot flaws that mine did. No one has the full picture. The only relevant one not under embargo atm was my recent casual 2-3 hour audit of Lifesize which I found right away had a number of alarming issues the company would not address. Some of these may of been found to be non issues under further inspection, but there was more than enough to assert security was not a major focus of the platform. This cursory look was all it took for me to feel confident in ending any consideration of that product to protect the interests of the entity considering it. I reported all these to the company and there was no response for over 90 days and made my findings public after due warning. They are available here: https://gist.github.com/lrvick/6e600d8484cfb415d1e2b06e8b3452c2 https://gist.github.com/lrvick/6e600d8484cfb415d1e2b06e8b345... (reminder: this was only 2-3 hours of work and by all means grain of salt)

tptacek 8y ago
I think you probably understand where I am coming from at this point. When I think about "audits" for secure messengers, I think about professional, specialized, contracted assessments with dedicated teams of experts (I'm a little biased, since that used to be my line of work). And my two objections to this discussion are: 1. I think LVH is right to point out that "audit" doesn't have much meaning in the document you've produced; it gloms together assessments of wildly differing depth and quality and condenses them all to a single pass/fail. 2. Secure messaging security is hard, much harder than secure transports (which you alluded to in the other subthread when you mentioned TLS). There are in fact not that many people in the world who can do a proper cryptographic assessment of a messenger at this point (not because it's prohibitively difficult; it should be well within the reach of everyone with a graduate degree in cryptography who enjoys coding --- rather, just because it's a specialized skill set that not many people have an opportunity to get good at). Like I said: I wouldn't say I'm qualified to perform such an assessment (LVH, different story). And all that "just" gets you the cryptography! If your messenger gets popular, the framework it's built on becomes one of the most important targets on the Internet! So I get itchy when people say things that amount to "I've audited things, I have an authoritative opinion about this stuff". Probably not? Maybe, but, like, if we were going to place a bet...? You can turn that logic around on me. But I'm not the one making an extraordinary claim. You are. So: what really bugs me is the idea that this is a problem domain you can reduce to a Wikipedia-style comparison chart. That kind of chart will get people hurt.
- lrvick 8y ago
  I don't know that I made any extrodinary claims. You did ask me if I audited things before and I answered honestly. I frequently report vulnerabilities in a range of open and closed systems and read a lot about those others find. I also don't consider myself an expert and generally distrust people that claim they are in this space because it is, as you say, really hard. I initially started a spreadsheet to document the very high level objective traits of messengers I find useful and others found it interesting/useful and helped extend it. It is far too much cognitive overhead to do deep dive evaluation of 75+ messengers, but what this list allows one to do is quickly eliminate services from consideration that lack features vital to address their use cases, platform targets, or threat profile: for instance the ability to self host, or end to end encryption. From there once you have your short list, you can then make more effective use of time reading code, doing audits, reviewing audits of others. If a list like this simply makes someone aware of new up and coming projects in the space, or old ones that have been quietly evolving in capability... then I feel justified in having shared it instead of having kept it as a private document of my personal notes. I for one learned about a number of new projects when putting this sheet together, and interesting new approaches on solving these problems. The thing that is pretty subjective about the list is how I sorted items based on my own research and threat profile. I won't sit here and say Matrix or briar or any other items near the top are perfect, or lacks any security flaws. I personally place my bets on Matrix at this point based on my deep dive evaluation of it and similarly featured alternatives, but that is subject to change! Others are free to share your view that a well funded but closed source messenger like whatsapp is the general best bet. To sober up on this topic a bit: Matrix has had glaring security flaws in the past, but also so have the options you personally recommend like Signal and Whatsapp. At the end of the day all one can do is collect data, look over all the options, deep dive into the relative merits and claims of each to the extent one can, look at the research provided by others, and make a judgement call. This list should be considered a starting point for research and a way to discover lesser known projects as it was for me. It should -not- be seen as an end all "use this it is most secure" recommendation by any means. Anyone that looks at any one high level list of binary data points and makes an exclusive choice on that alone is doomed to shoot themselves in the foot with or without my help. Hopefully that addresses your concerns about my intent here.

by lrvick 8y ago

tptacek 8y ago

I think you probably understand where I am coming from at this point. When I think about "audits" for secure messengers, I think about professional, specialized, contracted assessments with dedicated teams of experts (I'm a little biased, since that used to be my line of work). And my two objections to this discussion are: 1. I think LVH is right to point out that "audit" doesn't have much meaning in the document you've produced; it gloms together assessments of wildly differing depth and quality and condenses them all to a single pass/fail. 2. Secure messaging security is hard, much harder than secure transports (which you alluded to in the other subthread when you mentioned TLS). There are in fact not that many people in the world who can do a proper cryptographic assessment of a messenger at this point (not because it's prohibitively difficult; it should be well within the reach of everyone with a graduate degree in cryptography who enjoys coding --- rather, just because it's a specialized skill set that not many people have an opportunity to get good at). Like I said: I wouldn't say I'm qualified to perform such an assessment (LVH, different story). And all that "just" gets you the cryptography! If your messenger gets popular, the framework it's built on becomes one of the most important targets on the Internet! So I get itchy when people say things that amount to "I've audited things, I have an authoritative opinion about this stuff". Probably not? Maybe, but, like, if we were going to place a bet...? You can turn that logic around on me. But I'm not the one making an extraordinary claim. You are. So: what really bugs me is the idea that this is a problem domain you can reduce to a Wikipedia-style comparison chart. That kind of chart will get people hurt.

lrvick 8y ago
I don't know that I made any extrodinary claims. You did ask me if I audited things before and I answered honestly. I frequently report vulnerabilities in a range of open and closed systems and read a lot about those others find. I also don't consider myself an expert and generally distrust people that claim they are in this space because it is, as you say, really hard. I initially started a spreadsheet to document the very high level objective traits of messengers I find useful and others found it interesting/useful and helped extend it. It is far too much cognitive overhead to do deep dive evaluation of 75+ messengers, but what this list allows one to do is quickly eliminate services from consideration that lack features vital to address their use cases, platform targets, or threat profile: for instance the ability to self host, or end to end encryption. From there once you have your short list, you can then make more effective use of time reading code, doing audits, reviewing audits of others. If a list like this simply makes someone aware of new up and coming projects in the space, or old ones that have been quietly evolving in capability... then I feel justified in having shared it instead of having kept it as a private document of my personal notes. I for one learned about a number of new projects when putting this sheet together, and interesting new approaches on solving these problems. The thing that is pretty subjective about the list is how I sorted items based on my own research and threat profile. I won't sit here and say Matrix or briar or any other items near the top are perfect, or lacks any security flaws. I personally place my bets on Matrix at this point based on my deep dive evaluation of it and similarly featured alternatives, but that is subject to change! Others are free to share your view that a well funded but closed source messenger like whatsapp is the general best bet. To sober up on this topic a bit: Matrix has had glaring security flaws in the past, but also so have the options you personally recommend like Signal and Whatsapp. At the end of the day all one can do is collect data, look over all the options, deep dive into the relative merits and claims of each to the extent one can, look at the research provided by others, and make a judgement call. This list should be considered a starting point for research and a way to discover lesser known projects as it was for me. It should -not- be seen as an end all "use this it is most secure" recommendation by any means. Anyone that looks at any one high level list of binary data points and makes an exclusive choice on that alone is doomed to shoot themselves in the foot with or without my help. Hopefully that addresses your concerns about my intent here.