12 ms·

Some of you might recognize NCC Group, the company that performed the cryptographic review. They acquired Matasano, tptacek's prior company. Collectively they h

Some of you might recognize NCC Group, the company that performed the cryptographic review. They acquired Matasano, tptacek's prior company. Collectively they have some of the most talented pentesters and cryptographic analysts. It's a safe bet that the (n+1)sec protocol has an excellent security rating on the basis that they only found one low-severity issue and two informational issues during the analysis. https://github.com/equalitie/np1sec/raw/master/doc/NCC_Group_Equalitie_N1Sec_Report_2017-07-21_v1.pdf https://github.com/equalitie/np1sec/raw/master/doc/NCC_Group... Note the caveat: NCC Group reviewed the full specification, however, only the portions of the library that lined up with the protocol were reviewed. NCC Group did not perform a line-by-line review of the entire library. Additionally, the review focused on the library and not a chat client implementing the library. This means that the theory is sound, but the library itself wasn't actually pentested. That isn't an indication that there are problems -- just that no one has looked for them yet. As a specific example: https://github.com/equalitie/np1sec/blob/05b73b506b83be9724c61bdd46973eee96fa2e31/src/message.cc#L159-L162 https://github.com/equalitie/np1sec/blob/05b73b506b83be9724c... It sounds like assessing memory errors was outside the scope of the security review, so it seems unlikely that anyone was thinking about questions like "is there a way for an attacker to trick buffer.size() into becoming -1 and DoSing the system?" It was focused mostly on the mathematical soundness of the cryptosystem. It's extraordinarily expensive to schedule a two-week pentest, but implementation errors are far more commonly exploited than attacking the properties of a cryptosystem directly. It might be good to schedule an additional pentest if possible. That said, this is mostly incidental. Congrats on shipping!

ec109685 9y ago
How much is extraordinarily expensive?
- EGreg 9y ago
  I think sakurity charges $35,000 for example.
  - baby 9y ago
    From Homakov's website: > Any audit starts at $38,000 for two weeks He's a one man team though.
- sillysaurus3 9y ago
  I wouldn't be surprised if this cryptographic review cost $40k without breaking a sweat. There's an interesting blog post just waiting to be written about why this makes sense. It's something you have to experience as an insider to understand. It's why most attempts at creating a startup to do it cheaply at scale have failed, for example. It ends up a nice consulting business for those lone-wolf pentesters, but not a startup. If you're going to be a startup like Matasano you have to be up front about the cost and not compete primarily on cost. The reason this pricing structure makes sense is because there are two aspects to security: 1. being secure, 2. security theater. Critically, both are valuable, and dismissing #2 is a mistake. If something goes wrong, you need that security audit document to fall back on. NCC Group's reputation is at stake. When they say there is only one low severity finding and some info findings, what they're really saying is that a bad guy has to be smarter than the reviewers Alex Bladucci, Jake Meredith and Andy Lee working together to break your system for two solid weeks. And even then the bad guy might get lucky or genuinely be smarter. That's 10 days that they cannot do anything else. You can see that if you have 100 clients in your pipeline, this quickly becomes a problem. It's why Matasano didn't fall over when they started reaching this scale -- their recruitment challenges ensured they had the talent to meet the demand. Literally anyone off the street could do them and become a pentester. Corollary: If you're going to review 100 companies' codebases, you either need to raise the price to shocking levels, or you need to recruit a small pentester army. NCC's strategy for the latter seems to be to buy them all. With effective results. As a dev, you'll recognize the classic problem here: you can't have a baby in less than 9 months, no matter how much you try to shift the schedule around. It takes two people two weeks to pentest a client. And if you don't have two people working to break a codebase/cryptosystem for two weeks, I would be very hesitant to call it secure. It takes almost a week to get your mind into a state where you understand the target well enough to exploit it in creative ways. (There are usually lots of easier un-creative exploits lurking though, so that week is typically still productive.) My $40k number might be off by $20k in either direction, but it's fundamentally a shocking number. The salespeoples' job is to make people understand the value. And to capture the big companies, who can afford to do pentests every release cycle. That leaves a massive gap that seems like an opportunity: YC startups. They've only recently gotten enough investment in the initial stages where spending that amount of money can make some kind of sense. Spending 10% of your runway is incredibly scary. I've spent a long time thinking of how to resolve this problem without much success. For example, I'd be available to do it less expensively, but I'm one person. If something goes wrong, you can't say "well sillysaurus pentested us." I've thought about starting a company to do this, but then you're back to square one: competing on price alone. That inevitably devolves into a consulting business, not a startup. And even then you won't be able to keep those kinds of prices that low for very long. So what do you do if you want to create a startup to fill this gap? Make tools, and let the tools do the work. Sadly, this is about as easy as making that sufficiently smart compiler that we've been working on for the last 60 years. You could argue that there is value in letting an automated tool scan your codebase, and that maybe you could throw deep learning at the problem to achieve some success. Maybe. I have an xss.txt file that contains exploits that work on almost every client I throw them at. But occasionally you'll run into a situation where it works, but only with a twist. And those twists seem very hard to automate. Pentesting is as creative as programming, so when you claim you can make a startup to solve "the security problem" (aka nobody can afford it), what you're saying is you either have a way to recruit lots of pentesters or you've figured out how to automate a field that relies on creativity at its core.
- tptacek 9y ago
  This is NCC Cryptography Practice, so "significantly more expensive than a normal NCC audit".
- dsacco 9y ago
  $10,000 - $12,000 per week is normal for an application security assessment. Go ahead and double that for a specialization like protocol cryptanalysis. Ballpark $40,000; maybe more because there were three consultants, not two. I wouldn't be surprised at the price until it hit $60,000; I'd be outright shocked at $100,000. If you have two or three consultants working on this full time, you're charging $7000 - $10,000 per consultant-week of time. Those numbers would only be shocking to someone who has never been a consultant or who has never engaged in an assessment; the margin is lower than it looks when you account for the peripheral staff aside from the consultants working on the engagement. Consulting for a variety of software specializations very easily hits five figures per week.
  - sillysaurus3 9y ago
    Well, the third pentester was a shadow, which is a wildcard in terms of value and probably wasn't an influence during the initial sales negotiations. Ballpark $40,000; maybe more because there were three consultants, not two. In this case it was a cryptographic review, so it's probably closer to $80k. I forgot to take that into account in my other comment. $40k would be a pretty standard price point for a regular pentest. NCC also incurs additional risk to their reputation when organizations publish documents like this, so another $20k might've been necessary to get them to this point. I'm not positive it works that way, but I know that publishing a document like this is generally more significant than the usual "here's a secret document" deliverable. There have only been 17 public reports in total, for example: https://www.nccgroup.trust/us/our-research/?research=Public+Reports https://www.nccgroup.trust/us/our-research/?research=Public+... So it's a bigger deal on both sides of the table. I think your $100k figure is pretty on-point. Or at least not unrealistic to be closing in on that.
EGreg 9y ago
This looks like it holds amazing promise! But how do you achieve fully decentralized group chat while at the same time knowing the ordering of the messages? As far as I know, CRDTs and other techniques are not resilient in yhe face of byzantine failures. For one writer and many readers we have scuttlebutt type solutions. For many writers we have blockchains as the state of the art. How does this protocol solve the problem? Let's say in a multiplayer game I claim that I made a move before I saw the game state change. Or I made 1000 moves. How do you prevent this?
- mobiletelephone 9y ago
  Because the threat model is different. Why would your friends and colleagues in a group chat want to tinker with the order of messages? Just trust a sender's timestamp, provided it's within X minutes.
  - EGreg 9y ago
    The idea is that you never trust input from other clients. If I am playing my friends in video games then what guarantee do I have someone isn't cheating?

ec109685 9y ago

How much is extraordinarily expensive?

EGreg 9y ago
I think sakurity charges $35,000 for example.
- baby 9y ago
  From Homakov's website: > Any audit starts at $38,000 for two weeks He's a one man team though.
sillysaurus3 9y ago
I wouldn't be surprised if this cryptographic review cost $40k without breaking a sweat. There's an interesting blog post just waiting to be written about why this makes sense. It's something you have to experience as an insider to understand. It's why most attempts at creating a startup to do it cheaply at scale have failed, for example. It ends up a nice consulting business for those lone-wolf pentesters, but not a startup. If you're going to be a startup like Matasano you have to be up front about the cost and not compete primarily on cost. The reason this pricing structure makes sense is because there are two aspects to security: 1. being secure, 2. security theater. Critically, both are valuable, and dismissing #2 is a mistake. If something goes wrong, you need that security audit document to fall back on. NCC Group's reputation is at stake. When they say there is only one low severity finding and some info findings, what they're really saying is that a bad guy has to be smarter than the reviewers Alex Bladucci, Jake Meredith and Andy Lee working together to break your system for two solid weeks. And even then the bad guy might get lucky or genuinely be smarter. That's 10 days that they cannot do anything else. You can see that if you have 100 clients in your pipeline, this quickly becomes a problem. It's why Matasano didn't fall over when they started reaching this scale -- their recruitment challenges ensured they had the talent to meet the demand. Literally anyone off the street could do them and become a pentester. Corollary: If you're going to review 100 companies' codebases, you either need to raise the price to shocking levels, or you need to recruit a small pentester army. NCC's strategy for the latter seems to be to buy them all. With effective results. As a dev, you'll recognize the classic problem here: you can't have a baby in less than 9 months, no matter how much you try to shift the schedule around. It takes two people two weeks to pentest a client. And if you don't have two people working to break a codebase/cryptosystem for two weeks, I would be very hesitant to call it secure. It takes almost a week to get your mind into a state where you understand the target well enough to exploit it in creative ways. (There are usually lots of easier un-creative exploits lurking though, so that week is typically still productive.) My $40k number might be off by $20k in either direction, but it's fundamentally a shocking number. The salespeoples' job is to make people understand the value. And to capture the big companies, who can afford to do pentests every release cycle. That leaves a massive gap that seems like an opportunity: YC startups. They've only recently gotten enough investment in the initial stages where spending that amount of money can make some kind of sense. Spending 10% of your runway is incredibly scary. I've spent a long time thinking of how to resolve this problem without much success. For example, I'd be available to do it less expensively, but I'm one person. If something goes wrong, you can't say "well sillysaurus pentested us." I've thought about starting a company to do this, but then you're back to square one: competing on price alone. That inevitably devolves into a consulting business, not a startup. And even then you won't be able to keep those kinds of prices that low for very long. So what do you do if you want to create a startup to fill this gap? Make tools, and let the tools do the work. Sadly, this is about as easy as making that sufficiently smart compiler that we've been working on for the last 60 years. You could argue that there is value in letting an automated tool scan your codebase, and that maybe you could throw deep learning at the problem to achieve some success. Maybe. I have an xss.txt file that contains exploits that work on almost every client I throw them at. But occasionally you'll run into a situation where it works, but only with a twist. And those twists seem very hard to automate. Pentesting is as creative as programming, so when you claim you can make a startup to solve "the security problem" (aka nobody can afford it), what you're saying is you either have a way to recruit lots of pentesters or you've figured out how to automate a field that relies on creativity at its core.
tptacek 9y ago
This is NCC Cryptography Practice, so "significantly more expensive than a normal NCC audit".
dsacco 9y ago
$10,000 - $12,000 per week is normal for an application security assessment. Go ahead and double that for a specialization like protocol cryptanalysis. Ballpark $40,000; maybe more because there were three consultants, not two. I wouldn't be surprised at the price until it hit $60,000; I'd be outright shocked at $100,000. If you have two or three consultants working on this full time, you're charging $7000 - $10,000 per consultant-week of time. Those numbers would only be shocking to someone who has never been a consultant or who has never engaged in an assessment; the margin is lower than it looks when you account for the peripheral staff aside from the consultants working on the engagement. Consulting for a variety of software specializations very easily hits five figures per week.
- sillysaurus3 9y ago
  Well, the third pentester was a shadow, which is a wildcard in terms of value and probably wasn't an influence during the initial sales negotiations. Ballpark $40,000; maybe more because there were three consultants, not two. In this case it was a cryptographic review, so it's probably closer to $80k. I forgot to take that into account in my other comment. $40k would be a pretty standard price point for a regular pentest. NCC also incurs additional risk to their reputation when organizations publish documents like this, so another $20k might've been necessary to get them to this point. I'm not positive it works that way, but I know that publishing a document like this is generally more significant than the usual "here's a secret document" deliverable. There have only been 17 public reports in total, for example: https://www.nccgroup.trust/us/our-research/?research=Public+Reports https://www.nccgroup.trust/us/our-research/?research=Public+... So it's a bigger deal on both sides of the table. I think your $100k figure is pretty on-point. Or at least not unrealistic to be closing in on that.

EGreg 9y ago

This looks like it holds amazing promise! But how do you achieve fully decentralized group chat while at the same time knowing the ordering of the messages? As far as I know, CRDTs and other techniques are not resilient in yhe face of byzantine failures. For one writer and many readers we have scuttlebutt type solutions. For many writers we have blockchains as the state of the art. How does this protocol solve the problem? Let's say in a multiplayer game I claim that I made a move before I saw the game state change. Or I made 1000 moves. How do you prevent this?

mobiletelephone 9y ago
Because the threat model is different. Why would your friends and colleagues in a group chat want to tinker with the order of messages? Just trust a sender's timestamp, provided it's within X minutes.
- EGreg 9y ago
  The idea is that you never trust input from other clients. If I am playing my friends in video games then what guarantee do I have someone isn't cheating?