4 ms·

Consul looks awesome. The only thing I don't like about it is they chose to use a homebrew encryption scheme for Serf's gossip protocol: http://www.serfdom.io/

by bascule 12y ago

Consul looks awesome. The only thing I don't like about it is they chose to use a homebrew encryption scheme for Serf's gossip protocol: http://www.serfdom.io/docs/internals/security.html http://www.serfdom.io/docs/internals/security.html (If you're confused about how this relates to Consul, see this: http://www.consul.io/docs/internals/security.html http://www.consul.io/docs/internals/security.html) Worse, their justification for using this instead of standard schemes like (D)TLS seems to be that they don't need the sorts of features transport encryption normally needs because they lean on the protocol's state machine to provide some of the features transport encryption would normally provide for free, like replay attack avoidance. Not only does homebrewing encryption almost always carry with it design and implementation mistakes, but someone auditing the protocol not only needs to understand the cryptographic design, but how it interacts with the rest of the protocol's state machine. My advice would be to scrap their homebrew encryption and use (D)TLS

armon 12y ago
We did take a look at D(TLS) when implementing Serf and the associated encryption mechanism. D(TLS) is much better suited when you are doing point-to-point duplex communication. The gossip algorithm instead is doing peer-to-peer (N-to-N instead of 1-to-1) in a half-duplex model. Using DTLS would've been too heavy for our use case. We did develop the algorithm in the open (see this gist: https://gist.github.com/armon/7159161 https://gist.github.com/armon/7159161), and made a call on the community to provide feedback to help improve the design of the system. The crypto system used is actually pretty vanilla based on AES-GCM. We certainly didn't attempt to invent our crypto primitives, and instead stuck to the best practices around the most modern systems.
- jimktrains2 12y ago
  > The gossip algorithm instead is doing peer-to-peer (N-to-N instead of 1-to-1) in a half-duplex model. Unless you mean you're doing multicast, you're still doing 1-to-1 connections. > We did develop the algorithm in the open and made a call on the community to provide feedback to help improve the design of the system. I wouldn't trust myself, let alone many unknown people who are just volunteering their time and have no known credentials as a cryptographer. > instead stuck to the best practices around the most modern systems. Except for the whole "don't role your own crypto system" one. There are just so many things that can go wrong and when crypto fails it can do so very quietly. Additionally, if you ever want anyone to interoperate with you it'll just be a PITA for them, and depending on who it is, a PITA for you. The overhead of a proper protocol like (D)TLS isn't that much. > "On our production frontend machines, SSL/TLS accounts for less than 1% of the CPU load, less than 10 KB of memory per connection and less than 2% of network overhead. Many people believe that SSL/TLS takes a lot of CPU time and we hope the preceding numbers will help to dispel that." - Adam Langley, Google
  - armon 12y ago
    You are right, the communication is still unicast in nature. I should clarify to say that there isn't a persistent 1-to-1 communication, the nodes we gossip with are randomly selected on each interval. There is no connection or session establishment between peers. I guess it depends on your definition of roll your own. We didn't invent AES-GCM or implement it. We are using the implementation shipped with the Golang stdlib.
    - bascule 12y ago
      Nobody is questioning whether AES-GCM is a good algorithm or not, however you are using AES-GCM as part of a hand-rolled transport encryption protocol, and this is what's worrisome. Designing a transport encryption protocol is a difficult endeavor, and it seems you have skipped most of the steps (e.g. replay attack prevention) but suggest that it's irrelevant because other parts of the protocol provide security (e.g. the SWIM state machine). This makes your protocol difficult to audit: someone concerned about potential attacks can't just look at your protocol in isolation, but has to factor the underlying protocol state machine into the security of your transport encryption protocol.
    - jimktrains2 12y ago
      > There is no connection or session establishment between peers. OK, so I'm not understanding why isn't a 1-to-1 message, nor why DTLS isn't an option here. > I guess it depends on your definition of roll your own. We didn't invent AES-GCM or implement it. We are using the implementation shipped with the Golang stdlib. You need to read better. I didn't say "crypto primitives", I've said "crypto _system_". That includes everything, including primitives, key management, authentication, replay (which means your application protocol is now part of the crypto system, not a good sign), field concatenation for singing/hashing, &c. The most important thing is that when your improvised system fails, you will more-than-likely never know and it'll never cause any errors.
tptacek 12y ago
I'm pretty glad they're not using DTLS. TLS is the sane and (usually) correct choice for normal client-server protocols, but DTLS is not necessarily the conservative choice for group messaging systems. This isn't a particularly sophisticated cryptosystem --- AES-GCM with keys broadcasted between the trusted participants --- but it's simple and, while GCM is nobody's favorite AEAD, it is at least a formal AEAD mode, leaving not a whole lot of room to screw the basics up.

by bascule 12y ago

armon 12y ago

We did take a look at D(TLS) when implementing Serf and the associated encryption mechanism. D(TLS) is much better suited when you are doing point-to-point duplex communication. The gossip algorithm instead is doing peer-to-peer (N-to-N instead of 1-to-1) in a half-duplex model. Using DTLS would've been too heavy for our use case. We did develop the algorithm in the open (see this gist: https://gist.github.com/armon/7159161 https://gist.github.com/armon/7159161), and made a call on the community to provide feedback to help improve the design of the system. The crypto system used is actually pretty vanilla based on AES-GCM. We certainly didn't attempt to invent our crypto primitives, and instead stuck to the best practices around the most modern systems.

jimktrains2 12y ago
> The gossip algorithm instead is doing peer-to-peer (N-to-N instead of 1-to-1) in a half-duplex model. Unless you mean you're doing multicast, you're still doing 1-to-1 connections. > We did develop the algorithm in the open and made a call on the community to provide feedback to help improve the design of the system. I wouldn't trust myself, let alone many unknown people who are just volunteering their time and have no known credentials as a cryptographer. > instead stuck to the best practices around the most modern systems. Except for the whole "don't role your own crypto system" one. There are just so many things that can go wrong and when crypto fails it can do so very quietly. Additionally, if you ever want anyone to interoperate with you it'll just be a PITA for them, and depending on who it is, a PITA for you. The overhead of a proper protocol like (D)TLS isn't that much. > "On our production frontend machines, SSL/TLS accounts for less than 1% of the CPU load, less than 10 KB of memory per connection and less than 2% of network overhead. Many people believe that SSL/TLS takes a lot of CPU time and we hope the preceding numbers will help to dispel that." - Adam Langley, Google
- armon 12y ago
  You are right, the communication is still unicast in nature. I should clarify to say that there isn't a persistent 1-to-1 communication, the nodes we gossip with are randomly selected on each interval. There is no connection or session establishment between peers. I guess it depends on your definition of roll your own. We didn't invent AES-GCM or implement it. We are using the implementation shipped with the Golang stdlib.
  - bascule 12y ago
    Nobody is questioning whether AES-GCM is a good algorithm or not, however you are using AES-GCM as part of a hand-rolled transport encryption protocol, and this is what's worrisome. Designing a transport encryption protocol is a difficult endeavor, and it seems you have skipped most of the steps (e.g. replay attack prevention) but suggest that it's irrelevant because other parts of the protocol provide security (e.g. the SWIM state machine). This makes your protocol difficult to audit: someone concerned about potential attacks can't just look at your protocol in isolation, but has to factor the underlying protocol state machine into the security of your transport encryption protocol.
  - jimktrains2 12y ago
    > There is no connection or session establishment between peers. OK, so I'm not understanding why isn't a 1-to-1 message, nor why DTLS isn't an option here. > I guess it depends on your definition of roll your own. We didn't invent AES-GCM or implement it. We are using the implementation shipped with the Golang stdlib. You need to read better. I didn't say "crypto primitives", I've said "crypto _system_". That includes everything, including primitives, key management, authentication, replay (which means your application protocol is now part of the crypto system, not a good sign), field concatenation for singing/hashing, &c. The most important thing is that when your improvised system fails, you will more-than-likely never know and it'll never cause any errors.

tptacek 12y ago

I'm pretty glad they're not using DTLS. TLS is the sane and (usually) correct choice for normal client-server protocols, but DTLS is not necessarily the conservative choice for group messaging systems. This isn't a particularly sophisticated cryptosystem --- AES-GCM with keys broadcasted between the trusted participants --- but it's simple and, while GCM is nobody's favorite AEAD, it is at least a formal AEAD mode, leaving not a whole lot of room to screw the basics up.