16 ms·

Rolling Your Own Crypto

192 points by svenfaw 10y ago

rahrahrah 10y ago
> Erratum: Apparently, I don't know what I'm doing either. I committed a number of approximations, misdirections, and even errors in there. Some of them where pointed out by /u/sacundim on Reddit. If you thought you could trust me, read that comment. It's sobering. LOL brilliant. That was fast.
- neaden 10y ago
  I'm glad he put that up at the top instead of trying to hide it. The moral of the story appears to be indeed: Don't roll your own crypto.
  - inetsee 10y ago
    I think rolling your own crypto, as a learning exercise, can be very helpful when choosing which trusted crypto solutions to use for a particular use case.
    - lvh 10y ago
      Breaking crypto is certainly an excellent way to learn many kinds of crypto. The problem is that writing your own typically isn't enough by itself -- a lot of horribly broken crypto looks fine after casual inspection. You probably won't find bugs like CBC padding oracles or Bleichenbacher attacks all by yourself. You can, however, absolutely _implement_ them yourself. I recommend cryptopals, together with my own free book, Crypto 101 if you prefer a little prose versus trying to figure it out yourself.
  - loup-vaillant 10y ago
    I draw a different conclusion: don't roll crypto on your own. https://www.reddit.com/r/programming/comments/5iv1ti/rolling_your_own_crypto/dbg58iy/ https://www.reddit.com/r/programming/comments/5iv1ti/rolling... I have updated my article since, and also made clear you can still screw up even if you had help.
- oconnore 10y ago
  A lot of those criticisms are sort of unfair. For example, pointing out length extension attacks in a fabricated MAC construction not found in the article, especially when the OP explicitly recommends blake2b, which is not vulnerable to length extension (and also disclaims "How that magical MAC is built is beyond the scope of this paragraph."). Another example: criticizing that he mentions appending a MAC (you should choose an AEAD!)... when later he recommends using an AEAD.
  - dom0 10y ago
    AEAD normally only means actual AEAD constructions (think AES-GCM/OCB, Chacha20-Poly1305 - not combinations of primitives providing authentication (=EtM, eg. first encrypt with AES-CTR then MAC with some HMAC). But yeah, some of the criticism reads kinda nit-picky / "I'm going to find false things in here". But on the other hand he provides good hints how the article could be improved.
    - oconnore 10y ago
      The article recommends Chacha20 Poly1305, not some combination of primitives.
      - dom0 10y ago
        What do you mean by this then? > Another example: criticizing that he mentions appending a MAC (you should choose an AEAD!)... when later he recommends using an AEAD.
        oconnore 10y ago
        There is a section called "integrity: not optional", where he uses MACs (which I would consider to be conceptually simpler to understand than AEADs) to explain how integrity can be implemented. Later, in a section called "Level 2: choosing crypto", he recommends using ChaCha20 Poly1305. Anyway, this isn't really my crusade, I just think that if you read the informal blog post as an informal blog post, it's fine.
- Bartweiss 10y ago
  If nothing else, this was a hell of a lesson in just how hard rolling your own crypto really is.
- grkvlt 10y ago
  > Erratum: [...] Some of them where pointed out [...] And Muphry's Law[0] strikes again! [0] https://en.wikipedia.org/wiki/Muphry's_law https://en.wikipedia.org/wiki/Muphry's_law
ohyoutravel 10y ago
I am interested enough where I would read a long form article on this kind of stuff, and found this fascinating. But the end kind of left me feeling dejected, as the article I guess is just a parody. Which is fine, but I probably wouldn't have spent as much time trying to take in what I presumed the article was trying to teach. I would really enjoy an article like this that actually goes step-by-step so I could learn something like this start to finish (though I would never implement). From the article, "Last Step in Crypto:" Get a PhD in cryptography. You need to be an expert yourself if you ever hope to invent a primitive that works. Publish your shiny new primitive. It may withstand the merciless assaults of cryptanalists, and be vetted by the community. Wait. Your primitive needs to stand the test of time.
- Kurtz79 10y ago
  If you are interested in more than an article, this course is very good. https://www.coursera.org/learn/crypto https://www.coursera.org/learn/crypto Sorry to spoil it, but the conclusion will basically be the same as the article, as in "just don't".
  - StavrosK 10y ago
    Yes, but arriving at the conclusion is very fun. Although, don't hold your breath for Cryptography II. I've been waiting for it for years.
- j2kun 10y ago
  This may only be sort of relevant, but I write a math/programming blog on which I implemented a (very inefficient) elliptic curve crypto library from first principles. This includes all the math background, and implementations of a few working protocols. It might be a good starting point if you're looking for resources like this (exposition-wise, not in terms of a practical crypto solution). https://jeremykun.com/2014/02/08/introducing-elliptic-curves/ https://jeremykun.com/2014/02/08/introducing-elliptic-curves...
  - tptacek 10y ago
Exuma 10y ago
https://www.reddit.com/r/programming/comments/5iv1ti/rolling_your_own_crypto/dbbmwwo/ https://www.reddit.com/r/programming/comments/5iv1ti/rolling...
- shujito 10y ago
  yes, it says it in the very article
  - loup-vaillant 10y ago
    I had to admit I screwed up a little… I didn't expect merely writing about crypto would be so hard. Good thing I put this up for peer review. I have since revised my article. It should be less dangerous now.
edblarney 10y ago
The four most dangerous words in CS. But it's good that people look into it -> for fun and learning. Just not for realsies.
- bkanber 10y ago
  There's serious pushback against rolling your own crypto. Your attitude towards it is good, but the hivemind-y attitude is to never do it under any circumstances. I'm sure that attitude prevents many people from experimenting and becoming crypo experts in their own right. Roll your own crypto, but be aware of the stakes, and be prepared to drop it the second that the stakes become real.
  - pflats 10y ago
    >There's serious pushback against rolling your own crypto. Your attitude towards it is good, but the hivemind-y attitude is to never do it under any circumstances. I'm sure that attitude prevents many people from experimenting and becoming crypo experts in their own right. It's not a hivemind-y attitude, it's Software Engineering. If you cannot show that a bridge will support a given weight, you do not build the bridge. Period. You learn that in your first Engineering course, freshman year. It's part of the responsibility of being an Engineer. You can experiment with new bridge designs all you want, but until you can prove mathematically that the bridge is sound, you do not build the bridge. If you really want to roll your own bridge over the ditch in the your backyard, go for it, but think three times before inviting your friend to drive their truck over it, especially if you're not already an expert on bridges. Cryptography is not a small or simple subject. If you are not an expert in the field, it is incredibly unlikely that you will not know enough to critique your own work. Instead of starting out by rolling your own, start by studying.
    - bkanber 10y ago
      That's a reductionist ("Period.") and a strawman position. It's unfortunate that you invoked the responsibility of the engineer in your argument: I went to engineering school. I have two degrees in mechanical engineering from a top school, I've published peer reviewed papers, and have helped design hybrid cars for companies you'd recognize. I qualify as an engineer under every definition you can think of, understand firsthand the responsibilities of an engineer, have gone through rigorous safety training in order to test drive the cars that I've personally designed and built. And yet I still say that nothing you said here is at odds with my statement: know the stakes. But play anyway. You think engineers don't build wacky and dangerous shit for their friends to play with? Prototypes that stretch the limits of design and would be irresponsible to mass produce as-is? We do it all the time. It's part of the learning process. I've been physically hurt, shocked, burnt both by heat and chemicals, while working with other engineers' experiments. AND IT'S OK! It's ok to try your own solutions on your friends and peers while the stakes are low. Studying is only one half of learning. The other half is bringing textbook knowledge to the real world by building and prototyping and getting your peer group to throw hammers at it and see what happens. Engineering is not theory. Engineering is theory applied to the real world. You can't be an engineer if you don't have one foot on each side. Study on one side, build and play on the other. It definitely is a hivemind attitude, because the literal hundreds of other engineers that I know -- people who design actual bridges, and buildings, and medical devices, and cars, and oil rigs, and theater sets, things that hundreds of millions of people rely on to be safe -- don't share this attitude. Their attitude is: know the stakes. Most software projects go nowhere and are usually only used by a handful of people close to the developer. That's the definition of low stakes. And it's ok to experiment, and to roll your own crypto, when the stakes are low. I think it's damaging, and probably hubristic, to think that the stakes are high all the time. They're not.
niftich 10y ago
Quoting an older post of mine: The 'don't roll your own crypto' argument is mostly just shorthand to 'defer to the opinion of experts, use ready-made constructs when possible, and if not, then exercise caution when hooking crypto primitives together in unproven ways'. [1] Crypto code, like other library code, is question of trust. Do I trust Daniel Bernstein? Do I trust Joan Daemen, who is half of the AES team and a quarter of the Keccak team [2]? In practical matters, do I trust tptacek [3]? Yes, I trust them, until people more educated in cryptography than me cast enough doubt or prove it otherwise -- but you might have a different model of who you trust. But ultimately, you're the one who answers to your systems. It's also a bit like science where we come up with a hypothesis (this seems to work...) and then try very hard disprove it, so our knowledge evolves as we go. It's important to understand that there was a time when it was best practice to use certain primitives that are now considered broken, and this is okay -- assuming we all upgraded our systems since. For cases then that assumption can't hold true, we need to account for that risk. This is also why it's wise to go with cryptosystems that receive a good amount of peer scrutiny. Your homegrown secret sauce might indeed be super secure, but few will publish papers on how they can obtain collisions on a round-reduced version. It's network effects, it's 'given enough eyes' all over again. [1] https://news.ycombinator.com/item?id=12400040 https://news.ycombinator.com/item?id=12400040 [2] https://news.ycombinator.com/item?id=12766941 https://news.ycombinator.com/item?id=12766941 [3] https://gist.github.com/tqbf/be58d2d39690c3b366ad https://gist.github.com/tqbf/be58d2d39690c3b366ad
- mfukar 10y ago
  Would any body else put the same trust on whatever authority the external auditors or pentesters display?
  - lvh 10y ago
    Most cryptanalysis is public; there's little reason for public researchers to withhold cryptographic breaks. The only reason that'd happen is if you're an intelligence agency and you'd like to keep a vuln to yourself (those are less likely in crypto than in software in general). In that light, it'd be perhaps better to phrase trusting AES as "not distrusting the union of everyone who's tried to cryptanalyse AES", rather than trusting the Rijndael team specifically. In many cases, especially djb-brand crypto, there's even less need for trust: the way you derive the X25519 curve is extremely well defined (or, as djb puts it, "rigid").
slindsey 10y ago
Forget about creating your own crypto. I have had endless troubles just using crypto. Obscure options that aren't explained. Examples that don't actually work. C# had an option that I finally got to work but the documentation specifically said not to use it; but the other choices didn't work. I had to encrypt something in PHP then decrypt it in ColdFusion. Despite being the same algorithm, key, etc, it didn't work. Strangely, the exact same data in the other direction (CF->PHP) worked just fine. I'd just like the libraries I use to have a simple choice: Encrypt(data,key) and Decrypt(data,key) and be done.
- ori_b 10y ago
  Have you looked at Libnacl or Libsodium?
- dom0 10y ago
  If it's something sparkling new you probably want to use NaCl or for more involved use cases a Noise implementation.
  - tptacek 10y ago
    It's the opposite, I think: Noise is good for less-involved cases, the kinds of problems you would ordinarily solve by just using TLS. Nacl is better for when you need something customized to an application. Incidentally: Noise is great, but it's also fine to use TLS, if you're just a little careful. See also: https://gist.github.com/tqbf/be58d2d39690c3b366ad https://gist.github.com/tqbf/be58d2d39690c3b366ad
    - jjnoakes 10y ago
      The "Avoid /dev/random" recommendation, can you elaborate on that?
      - tptacek 10y ago
        https://sockpuppet.org/blog/2014/02/25/safely-generate-random-numbers/ https://sockpuppet.org/blog/2014/02/25/safely-generate-rando...
pselbert 10y ago
You have to applaud him for trying. Building anything as a learning exercise is an investment in yourself, and I'm encouraged when others are ambitious about it. We've all had "don't roll your own crypto" pounded into us, but nothing teaches a lesson as soundly as trying something yourself. I've tried, failed and learned a lesson from far less ambitions endeavors.
- bisby 10y ago
  a great rule would be "try to roll your own crypto, but only as a learning tool, don't try to use it"
  - stouset 10y ago
    Roll your own crypto, but for the love of god don't release it.
ivanhoe 10y ago
The only reason I'm aware of for rolling a custom crypto would be fighting against automated attacks. Even ridiculously simple protection can prevent a script from automatically collecting the data. If there's a lot of data streams and each requires human interaction to decrypt it, it can significantly raise the cost of breaking such data. In that case it's probably best to make it just an additional layer over some well-tested method.
- lvh 10y ago
  I'm not sure I understand the rationale. Automated attacks can't beat real crypto either -- so why even bother rolling your own? (And automated attacks can certainly see non-cryptographic encodings like Base64, Base32, ROT13...)
jwtadvice 10y ago
I like how the author organizes 'levels' of rolling cryptography, and generally agree with his assessment of how those strata should be organized.
lvh 10y ago
As one might expect[1], I have some thoughts. There are a few "layers" of rolling your own crypto. Often, people think they're not rolling their own crypto because they're using AES instead of some hand-rolled bizarro cipher. This is not that: the primitives the post references are solid (e.g. BLAKE2b, ChaCha20, X25519). There are definitely challenges with explaining crypto to programmers in a way that is non-scary and at the same time factually accurate; I walk that tightrope constantly with Crypto 101. There are significant factual inaccuracies in this post that matter and aren't just a temporary educational tool so you can get a concept across quickly. Someone on Reddit[2] already pointed out most of the ones that were glaring at me after a cursory review. To the author's credit, he's put that link up by the start of the post. There are at least two kinds of crypto education for programmers. One is practical advice or libs to help people build better applications with crypto. This means building better tools. Don't "pick Ed25519" -- pick a library like libsodium that did signatures right for you. Saying "compose ChaCha20 and append a MAC" isn't the best level of advice we can strive for. Instead, "use Fernet" and possibly "use libsodium's secretbox" is (although I think having to specify a nonce may not be the greatest default -- I'm working on fixing that). The other kind of crypto education for programmers is to help people break things. That can help you become an expert, but I don't think we should expect any meaningful percentage of programmers will spend a bunch of time finishing all 8 sets of Cryptopals. [3] It is, however, a reasonable assumption that a (super)majority of programmers will at some point touch some crypto: the question is what they find when they do that. Will they have secure password storage and not even notice because that was the right default in the software they were using, or will they have to cobble together their own? I for one am hoping it's the former. Fundamentally, there are two reasons why you don't want to roll your own crypto. One is because there's a good chance you'll mess it up. The other is because that's effort, and you are not going to do better than the programmers that already have done the hard work for you. This is sometimes true for primitives (don't generate your own FFDH prime/implement your own FFDH -- you will probably have small subgroups!), but especially true for high-level recipes at varying levels of complexity from authenticated encryption (you won't be better than OCB or secretbox) all the way to cryptographic ratchets. Those reasons put together form my main criticism for this article. Sure, it has fundamental inaccuracies, they're not trivial, and that's not OK. But even if it was technically perfect, it doesn't help the next piece of software be safer. [1]: I am a cryptographer. I did https://www.crypto101.io https://www.crypto101.io. I am Latacora's crypto nerd. [2]: https://www.reddit.com/r/programming/comments/5iv1ti/rolling_your_own_crypto/dbbmwwo/ https://www.reddit.com/r/programming/comments/5iv1ti/rolling... [3]: http://cryptopals.com http://cryptopals.com
deleted 10y ago
[deleted]
kjs3 10y ago
The problem with "roll your own crypto" isn't the rolling, per se, it's that there's about 5 orders of magnitude more people who are sure that they can do it right than people who actually can do it right.
jackgavigan 10y ago
I feel like no discussion of rolling one's own crypto is complete without a link to Renaissance artist Pieter Bruegel the Elder's painting of the subject (c. 1562): http://classicprogrammerpaintings.com/post/148027314949/we-rolled-our-own-crypto-pieter-bruegel-the http://classicprogrammerpaintings.com/post/148027314949/we-r...
danbruc 10y ago
Poly1305 is one of the few primitives out there that are provably secure. That doesn't sound right. I guess this should read something like »is provably as hard to break as the underlying block cipher« in the same way that that a hash function built using the Merkle-Damgård construction is provably as hard to break as the compression function used. The article easily gives the impression that Poly1305 is provably secure in the same way as one-time pad.
- loup-vaillant 10y ago
  I have thought about it and re-read the original paper. I stand by my claim. The truth of the matter is, poly1305, while provably secure, is almost as impractical as a one time pad: it relies on a shared random authentication key. Where does that key is supposed to come from? In practice, you'd derive it from a session key (and so rely on a symmetric cipher such as AES or Chacha), or from a key exchange scheme such as Diffie-Hellman, and you rely on that. I left my phrasing as it was, because it is (I think) closer to the truth, and not dangerous. I do reckon the AEAD scheme I recommend only has a reduction proof, not a safety proof. I could be more precise, but I don't like clutter.
lisper 10y ago
SC4 is a PGP replacement designed to be simple and easy to use: https://github.com/Spark-Innovations/SC4 https://github.com/Spark-Innovations/SC4 Some (but not all) of the code has been audited. Based on DJB's TweetNaCl.
wfunction 10y ago
Uhm, you should roll your own crypto. Just make sure it's not the only layer of crypto -- i.e., that there's a standard layer too. Not going to rehash the arguments, but this was discussed recently: https://news.ycombinator.com/item?id=13199471 https://news.ycombinator.com/item?id=13199471
- Nomentatus 10y ago
  Yup. That's the answer - that's what XOR is for, so that you don't have to encrypt twice (or more) serially, but in parallel. Note too, that encrypting a pad can be done in advance, with spare cycles, at both ends, for many methods, then XOR with the pad extremely quickly to communicate. In the Vietnam war the North Vietnamese army used both a one-time pad layer and a book code layer. But without computers it was too cumbersome a method and most messages therefore went out in the clear.
  - wfunction 10y ago
    It's actually not what I was thinking of (I was in fact thinking of encrypting sequentially and hashing+XOR in parallel), but that's a good point, that might work too! I'd need to think about it a bit more to convince myself it doesn't have any significant downsides though; it's less obvious to me. For one thing, requiring an OTP approach seems to constrain the set of encryption schemes you can use. (e.g. imagine an algorithm that reverses all the bits of every block before doing anything else.) Furthermore, if your custom encryption scheme happens to leave any kind of "watermark" on the ciphertext that makes it obvious it wasn't something standard like AES, applying the standard layer last will prevent the attacker from realizing you have a custom layer at all, until the standard layer is broken. Whereas if you use the OTP+XOR scheme, it might become obvious something else is going on too. That said, this approach might actually be stronger than the layered approach, so it might actually be better. Need to think about it more :)
    - Nomentatus 10y ago
      I'll have to look back to see if I in fact have added to the fundamental meaning of what others have called "layering." The pad doesn't absolutely have to be an OTP, particularly when rolling your own layer, if the pad isn't the only key, and the key can change. Watermarks are extinguished when XORed with any genuinely decent (random-imitating, not frequency-bustable) encryption, true. And you're right, at least one layer should not have a watermark. Order doesn't matter though, there really isn't an order for the layers, they're merged with one another. A not-very-good roll-your-own encryption scheme, if it introduces a very large number of permutations (such as starting pad, key) can be very helpful as a layer even though it would be a huge fail as stand-alone encryption. Mechanisms to increase adjust the size (of all but one layer, anyway) are advisable.
nerdponx 10y ago
The idea that anyone would even want to roll their own crypto makes no sense to me. It would be like rolling your own linear algebra routines instead of using BLAS, or rolling your own numeric optimizer instead of using BFGS (or whatever other optimizer makes sense). Crypto is no less esoteric and no less mathematical.
gioscarab 10y ago
I Wrote my own crypto, here it is: https://github.com/gioblu/Cape https://github.com/gioblu/Cape can you brake it? I do think it is necessary to write your own crypto to understand what "crypto" is.
- loup-vaillant 10y ago
  I'd have one objection: the Readme doesn't say anything about the underlying primitives (are you using something standard or did you invent your own cipher?), and doesn't say anything about external validation (are you alone so far, or did your code get reviewed?). If it's a learning exercise, warning wannabe users would be nice. That said, your code looks too simple not to be easily breakable. I'm no expert, but that piqued my interest. I'll take a look. If I break it, I'll write an article about it.
  - loup-vaillant 10y ago
    Okay, I think I have cracked it. Haven't tested it, but a qualitative reasoning should be enough to get you going. I concentrated my efforts on your Cape::encrypt() method. Inlining Cape::crypt() and simplifying the resulting code gives something like this (it appears some computations cancelled each other): void Cape::encrypt(char *source, char *destination, uint8_t length) { uint8_t iv = this->generate_IV(); char real_key[255]; for (i = 0; i < _key_length; i++) real_key[i] = _key[(i ^ _reduced_key) % _key_length]; for (i = 0; i < length; i++) destination[i] = source[i] ^ iv ^ i ^ real_key[i]; destination[length] = iv ^ _reduced_key; }; Now the weaknesses of your algorithm are clear. First, I don't believe your IV is truly randomly generated. I don't know of a real CSPRNG on Arduino, so… probably not your fault. Second, your IV is only one byte. You cannot securely send more than 256 messages with the same key. Third, the attacker can get rid of the IV anyway: since it is given at the end of the ciphertext, I can just XOR it with the rest of the ciphertext before trying to crack it. This makes your IV irrelevant. You are now limited to one message per key. In practical terms, you don't do better than a one time pad. Fourth, the way you access the key in a weird order doesn't matter. It's the same as using a different key. Assuming your key is properly unpredictable in the first place, there is no need to access it in a weird order. I made this clear by reordering the key in a real_key. Fifth, XORing the message with the index doesn't buy you anything: the attacker will just reverse it before trying to crack the rest. Sixth, if your key is shorter than the message, you have a repeating key. Repeating keys are shown to be broken since… a long time. The reason why is because XORing parts of the ciphertext encoded with the same key reveals the XOR of parts of the plaintext, and that is easily broken in practice. So you're forced to use keys as long as the message. Conclusion: you have a fancy one time pad. It is secure if your key is as long as the message, and you use it only once. I have to say your home-made crypto is not completely broken. But… a simple one time pad would have achieved the same results more simply. A word of (dangerous) advice: you won't accomplish much with XOR alone. Modern ciphers tend to also to rearrange the order of the bits as well. If I may, we have a real cryptographer here, and he has written a course: https://www.crypto101.io/ https://www.crypto101.io/
mlosapio 10y ago
Rolling your own crypto - how about "don't"

192 points by svenfaw 10y ago

> Erratum: Apparently, I don't know what I'm doing either. I committed a number of approximations, misdirections, and even errors in there. Some of them where pointed out by /u/sacundim on Reddit. If you thought you could trust me, read that comment. It's sobering. LOL brilliant. That was fast.

neaden 10y ago
I'm glad he put that up at the top instead of trying to hide it. The moral of the story appears to be indeed: Don't roll your own crypto.
- inetsee 10y ago
  I think rolling your own crypto, as a learning exercise, can be very helpful when choosing which trusted crypto solutions to use for a particular use case.
  - lvh 10y ago
    Breaking crypto is certainly an excellent way to learn many kinds of crypto. The problem is that writing your own typically isn't enough by itself -- a lot of horribly broken crypto looks fine after casual inspection. You probably won't find bugs like CBC padding oracles or Bleichenbacher attacks all by yourself. You can, however, absolutely _implement_ them yourself. I recommend cryptopals, together with my own free book, Crypto 101 if you prefer a little prose versus trying to figure it out yourself.
- loup-vaillant 10y ago
  I draw a different conclusion: don't roll crypto on your own. https://www.reddit.com/r/programming/comments/5iv1ti/rolling_your_own_crypto/dbg58iy/ https://www.reddit.com/r/programming/comments/5iv1ti/rolling... I have updated my article since, and also made clear you can still screw up even if you had help.
oconnore 10y ago
A lot of those criticisms are sort of unfair. For example, pointing out length extension attacks in a fabricated MAC construction not found in the article, especially when the OP explicitly recommends blake2b, which is not vulnerable to length extension (and also disclaims "How that magical MAC is built is beyond the scope of this paragraph."). Another example: criticizing that he mentions appending a MAC (you should choose an AEAD!)... when later he recommends using an AEAD.
- dom0 10y ago
  AEAD normally only means actual AEAD constructions (think AES-GCM/OCB, Chacha20-Poly1305 - not combinations of primitives providing authentication (=EtM, eg. first encrypt with AES-CTR then MAC with some HMAC). But yeah, some of the criticism reads kinda nit-picky / "I'm going to find false things in here". But on the other hand he provides good hints how the article could be improved.
  - oconnore 10y ago
    The article recommends Chacha20 Poly1305, not some combination of primitives.
    - dom0 10y ago
      What do you mean by this then? > Another example: criticizing that he mentions appending a MAC (you should choose an AEAD!)... when later he recommends using an AEAD.
      - oconnore 10y ago
        There is a section called "integrity: not optional", where he uses MACs (which I would consider to be conceptually simpler to understand than AEADs) to explain how integrity can be implemented. Later, in a section called "Level 2: choosing crypto", he recommends using ChaCha20 Poly1305. Anyway, this isn't really my crusade, I just think that if you read the informal blog post as an informal blog post, it's fine.
Bartweiss 10y ago
If nothing else, this was a hell of a lesson in just how hard rolling your own crypto really is.
grkvlt 10y ago
> Erratum: [...] Some of them where pointed out [...] And Muphry's Law[0] strikes again! [0] https://en.wikipedia.org/wiki/Muphry's_law https://en.wikipedia.org/wiki/Muphry's_law

ohyoutravel 10y ago

I am interested enough where I would read a long form article on this kind of stuff, and found this fascinating. But the end kind of left me feeling dejected, as the article I guess is just a parody. Which is fine, but I probably wouldn't have spent as much time trying to take in what I presumed the article was trying to teach. I would really enjoy an article like this that actually goes step-by-step so I could learn something like this start to finish (though I would never implement). From the article, "Last Step in Crypto:" Get a PhD in cryptography. You need to be an expert yourself if you ever hope to invent a primitive that works. Publish your shiny new primitive. It may withstand the merciless assaults of cryptanalists, and be vetted by the community. Wait. Your primitive needs to stand the test of time.

Kurtz79 10y ago
If you are interested in more than an article, this course is very good. https://www.coursera.org/learn/crypto https://www.coursera.org/learn/crypto Sorry to spoil it, but the conclusion will basically be the same as the article, as in "just don't".
- StavrosK 10y ago
  Yes, but arriving at the conclusion is very fun. Although, don't hold your breath for Cryptography II. I've been waiting for it for years.
j2kun 10y ago
This may only be sort of relevant, but I write a math/programming blog on which I implemented a (very inefficient) elliptic curve crypto library from first principles. This includes all the math background, and implementations of a few working protocols. It might be a good starting point if you're looking for resources like this (exposition-wise, not in terms of a practical crypto solution). https://jeremykun.com/2014/02/08/introducing-elliptic-curves/ https://jeremykun.com/2014/02/08/introducing-elliptic-curves...
- tptacek 10y ago

Exuma 10y ago

https://www.reddit.com/r/programming/comments/5iv1ti/rolling_your_own_crypto/dbbmwwo/ https://www.reddit.com/r/programming/comments/5iv1ti/rolling...

shujito 10y ago
yes, it says it in the very article
- loup-vaillant 10y ago
  I had to admit I screwed up a little… I didn't expect merely writing about crypto would be so hard. Good thing I put this up for peer review. I have since revised my article. It should be less dangerous now.

edblarney 10y ago

The four most dangerous words in CS. But it's good that people look into it -> for fun and learning. Just not for realsies.

bkanber 10y ago
There's serious pushback against rolling your own crypto. Your attitude towards it is good, but the hivemind-y attitude is to never do it under any circumstances. I'm sure that attitude prevents many people from experimenting and becoming crypo experts in their own right. Roll your own crypto, but be aware of the stakes, and be prepared to drop it the second that the stakes become real.
- pflats 10y ago
  >There's serious pushback against rolling your own crypto. Your attitude towards it is good, but the hivemind-y attitude is to never do it under any circumstances. I'm sure that attitude prevents many people from experimenting and becoming crypo experts in their own right. It's not a hivemind-y attitude, it's Software Engineering. If you cannot show that a bridge will support a given weight, you do not build the bridge. Period. You learn that in your first Engineering course, freshman year. It's part of the responsibility of being an Engineer. You can experiment with new bridge designs all you want, but until you can prove mathematically that the bridge is sound, you do not build the bridge. If you really want to roll your own bridge over the ditch in the your backyard, go for it, but think three times before inviting your friend to drive their truck over it, especially if you're not already an expert on bridges. Cryptography is not a small or simple subject. If you are not an expert in the field, it is incredibly unlikely that you will not know enough to critique your own work. Instead of starting out by rolling your own, start by studying.
  - bkanber 10y ago
    That's a reductionist ("Period.") and a strawman position. It's unfortunate that you invoked the responsibility of the engineer in your argument: I went to engineering school. I have two degrees in mechanical engineering from a top school, I've published peer reviewed papers, and have helped design hybrid cars for companies you'd recognize. I qualify as an engineer under every definition you can think of, understand firsthand the responsibilities of an engineer, have gone through rigorous safety training in order to test drive the cars that I've personally designed and built. And yet I still say that nothing you said here is at odds with my statement: know the stakes. But play anyway. You think engineers don't build wacky and dangerous shit for their friends to play with? Prototypes that stretch the limits of design and would be irresponsible to mass produce as-is? We do it all the time. It's part of the learning process. I've been physically hurt, shocked, burnt both by heat and chemicals, while working with other engineers' experiments. AND IT'S OK! It's ok to try your own solutions on your friends and peers while the stakes are low. Studying is only one half of learning. The other half is bringing textbook knowledge to the real world by building and prototyping and getting your peer group to throw hammers at it and see what happens. Engineering is not theory. Engineering is theory applied to the real world. You can't be an engineer if you don't have one foot on each side. Study on one side, build and play on the other. It definitely is a hivemind attitude, because the literal hundreds of other engineers that I know -- people who design actual bridges, and buildings, and medical devices, and cars, and oil rigs, and theater sets, things that hundreds of millions of people rely on to be safe -- don't share this attitude. Their attitude is: know the stakes. Most software projects go nowhere and are usually only used by a handful of people close to the developer. That's the definition of low stakes. And it's ok to experiment, and to roll your own crypto, when the stakes are low. I think it's damaging, and probably hubristic, to think that the stakes are high all the time. They're not.

niftich 10y ago

Quoting an older post of mine: The 'don't roll your own crypto' argument is mostly just shorthand to 'defer to the opinion of experts, use ready-made constructs when possible, and if not, then exercise caution when hooking crypto primitives together in unproven ways'. [1] Crypto code, like other library code, is question of trust. Do I trust Daniel Bernstein? Do I trust Joan Daemen, who is half of the AES team and a quarter of the Keccak team [2]? In practical matters, do I trust tptacek [3]? Yes, I trust them, until people more educated in cryptography than me cast enough doubt or prove it otherwise -- but you might have a different model of who you trust. But ultimately, you're the one who answers to your systems. It's also a bit like science where we come up with a hypothesis (this seems to work...) and then try very hard disprove it, so our knowledge evolves as we go. It's important to understand that there was a time when it was best practice to use certain primitives that are now considered broken, and this is okay -- assuming we all upgraded our systems since. For cases then that assumption can't hold true, we need to account for that risk. This is also why it's wise to go with cryptosystems that receive a good amount of peer scrutiny. Your homegrown secret sauce might indeed be super secure, but few will publish papers on how they can obtain collisions on a round-reduced version. It's network effects, it's 'given enough eyes' all over again. [1] https://news.ycombinator.com/item?id=12400040 https://news.ycombinator.com/item?id=12400040 [2] https://news.ycombinator.com/item?id=12766941 https://news.ycombinator.com/item?id=12766941 [3] https://gist.github.com/tqbf/be58d2d39690c3b366ad https://gist.github.com/tqbf/be58d2d39690c3b366ad

mfukar 10y ago
Would any body else put the same trust on whatever authority the external auditors or pentesters display?
- lvh 10y ago
  Most cryptanalysis is public; there's little reason for public researchers to withhold cryptographic breaks. The only reason that'd happen is if you're an intelligence agency and you'd like to keep a vuln to yourself (those are less likely in crypto than in software in general). In that light, it'd be perhaps better to phrase trusting AES as "not distrusting the union of everyone who's tried to cryptanalyse AES", rather than trusting the Rijndael team specifically. In many cases, especially djb-brand crypto, there's even less need for trust: the way you derive the X25519 curve is extremely well defined (or, as djb puts it, "rigid").

slindsey 10y ago

Forget about creating your own crypto. I have had endless troubles just using crypto. Obscure options that aren't explained. Examples that don't actually work. C# had an option that I finally got to work but the documentation specifically said not to use it; but the other choices didn't work. I had to encrypt something in PHP then decrypt it in ColdFusion. Despite being the same algorithm, key, etc, it didn't work. Strangely, the exact same data in the other direction (CF->PHP) worked just fine. I'd just like the libraries I use to have a simple choice: Encrypt(data,key) and Decrypt(data,key) and be done.

ori_b 10y ago
Have you looked at Libnacl or Libsodium?
dom0 10y ago
If it's something sparkling new you probably want to use NaCl or for more involved use cases a Noise implementation.
- tptacek 10y ago
  It's the opposite, I think: Noise is good for less-involved cases, the kinds of problems you would ordinarily solve by just using TLS. Nacl is better for when you need something customized to an application. Incidentally: Noise is great, but it's also fine to use TLS, if you're just a little careful. See also: https://gist.github.com/tqbf/be58d2d39690c3b366ad https://gist.github.com/tqbf/be58d2d39690c3b366ad
  - jjnoakes 10y ago
    The "Avoid /dev/random" recommendation, can you elaborate on that?
    - tptacek 10y ago
      https://sockpuppet.org/blog/2014/02/25/safely-generate-random-numbers/ https://sockpuppet.org/blog/2014/02/25/safely-generate-rando...

pselbert 10y ago

You have to applaud him for trying. Building anything as a learning exercise is an investment in yourself, and I'm encouraged when others are ambitious about it. We've all had "don't roll your own crypto" pounded into us, but nothing teaches a lesson as soundly as trying something yourself. I've tried, failed and learned a lesson from far less ambitions endeavors.

bisby 10y ago
a great rule would be "try to roll your own crypto, but only as a learning tool, don't try to use it"
- stouset 10y ago
  Roll your own crypto, but for the love of god don't release it.

ivanhoe 10y ago

The only reason I'm aware of for rolling a custom crypto would be fighting against automated attacks. Even ridiculously simple protection can prevent a script from automatically collecting the data. If there's a lot of data streams and each requires human interaction to decrypt it, it can significantly raise the cost of breaking such data. In that case it's probably best to make it just an additional layer over some well-tested method.

lvh 10y ago
I'm not sure I understand the rationale. Automated attacks can't beat real crypto either -- so why even bother rolling your own? (And automated attacks can certainly see non-cryptographic encodings like Base64, Base32, ROT13...)

jwtadvice 10y ago

I like how the author organizes 'levels' of rolling cryptography, and generally agree with his assessment of how those strata should be organized.

lvh 10y ago

As one might expect[1], I have some thoughts. There are a few "layers" of rolling your own crypto. Often, people think they're not rolling their own crypto because they're using AES instead of some hand-rolled bizarro cipher. This is not that: the primitives the post references are solid (e.g. BLAKE2b, ChaCha20, X25519). There are definitely challenges with explaining crypto to programmers in a way that is non-scary and at the same time factually accurate; I walk that tightrope constantly with Crypto 101. There are significant factual inaccuracies in this post that matter and aren't just a temporary educational tool so you can get a concept across quickly. Someone on Reddit[2] already pointed out most of the ones that were glaring at me after a cursory review. To the author's credit, he's put that link up by the start of the post. There are at least two kinds of crypto education for programmers. One is practical advice or libs to help people build better applications with crypto. This means building better tools. Don't "pick Ed25519" -- pick a library like libsodium that did signatures right for you. Saying "compose ChaCha20 and append a MAC" isn't the best level of advice we can strive for. Instead, "use Fernet" and possibly "use libsodium's secretbox" is (although I think having to specify a nonce may not be the greatest default -- I'm working on fixing that). The other kind of crypto education for programmers is to help people break things. That can help you become an expert, but I don't think we should expect any meaningful percentage of programmers will spend a bunch of time finishing all 8 sets of Cryptopals. [3] It is, however, a reasonable assumption that a (super)majority of programmers will at some point touch some crypto: the question is what they find when they do that. Will they have secure password storage and not even notice because that was the right default in the software they were using, or will they have to cobble together their own? I for one am hoping it's the former. Fundamentally, there are two reasons why you don't want to roll your own crypto. One is because there's a good chance you'll mess it up. The other is because that's effort, and you are not going to do better than the programmers that already have done the hard work for you. This is sometimes true for primitives (don't generate your own FFDH prime/implement your own FFDH -- you will probably have small subgroups!), but especially true for high-level recipes at varying levels of complexity from authenticated encryption (you won't be better than OCB or secretbox) all the way to cryptographic ratchets. Those reasons put together form my main criticism for this article. Sure, it has fundamental inaccuracies, they're not trivial, and that's not OK. But even if it was technically perfect, it doesn't help the next piece of software be safer. [1]: I am a cryptographer. I did https://www.crypto101.io https://www.crypto101.io. I am Latacora's crypto nerd. [2]: https://www.reddit.com/r/programming/comments/5iv1ti/rolling_your_own_crypto/dbbmwwo/ https://www.reddit.com/r/programming/comments/5iv1ti/rolling... [3]: http://cryptopals.com http://cryptopals.com

deleted 10y ago

[deleted]

kjs3 10y ago

The problem with "roll your own crypto" isn't the rolling, per se, it's that there's about 5 orders of magnitude more people who are sure that they can do it right than people who actually can do it right.

jackgavigan 10y ago

I feel like no discussion of rolling one's own crypto is complete without a link to Renaissance artist Pieter Bruegel the Elder's painting of the subject (c. 1562): http://classicprogrammerpaintings.com/post/148027314949/we-rolled-our-own-crypto-pieter-bruegel-the http://classicprogrammerpaintings.com/post/148027314949/we-r...

danbruc 10y ago

Poly1305 is one of the few primitives out there that are provably secure. That doesn't sound right. I guess this should read something like »is provably as hard to break as the underlying block cipher« in the same way that that a hash function built using the Merkle-Damgård construction is provably as hard to break as the compression function used. The article easily gives the impression that Poly1305 is provably secure in the same way as one-time pad.

loup-vaillant 10y ago
I have thought about it and re-read the original paper. I stand by my claim. The truth of the matter is, poly1305, while provably secure, is almost as impractical as a one time pad: it relies on a shared random authentication key. Where does that key is supposed to come from? In practice, you'd derive it from a session key (and so rely on a symmetric cipher such as AES or Chacha), or from a key exchange scheme such as Diffie-Hellman, and you rely on that. I left my phrasing as it was, because it is (I think) closer to the truth, and not dangerous. I do reckon the AEAD scheme I recommend only has a reduction proof, not a safety proof. I could be more precise, but I don't like clutter.

lisper 10y ago

SC4 is a PGP replacement designed to be simple and easy to use: https://github.com/Spark-Innovations/SC4 https://github.com/Spark-Innovations/SC4 Some (but not all) of the code has been audited. Based on DJB's TweetNaCl.

wfunction 10y ago

Uhm, you should roll your own crypto. Just make sure it's not the only layer of crypto -- i.e., that there's a standard layer too. Not going to rehash the arguments, but this was discussed recently: https://news.ycombinator.com/item?id=13199471 https://news.ycombinator.com/item?id=13199471

Nomentatus 10y ago
Yup. That's the answer - that's what XOR is for, so that you don't have to encrypt twice (or more) serially, but in parallel. Note too, that encrypting a pad can be done in advance, with spare cycles, at both ends, for many methods, then XOR with the pad extremely quickly to communicate. In the Vietnam war the North Vietnamese army used both a one-time pad layer and a book code layer. But without computers it was too cumbersome a method and most messages therefore went out in the clear.
- wfunction 10y ago
  It's actually not what I was thinking of (I was in fact thinking of encrypting sequentially and hashing+XOR in parallel), but that's a good point, that might work too! I'd need to think about it a bit more to convince myself it doesn't have any significant downsides though; it's less obvious to me. For one thing, requiring an OTP approach seems to constrain the set of encryption schemes you can use. (e.g. imagine an algorithm that reverses all the bits of every block before doing anything else.) Furthermore, if your custom encryption scheme happens to leave any kind of "watermark" on the ciphertext that makes it obvious it wasn't something standard like AES, applying the standard layer last will prevent the attacker from realizing you have a custom layer at all, until the standard layer is broken. Whereas if you use the OTP+XOR scheme, it might become obvious something else is going on too. That said, this approach might actually be stronger than the layered approach, so it might actually be better. Need to think about it more :)
  - Nomentatus 10y ago
    I'll have to look back to see if I in fact have added to the fundamental meaning of what others have called "layering." The pad doesn't absolutely have to be an OTP, particularly when rolling your own layer, if the pad isn't the only key, and the key can change. Watermarks are extinguished when XORed with any genuinely decent (random-imitating, not frequency-bustable) encryption, true. And you're right, at least one layer should not have a watermark. Order doesn't matter though, there really isn't an order for the layers, they're merged with one another. A not-very-good roll-your-own encryption scheme, if it introduces a very large number of permutations (such as starting pad, key) can be very helpful as a layer even though it would be a huge fail as stand-alone encryption. Mechanisms to increase adjust the size (of all but one layer, anyway) are advisable.

nerdponx 10y ago

The idea that anyone would even want to roll their own crypto makes no sense to me. It would be like rolling your own linear algebra routines instead of using BLAS, or rolling your own numeric optimizer instead of using BFGS (or whatever other optimizer makes sense). Crypto is no less esoteric and no less mathematical.

gioscarab 10y ago

I Wrote my own crypto, here it is: https://github.com/gioblu/Cape https://github.com/gioblu/Cape can you brake it? I do think it is necessary to write your own crypto to understand what "crypto" is.

loup-vaillant 10y ago
I'd have one objection: the Readme doesn't say anything about the underlying primitives (are you using something standard or did you invent your own cipher?), and doesn't say anything about external validation (are you alone so far, or did your code get reviewed?). If it's a learning exercise, warning wannabe users would be nice. That said, your code looks too simple not to be easily breakable. I'm no expert, but that piqued my interest. I'll take a look. If I break it, I'll write an article about it.
- loup-vaillant 10y ago
  Okay, I think I have cracked it. Haven't tested it, but a qualitative reasoning should be enough to get you going. I concentrated my efforts on your Cape::encrypt() method. Inlining Cape::crypt() and simplifying the resulting code gives something like this (it appears some computations cancelled each other): void Cape::encrypt(char *source, char *destination, uint8_t length) { uint8_t iv = this->generate_IV(); char real_key[255]; for (i = 0; i < _key_length; i++) real_key[i] = _key[(i ^ _reduced_key) % _key_length]; for (i = 0; i < length; i++) destination[i] = source[i] ^ iv ^ i ^ real_key[i]; destination[length] = iv ^ _reduced_key; }; Now the weaknesses of your algorithm are clear. First, I don't believe your IV is truly randomly generated. I don't know of a real CSPRNG on Arduino, so… probably not your fault. Second, your IV is only one byte. You cannot securely send more than 256 messages with the same key. Third, the attacker can get rid of the IV anyway: since it is given at the end of the ciphertext, I can just XOR it with the rest of the ciphertext before trying to crack it. This makes your IV irrelevant. You are now limited to one message per key. In practical terms, you don't do better than a one time pad. Fourth, the way you access the key in a weird order doesn't matter. It's the same as using a different key. Assuming your key is properly unpredictable in the first place, there is no need to access it in a weird order. I made this clear by reordering the key in a real_key. Fifth, XORing the message with the index doesn't buy you anything: the attacker will just reverse it before trying to crack the rest. Sixth, if your key is shorter than the message, you have a repeating key. Repeating keys are shown to be broken since… a long time. The reason why is because XORing parts of the ciphertext encoded with the same key reveals the XOR of parts of the plaintext, and that is easily broken in practice. So you're forced to use keys as long as the message. Conclusion: you have a fancy one time pad. It is secure if your key is as long as the message, and you use it only once. I have to say your home-made crypto is not completely broken. But… a simple one time pad would have achieved the same results more simply. A word of (dangerous) advice: you won't accomplish much with XOR alone. Modern ciphers tend to also to rearrange the order of the bits as well. If I may, we have a real cryptographer here, and he has written a course: https://www.crypto101.io/ https://www.crypto101.io/

mlosapio 10y ago

Rolling your own crypto - how about "don't"