10 ms·

I am terrified that I do not consider myself competent enough to write a crypto library, and yet there isn't a single mention - in this article, nor at the time

by ZoFreX 9y ago

I am terrified that I do not consider myself competent enough to write a crypto library, and yet there isn't a single mention - in this article, nor at the time of writing the comments here on Hacker News - of many of the pitfalls I know to avoid when undertaking such an endeavour. There is even a list of "you have to do A, B, C, and that's about it" that is missing some major - well known, even! - items. I know "don't roll your own crypto" comes across as dismissive or patronising. I'm not a huge fan of the phrasing. But as a first-order approximation, it is correct. If you want to roll your own crypto, that needs to be your _thing_. It's very unlikely you're going to be a fantastic full-stack web developer _and_ be able to do that. If it's really what you want to do then awesome! Go study it, learn it, practice it. But if you're doing it as a side-hobby, never put it into production.

loup-vaillant 9y ago
Could you list the main pitfalls you are thinking about? Also, I may have mentioned some of them in this earlier article: http://loup-vaillant.fr/articles/rolling-your-own-crypto http://loup-vaillant.fr/articles/rolling-your-own-crypto
- msl09 9y ago
  Hey, like you I decided to dive into cryptography coming from a different background. Although this quote is not directly related to your problem it can be safely applied to it: "Almost certainly you will get the urge to invent new cryptographic algorithms, and will believe that they are unbreakable. Don't resist the urge; this is one of the fun parts. But resist the belief; almost certainly your creations will be breakable, and almost certainly no one will spend the time breaking them for you. You can break them yourself as you get better."[1] The problem that Schneier is referring to is that doing a careful analysis (that is required if you want to use your crypto in production) is tedious, time consuming, and requires expertise in the area to know most blank spots of the algorithms (heck a single one is hard enough). That's why he recommends you to be have enough experience breaking many algorithms before you make any serious claim about your the safety of your crypto. So all in all it's great that you got the interest in the area, there is a lot of work needed in OSS. But be very careful about claiming that your lib is ready for production. What you got is a bunch of volunteers to glance at your code. Few cryptographers (if any) will seriously try to break it. [1] https://www.schneier.com/crypto-gram/archives/1999/1015.html#SoYouWanttobeaCryptographer https://www.schneier.com/crypto-gram/archives/1999/1015.html...
  - Piskvorrr 9y ago
    Note that the author is not inventing crypto algorithms, rather implementing them (although the part about XChaCha20 being a mix of ChaCha20 and XSalsa20 is IMHO dancing on the line). Still a risky business, and tricky to get right, but several orders of magnitude safer than "hey, what if we just XORed everything with a random number? Unbreakable, eh?"
    - pkolaczk 9y ago
      What's wrong with xoring with a stream of random numbers? Isn't it how stream ciphers work? Get a good cryptographic RNG, initialize it properly with a long enough key and you should be fine.
      - baby 9y ago
        There's nothing wrong with that. You are right, that's how stream ciphers work.
      - Piskvorrr 9y ago
        Thank you, that's my point exactly. Nothing wrong with what you wrote - but note the subtle difference between "XORing a stream of random numbers" (potentially viable, even unbreakable when using a OTP) and "XORing a random number" (essentially a Caesar cipher, kid-sister encryption).
        thaumasiotes 9y ago
        > note the subtle difference between "XORing a stream of random numbers" (potentially viable, even unbreakable when using a OTP) and "XORing a random number" (essentially a Caesar cipher, kid-sister encryption) No, that's not a difference. A "stream of numbers" is just one very large number. What you're worrying about is how large the numbers are, not whether there's one or more than one.
        dfox 9y ago
- erikbye 9y ago
  And instead of saying nothing those who harp on "C's pitfalls" could politely enumerate those relevant to the issue at hand, that way we actually have something discrete to discuss.
Sylos 9y ago
I generally also think that it's a bad idea to do, but I feel like it's a discussion that needs to be had with governments around the world wanting to backdoor everything. It means that normal citizens won't be completely helpless against tyrannic governments and it helps to educate that writing crypto from scratch is most definitely something that criminals can do. The government making it illegal or backdooring it will only help against lowest-effort criminals, not against organized terrorist groups. Sure, those probably won't write unbreakable encryption, but it'll be enough to bypass a government that's expecting everything to be in plain text.
- bpicolo 9y ago
  > but I feel like it's a discussion that needs to be had with governments around the world wanting to backdoor everything. On the contrary, this is -exactly- why you don't want to roll your own crypto. Because they can and will break it. Good crypto takes serious thought and effort from people who put a lot of thought and effort into it.
JohnStrange 9y ago
I apologize if that sounds dismissive (it's not intended to) but your post is not more than the usual vague FUD. Take a look at so-called 'professional' cryptographers and their libraries, and you'll find that practically all of their code had severe bugs. Professionals make errors like everyone else, 'professional' just means you're being paid for it. You will have a hard time finding a professional and widely used crypto library that wasn't essentially compromised in one of its functions at one time or another. Not only that, professionals from the closed-source department have a long history of coming up with compromised or bogus, sometimes even ridiculous cryptographic algorithms and implementations. Two typical examples: CMEA cell phone encryption, and the Crypto AG backdoor debacle. Surely people who invent and implement cryptographic algorithms for a living can be expected to be better than your run-off-the-mill programmer, but as I've said elsewhere implementing crypto is also not a black art. It requires about the same level of skill as e.g. implementing a production-ready low-level network protocol or a file system. That rules out many programmers but not all. That concerns implementations. As for algorithms, their development requires extensive experience in cryptanalysis (not just applied cryptography!). Some pros have that, others don't. Last but not least, many popular and widely used crypto libraries started as a side hobby. For example, libtomcrypt started that way. In fact, many widely respected cryptographers started as hobbyists. For example, Bruce Schneier. Of course, the ones who are widely accepted as peers are also those who are in the 'hire me as a consultant, not the other guy' business, and they will naturally advise everyone to not roll their own crypto but to rely on their expertise...
- dsacco 9y ago
  So I have two points for you: > Take a look at so-called 'professional' cryptographers and their libraries, and you'll find that practically all of their code had severe bugs. Professionals make errors like everyone else, 'professional' just means you're being paid for it. You will have a hard time finding a professional and widely used crypto library that wasn't essentially compromised in one of its functions at one time or another. Not only that, professionals from the closed-source department have a long history of coming up with compromised or bogus, sometimes even ridiculous cryptographic algorithms and implementations. Two typical examples: CMEA cell phone encryption, and the Crypto AG backdoor debacle. This doesn't diminish the claim that the modal individual should not attempt to "roll their own crypto." You're pointing out that professionals make mistakes, but of course they do - that indicates nothing about the propensity for amateurs to make mistakes. > It requires about the same level of skill as e.g. implementing a production-ready low-level network protocol or a file system. I have never implemented a low level network protocol or a file system (although at least the former sounds like a fun project) - so I cannot claim you're wrong here. However, I will point out that a network protocol is not designed to be error-proof in an intentionally antagonistic environment, which is a difficulty unique to cryptographic software. There are incentives involved in breaking crypto code that do not present themselves in other areas of software development, even if the programming difficulty itself is not entirely dissimilar. > Last but not least, many popular and widely used crypto libraries started as a side hobby. For example, libtomcrypt started that way. In fact, many widely respected cryptographers started as hobbyists. For example, Bruce Schneier. Of course, the ones who are widely accepted as peers are also those who are in the 'hire me as a consultant, not the other guy' business, and they will naturally advise everyone to not roll their own crypto but to rely on their expertise... They can start as side hobbies, but they are not going to remain that way for any meaningful amount of time if they are to be widely deployed and safe. You can only choose two out of those three. More importantly, there is no cabal of cryptographers spreading FUD to line their own pockets with consulting fees. The cryptography consulting industry is very small compared to the actual security consulting industry. In the application security consulting industry I do believe there are firms that try to secure business this way, but that industry is much larger (and has firms which try to misrepresent cryptographic competency). I have had significant interactions with engineers at Riscure and NCC Crypto, which is a sizable portion of all the "real" cryptanalytic consulting work that occurs (at least in the United States) - they never struck me as being the sort to suggest what you're saying. That leaves Rambus/CR and a select few other firms doing real crypto consulting, which leads me to believe the industry is, on the whole, very legitimate.
madez 9y ago
You can do crypto as a side-hobby and safely put it into production. Of course, you can also do it wrong, but it is with little work possible to do it correctly. Normally I wouldn't speak against what you said because the critique is only very minor, but I heard this too often. Your warning is too strong. People should try their own crypto and with little care it is not unsafer. Maybe giving a list of do's for that would be a good start.
- marcoperaza 9y ago
  Sorry, but this is not good advice. Widely-used crypto implementations have had the benefit of years of analysis by dozens of high-expertise stakeholders who have a lot to lose should the crypto fail. Even that isn't always enough to catch all weaknesses and vulnerabilities. This cowboy-programming attitude being extended to security is no small part of why we are so vulnerable as a society to attacks on our computer systems that can compromise our core infrastructure[1], our secrets[2], and our economic security[3]. [1] Think weeks-long country-wide power outages [2] Think juicy blackmail material on anyone with their hands on levers of power. Look at the damage that North Korea's attacks did to Sony Pictures, for example. [3] Think small-scale attacks on services that even a single major company, or many small companies, rely on.
  - madez 9y ago
    You are wrongly assuming I say to throw away the work of others. I never said so.
- deleted 9y ago
  [deleted]
- dsacco 9y ago
  > You can do crypto as a side-hobby and safely put it into production. No, you really can't, unless you stretch the definition of "side-hobby" to the point of breaking at the seams. I'd be shocked if a single person who professionally works in security or cryptography is going to agree with you in this thread. You need far more than a "little care" to assure that a new cryptographic library is safe. It's an endeavor suited to a collaborative academic or corporate environment, with robust human capital and strong oversight. I'm willing to accept that someone can safely implement a primitive or a construction in a new library with significant time, effort and feedback from others. But that level of effort doesn't really qualify as a side-hobby, and until the implementation was formally checked by professional cryptographers I'd consider it suspect. This is to say nothing of designing novel primitives or constructions, which I would consider as far removed from a "side-hobby" as a local 5K is to the Olympics. However, I absolutely think people should attempt to implement their own cryptography if they're curious about it and want to learn. Just don't use it in production and assume it's unsafe. People who work in the industry don't make these claims because we think it's fun, we do it because we've all seen the consequences.
beckler 9y ago
I've found some helpful guidelines, but I know it is by no means an exhaustive list. https://cryptocoding.net/index.php/Coding_rules https://cryptocoding.net/index.php/Coding_rules
logfromblammo 9y ago
Anyone who can write code can write a crypto library. The trick is to have a second person helping you, who has all your mathematics notes, a copy of your source code, physical access to your hardware, and the ability to run your program inside a debugger. You don't even get to know whether they put cream in their coffee. If you can keep a secret from them, you're doing okay. So then you rent a smarter person to do the same thing to make sure all the tricks and shortcuts have been attempted. If someone manages to break it in the wild after that, you're still doing about as well as some of the highest-paid professionals in the industry. Defense is difficult.
stcredzero 9y ago
There is even a list of "you have to do A, B, C, and that's about it" that is missing some major - well known, even! - items. Our "field" of programming is so weak in its preservation of institutional knowledge, that we have to occasionally fight the recurrence of bad ideas about very basic things, like using string filtering to avoid SQL injection. Yes, this happens, even in "hip" new parts of the field, like the communities around newer languages. (I have a specific example in mind, but I don't want to deal with the resulting war.)
catoc 9y ago
Definition question: would you say that AgileBits (1password) are "rolling their own crypto" or 'merely' implementing proven third-party crypto?
- ZoFreX 9y ago
  I would have to read their source code to answer that question (I know almost nothing about 1Password). Some password managers just call a PGP executable. Others are assembling crypto primitives into larger pieces and making choices like "let's use CBC mode" themselves.

by ZoFreX 9y ago

loup-vaillant 9y ago

Could you list the main pitfalls you are thinking about? Also, I may have mentioned some of them in this earlier article: http://loup-vaillant.fr/articles/rolling-your-own-crypto http://loup-vaillant.fr/articles/rolling-your-own-crypto

msl09 9y ago
Hey, like you I decided to dive into cryptography coming from a different background. Although this quote is not directly related to your problem it can be safely applied to it: "Almost certainly you will get the urge to invent new cryptographic algorithms, and will believe that they are unbreakable. Don't resist the urge; this is one of the fun parts. But resist the belief; almost certainly your creations will be breakable, and almost certainly no one will spend the time breaking them for you. You can break them yourself as you get better."[1] The problem that Schneier is referring to is that doing a careful analysis (that is required if you want to use your crypto in production) is tedious, time consuming, and requires expertise in the area to know most blank spots of the algorithms (heck a single one is hard enough). That's why he recommends you to be have enough experience breaking many algorithms before you make any serious claim about your the safety of your crypto. So all in all it's great that you got the interest in the area, there is a lot of work needed in OSS. But be very careful about claiming that your lib is ready for production. What you got is a bunch of volunteers to glance at your code. Few cryptographers (if any) will seriously try to break it. [1] https://www.schneier.com/crypto-gram/archives/1999/1015.html#SoYouWanttobeaCryptographer https://www.schneier.com/crypto-gram/archives/1999/1015.html...
- Piskvorrr 9y ago
  Note that the author is not inventing crypto algorithms, rather implementing them (although the part about XChaCha20 being a mix of ChaCha20 and XSalsa20 is IMHO dancing on the line). Still a risky business, and tricky to get right, but several orders of magnitude safer than "hey, what if we just XORed everything with a random number? Unbreakable, eh?"
  - pkolaczk 9y ago
    What's wrong with xoring with a stream of random numbers? Isn't it how stream ciphers work? Get a good cryptographic RNG, initialize it properly with a long enough key and you should be fine.
    - baby 9y ago
      There's nothing wrong with that. You are right, that's how stream ciphers work.
    - Piskvorrr 9y ago
      Thank you, that's my point exactly. Nothing wrong with what you wrote - but note the subtle difference between "XORing a stream of random numbers" (potentially viable, even unbreakable when using a OTP) and "XORing a random number" (essentially a Caesar cipher, kid-sister encryption).
      - thaumasiotes 9y ago
        > note the subtle difference between "XORing a stream of random numbers" (potentially viable, even unbreakable when using a OTP) and "XORing a random number" (essentially a Caesar cipher, kid-sister encryption) No, that's not a difference. A "stream of numbers" is just one very large number. What you're worrying about is how large the numbers are, not whether there's one or more than one.
        dfox 9y ago
erikbye 9y ago
And instead of saying nothing those who harp on "C's pitfalls" could politely enumerate those relevant to the issue at hand, that way we actually have something discrete to discuss.

Sylos 9y ago

I generally also think that it's a bad idea to do, but I feel like it's a discussion that needs to be had with governments around the world wanting to backdoor everything. It means that normal citizens won't be completely helpless against tyrannic governments and it helps to educate that writing crypto from scratch is most definitely something that criminals can do. The government making it illegal or backdooring it will only help against lowest-effort criminals, not against organized terrorist groups. Sure, those probably won't write unbreakable encryption, but it'll be enough to bypass a government that's expecting everything to be in plain text.

bpicolo 9y ago
> but I feel like it's a discussion that needs to be had with governments around the world wanting to backdoor everything. On the contrary, this is -exactly- why you don't want to roll your own crypto. Because they can and will break it. Good crypto takes serious thought and effort from people who put a lot of thought and effort into it.

JohnStrange 9y ago

I apologize if that sounds dismissive (it's not intended to) but your post is not more than the usual vague FUD. Take a look at so-called 'professional' cryptographers and their libraries, and you'll find that practically all of their code had severe bugs. Professionals make errors like everyone else, 'professional' just means you're being paid for it. You will have a hard time finding a professional and widely used crypto library that wasn't essentially compromised in one of its functions at one time or another. Not only that, professionals from the closed-source department have a long history of coming up with compromised or bogus, sometimes even ridiculous cryptographic algorithms and implementations. Two typical examples: CMEA cell phone encryption, and the Crypto AG backdoor debacle. Surely people who invent and implement cryptographic algorithms for a living can be expected to be better than your run-off-the-mill programmer, but as I've said elsewhere implementing crypto is also not a black art. It requires about the same level of skill as e.g. implementing a production-ready low-level network protocol or a file system. That rules out many programmers but not all. That concerns implementations. As for algorithms, their development requires extensive experience in cryptanalysis (not just applied cryptography!). Some pros have that, others don't. Last but not least, many popular and widely used crypto libraries started as a side hobby. For example, libtomcrypt started that way. In fact, many widely respected cryptographers started as hobbyists. For example, Bruce Schneier. Of course, the ones who are widely accepted as peers are also those who are in the 'hire me as a consultant, not the other guy' business, and they will naturally advise everyone to not roll their own crypto but to rely on their expertise...

dsacco 9y ago
So I have two points for you: > Take a look at so-called 'professional' cryptographers and their libraries, and you'll find that practically all of their code had severe bugs. Professionals make errors like everyone else, 'professional' just means you're being paid for it. You will have a hard time finding a professional and widely used crypto library that wasn't essentially compromised in one of its functions at one time or another. Not only that, professionals from the closed-source department have a long history of coming up with compromised or bogus, sometimes even ridiculous cryptographic algorithms and implementations. Two typical examples: CMEA cell phone encryption, and the Crypto AG backdoor debacle. This doesn't diminish the claim that the modal individual should not attempt to "roll their own crypto." You're pointing out that professionals make mistakes, but of course they do - that indicates nothing about the propensity for amateurs to make mistakes. > It requires about the same level of skill as e.g. implementing a production-ready low-level network protocol or a file system. I have never implemented a low level network protocol or a file system (although at least the former sounds like a fun project) - so I cannot claim you're wrong here. However, I will point out that a network protocol is not designed to be error-proof in an intentionally antagonistic environment, which is a difficulty unique to cryptographic software. There are incentives involved in breaking crypto code that do not present themselves in other areas of software development, even if the programming difficulty itself is not entirely dissimilar. > Last but not least, many popular and widely used crypto libraries started as a side hobby. For example, libtomcrypt started that way. In fact, many widely respected cryptographers started as hobbyists. For example, Bruce Schneier. Of course, the ones who are widely accepted as peers are also those who are in the 'hire me as a consultant, not the other guy' business, and they will naturally advise everyone to not roll their own crypto but to rely on their expertise... They can start as side hobbies, but they are not going to remain that way for any meaningful amount of time if they are to be widely deployed and safe. You can only choose two out of those three. More importantly, there is no cabal of cryptographers spreading FUD to line their own pockets with consulting fees. The cryptography consulting industry is very small compared to the actual security consulting industry. In the application security consulting industry I do believe there are firms that try to secure business this way, but that industry is much larger (and has firms which try to misrepresent cryptographic competency). I have had significant interactions with engineers at Riscure and NCC Crypto, which is a sizable portion of all the "real" cryptanalytic consulting work that occurs (at least in the United States) - they never struck me as being the sort to suggest what you're saying. That leaves Rambus/CR and a select few other firms doing real crypto consulting, which leads me to believe the industry is, on the whole, very legitimate.

madez 9y ago

You can do crypto as a side-hobby and safely put it into production. Of course, you can also do it wrong, but it is with little work possible to do it correctly. Normally I wouldn't speak against what you said because the critique is only very minor, but I heard this too often. Your warning is too strong. People should try their own crypto and with little care it is not unsafer. Maybe giving a list of do's for that would be a good start.

marcoperaza 9y ago
Sorry, but this is not good advice. Widely-used crypto implementations have had the benefit of years of analysis by dozens of high-expertise stakeholders who have a lot to lose should the crypto fail. Even that isn't always enough to catch all weaknesses and vulnerabilities. This cowboy-programming attitude being extended to security is no small part of why we are so vulnerable as a society to attacks on our computer systems that can compromise our core infrastructure[1], our secrets[2], and our economic security[3]. [1] Think weeks-long country-wide power outages [2] Think juicy blackmail material on anyone with their hands on levers of power. Look at the damage that North Korea's attacks did to Sony Pictures, for example. [3] Think small-scale attacks on services that even a single major company, or many small companies, rely on.
- madez 9y ago
  You are wrongly assuming I say to throw away the work of others. I never said so.
deleted 9y ago
[deleted]
dsacco 9y ago
> You can do crypto as a side-hobby and safely put it into production. No, you really can't, unless you stretch the definition of "side-hobby" to the point of breaking at the seams. I'd be shocked if a single person who professionally works in security or cryptography is going to agree with you in this thread. You need far more than a "little care" to assure that a new cryptographic library is safe. It's an endeavor suited to a collaborative academic or corporate environment, with robust human capital and strong oversight. I'm willing to accept that someone can safely implement a primitive or a construction in a new library with significant time, effort and feedback from others. But that level of effort doesn't really qualify as a side-hobby, and until the implementation was formally checked by professional cryptographers I'd consider it suspect. This is to say nothing of designing novel primitives or constructions, which I would consider as far removed from a "side-hobby" as a local 5K is to the Olympics. However, I absolutely think people should attempt to implement their own cryptography if they're curious about it and want to learn. Just don't use it in production and assume it's unsafe. People who work in the industry don't make these claims because we think it's fun, we do it because we've all seen the consequences.

beckler 9y ago

I've found some helpful guidelines, but I know it is by no means an exhaustive list. https://cryptocoding.net/index.php/Coding_rules https://cryptocoding.net/index.php/Coding_rules

logfromblammo 9y ago

Anyone who can write code can write a crypto library. The trick is to have a second person helping you, who has all your mathematics notes, a copy of your source code, physical access to your hardware, and the ability to run your program inside a debugger. You don't even get to know whether they put cream in their coffee. If you can keep a secret from them, you're doing okay. So then you rent a smarter person to do the same thing to make sure all the tricks and shortcuts have been attempted. If someone manages to break it in the wild after that, you're still doing about as well as some of the highest-paid professionals in the industry. Defense is difficult.

stcredzero 9y ago

There is even a list of "you have to do A, B, C, and that's about it" that is missing some major - well known, even! - items. Our "field" of programming is so weak in its preservation of institutional knowledge, that we have to occasionally fight the recurrence of bad ideas about very basic things, like using string filtering to avoid SQL injection. Yes, this happens, even in "hip" new parts of the field, like the communities around newer languages. (I have a specific example in mind, but I don't want to deal with the resulting war.)

catoc 9y ago

Definition question: would you say that AgileBits (1password) are "rolling their own crypto" or 'merely' implementing proven third-party crypto?

ZoFreX 9y ago
I would have to read their source code to answer that question (I know almost nothing about 1Password). Some password managers just call a PGP executable. Others are assembling crypto primitives into larger pieces and making choices like "let's use CBC mode" themselves.