8 ms·

OpenSSL is written by monkeys

115 points by dfox 16y ago

petrilli 16y ago
I suspect he'd be happier with Cryptlib. It's a much saner toolkit, and already has most of a CA built in.
Hoff 16y ago
For what should be simple, it's not. After creating roughly twelve hundred lines of C and hundred or so lines of command language to get OpenSSL working for network connections and to get a CA and signed certs a few weeks back, I can well appreciate the author's frustration with the OpenSSL library. And at the same time this was being developed, one of the associated OS platforms went through an incompatible-API OpenSSL upgrade; you got to find and rebuild everything that was built against it, or you saw, um, cryptic failures.
dkarl 16y ago
I have had this experience many times when I tried to use a library where every function was tailored for a very specific concrete situation. The perfect example here is finding an extremely complicated mass of code to read a cert from a file, but no way to read a cert from any other source, not even from a buffer in memory. So if you read it from some other location, or read it from a file yourself, or constructed it yourself in memory as part of a test, you're out of luck. People who write code like that tend to be pretty incredulous and insulting when you explain what you're trying to do. "I mean, if you want to read a cert that isn't in a file, obviously I need to educate you until you stop wanting to do such a stupid thing. Not that it's stupid in itself, but you are probably solving your problem in such a dumb way that you should stop right now and start over when you understand what you're doing. If you're super-polite I might even help you." That kind of thing.
- dchest 16y ago
  It is not the first time I read that there's no way to read a certificate from memory in OpenSSL, but it's not true. Here's how to do it: http://gist.github.com/574388 http://gist.github.com/574388 I agree with the opinion, though, because it took me a lot of time to figure this out.
- megrimlock 16y ago
  There's a school of API design that believes you can guide users to reflect on what they are doing by deliberately excluding certain things, without explaining why they are absent. This springs from a prescriptive viewpoint that presumes the reader will inevitably arrive at the same conclusions as the author about the right way to do things. Since no one is always right, and API's are too often designed without considering what actual application usage would be like, this can be extremely frustrating. The best API's that somehow hit the sweet spot of minimal, efficient, and obvious are always extremely impressive.
- hackermom 16y ago
  The good old "I really know better than you. I even know better than you what it is that you want/need to do. Do it the way I tell you to, because that's how I would do it, and that matters more than what you want/need to do."-personality so incredibly prevalent in a certain sphere of programmers we are all very familiar with, who gladly spend 5 minutes pushing you down instead of spending 20 seconds to just kindly, sociably and helpfully answer your question.
  - albertzeyer 16y ago
    The problem is that in many cases, there is not an answer that easy. And instead of admitting that the current design is too crippled to allow such a thing or makes such a thing pretty horrible and difficult, they try to defend the design by declaring that thing as something you don't want to do anyway normally.
- e40 16y ago
  Perfect recent example. Plex v9 doesn't allow one to delete a file after you watch it. v8 had the feature and LOTS of us used it. The forums are abuzz with people complaining, and the devs are just now coming around that they might implement it (though when is anyone's guess). In the mean time, the lot of us has gone back to v8.
- bborud 16y ago
  If it was only that. But the OpenSSL code is genuinely bad.
fleitz 16y ago
Having had to reverse engineer the way openssl generates keys from passphrases for certificates I can attest to the general f'ed upness of openssl. Why did I have to reverse engineer it? Because the documentation was both lacking and wrong.
- baguasquirrel 16y ago
  The only documentation I found useful at all was the RFC. It seems that if you want to use OpenSSL, you need to read both the RFC and the source. It's also preferable to have examples of its usage in other OSS code.
- bborud 16y ago
  ...and the code is written in such a convoluted way that most people who have to deal with it never want to touch it again if they can help it.
jcroberts 16y ago
BIAS: I'm an editor on undeadly.org If anyone is interested, an overview/interview with Marco Peereboom was recently published on the OpenBSD news site. http://www.undeadly.org/cgi?action=article&sid=20100907204555&mode=expanded&count=2 http://www.undeadly.org/cgi?action=article&sid=201009072...
- sbierwagen 16y ago
  "In 2004, the head flying height was equivalent to a Boeing 747 airliner flying at 0.05 cm above the ground and travelling at 92 Km/h (7200 RPM drive)." That was in 2004. What would that translate into these days? The... same? At least, if you're using a 7200 RPM drive, but 10k and 15k drives existed in 2004 as well. Bit density has increased dramatically over the last six years, but rotational speed has remained constant, thanks to so some hard physical limits; and now that SSDs are no longer a howling black hole of suck on a dollar/GB basis, we won't see any more breakthroughs done with mechanical hard drives. Solid state drives are just better.
JeremyChase 16y ago
I'm not going to defend or support OpenSSL, but I find your tone extremely frustrating. For better or worse OpenSSL is an open library for you to use, and you should be thankful for that. Also, insulting the people working on it is no way to improve the situation.
- _delirium 16y ago
  Imagine my surprise at discovering that someone who writes with a tone like that is a prominent OpenBSD developer. ;-)
  - jcroberts 16y ago
    Have you ever actually talked with any of the OpenBSD developers? Marco is a fantastic human being and absolutely hilarious. On top of all that, he's also an amazing programmer. Yes, I've met him in person, and we've traded emails and packages for years. In fact, there's a half a pallet of donated gear sitting behind me in need of being shipped out to him. --It should go without saying, but he's a friend and I have a strong bias. Getting frustrated by widely deployed but poorly written software should be expected. Just voicing said frustrations solves nothing and wastes time, but voicing frustrations while providing an alternative is actually beneficial.
    - _delirium 16y ago
      It wasn't intended as a comment on anyone's personal worth as a human being, or what they're like in person. I have spent a good amount of time following OpenBSD-related mailing lists, though, and I'd say "polite and collegial" is not the prevailing tone--- hyperbolically trashing other people's work and calling them stupid monkeys is more par for the course. Admittedly, they don't have a monopoly on that; plenty of GNU mailing lists are similar (esp. anything RMS or Ulrich Drepper regularly posts to).
      - jcroberts 16y ago
        The majority of open source projects, OpenBSD included, have difficulty differentiating between attacking a problem and attacking a person. If you read the recent HN article: "How to keep someone with you forever" http://news.ycombinator.com/item?id=1677013 http://news.ycombinator.com/item?id=1677013 And ponder it a bit, you'll see how it applies to open source projects, and interactions on mailing lists, or as the case may be, a homepage article by an open source developer. For me at least, the more fascinating question is why open source projects eventually degrade into "sick systems" of interaction? --I wish I had an answer, but the only speculation I have is it's the result of frustration.
gw 16y ago
I don't understand why he persisted in using OpenSSL. There may not have been a lot of alternatives ten years ago, but we developers are practically spoiled with choices now. I heartily recommend Botan, which is BSD licensed, written in C++, and has a clean API. I loved it so much I donated to the developer.
- jcroberts 16y ago
  You're only seeing the very first steps, but not noticing where the intent is. Building an entirely new replacement implementation will fight the battle of adoption. By wrapping initially an abstraction around the existing code, you can then go back and replace the original while at the same time, keeping everything relying on the original working. If you see Marco's initial work as a "shim" to enable replacement, it makes a lot more sense.
jrockway 16y ago
If there's one thing we don't hear enough in the programming world, it's "all this code I didn't write sucks!" Oh wait, that's about all we ever hear. It makes for boring reading (and boring comments), so I'm flagging the article. When you write about how you've rewritten OpenSSL to be more flexible and featureful, I will enjoy reading about that. This? It just makes me mad. (If you want to whine about a particular issue with a particular library, at least try to generalize it to something meaningful. The lesson here is "be careful writing a library that only one app uses and then calling it a library". A lot of C programs make this mistake; sure, they have a small app that calls libapp to do all the real work... but the functions that do the real work are things like read_config_file_and_then_handle_the_applications_foo_command, which is really unhelpful to everyone.) Anyway... OpenBSD guy whining about code he didn't write? Big surprise. On HN, I want valuable articles that show me something cool or teach me how not to be uncool. This is just a rant that has no real value.
- dschobel 16y ago
  In his defense, he does provide all of the code for how he ultimately solved his problem.
- rjurney 16y ago
  He did rewrite it to be more flexible. So while you are right, I'm down-voting you for not reading the fucking article ;)
  - acqq 16y ago
    I've looked at his code. He didn't rewrite it, just wrote the wrappers to the API calls to make his usage cases more convenient for him, which is what every beginner would also do.
    - rjurney 16y ago
      He gave it a sane interface for a broader use case, with sane code that he published. Mission accomplished. Not just complaining.
manelito56 16y ago
Try gnutls
- manelito56 16y ago
  and there ares BIOs to read from memory so your point is wrong. The documentation is a mess though.
viraptor 16y ago
I have to wonder - if the API was so bad, but he didn't have to do anything that isn't available from openssl "the tool"... why did he write the code at all? Excerpt from openssl x509 help: usage: x509 args -in arg - input file - default stdin -out arg - output file - default stdout It does reading / writing via std{in,out} pipes - why not just spawn openssl with the right options and interact with it?
- ciupicri 16y ago
  Maybe he wanted something a bit more efficient.
  - hapless 16y ago
    If you're generating new CAs in such volume that the (in)efficiency of a UNIX pipe becomes a problem, you should just stop right there. That should be a red flag.
    - FooBarWidget 16y ago
      It's not the pipes, it's fork() and exec() and re-main()-initialization. Try doing that a few ten thousand times and see how long it takes compare to doing everything in-process where you only initialize once.
      - CGamesPlay 16y ago
        The entire internet gets by on a few hundred CAs. If you are generating so many CAs, you are doing something wrong. Performance is a misdirected concern, here. Error-handling and reliability are better reasons to use the API over the tool.
    - ciupicri 16y ago
      While I'm not a fan of premature optimization, I don't see nothing wrong here even if I like the Unix philosophy of "Write programs that do one thing and do it well". Why run an external program when the same thing should be possible by calling functions from a (well written) library?
squanderingtime 16y ago
While I generally agree that the OpenSSL library is somewhat horrendous, this specific use-case addresses a very tiny subset of what the library does internally to do some real magic. I used to work for a very large public facing certificate authority and there was a period in which one of the public CA systems used OpenSSL, so we had to be able to call several of the signing functions via a JNI extension. Why? Because OpenSSL has fairly remarkable support fir internal context objects that allow you to designate PKCS#11-interfaced hardware security modules for private key operations. Yes, the levels of indirection are painful to read through and when you get down to the levels where you can actually do a signing operation you have multi-level points (something) of strange types, but it works almost magically when you need it to. It's also the combined with the intersection of handling/reading certificates. The IO interface has to correctly handle PEM and DER information. Even if you can parse those structures all you're left with is ASN.1 annotated structures and ASN.1 is an exercise in pain. So a library that can read a broad format of single-bit sensitive data, correctly decode and preserve the structure of ASN.1 entries, perform crypto operations, and then somehow make sense of it all across how many years is mostly likely going to be complicated as hell. Or at the very least, given my proficiency with C I think I would have a hard time coming up with a library that worked on as many platforms as transparently for as many years.
cicada 16y ago
Shouldn't there be a debate about the technical merits of the idea? I want my libraries and tools to make unwise things hard to do, and the wise easy and obvious. It's a framing issue, one that I think Marco's right about. But I'd like to hear what OpenSSL has to say about this and other, other, other things.
m4dc4p 16y ago
old
ChuckN 16y ago
It seems like he should use a higher level language to do this stuff: http://docs.python.org/dev/library/ssl.html http://docs.python.org/dev/library/ssl.html Very well documented and does most of what he wants.
16s 16y ago
If you do C++, try Crypto++ it's very nice.
deleted 16y ago
[deleted]
phenylene 16y ago
I'm using MatrixSSL (http://www.matrixssl.org http://www.matrixssl.org) in my product. It's cross platform, small footprint, has dual commercial / GPL license options and a sane API.
soitgoes 16y ago
Complex code is never good for security. In 2006, a small change was made to the Debian port of OpenSSL to do something like remove a few compiler warning messages. Unfortunately, it had the side effect of making the random number generator predictable. This security flaw wasn't discovered for over a year and in that time many weak keys were generated and used. See here for more info: http://wiki.debian.org/SSLkeys http://wiki.debian.org/SSLkeys
mike-cardwell 16y ago
I'd be interested to know how GNUTLS compares

115 points by dfox 16y ago

I suspect he'd be happier with Cryptlib. It's a much saner toolkit, and already has most of a CA built in.

Hoff 16y ago

For what should be simple, it's not. After creating roughly twelve hundred lines of C and hundred or so lines of command language to get OpenSSL working for network connections and to get a CA and signed certs a few weeks back, I can well appreciate the author's frustration with the OpenSSL library. And at the same time this was being developed, one of the associated OS platforms went through an incompatible-API OpenSSL upgrade; you got to find and rebuild everything that was built against it, or you saw, um, cryptic failures.

dkarl 16y ago

I have had this experience many times when I tried to use a library where every function was tailored for a very specific concrete situation. The perfect example here is finding an extremely complicated mass of code to read a cert from a file, but no way to read a cert from any other source, not even from a buffer in memory. So if you read it from some other location, or read it from a file yourself, or constructed it yourself in memory as part of a test, you're out of luck. People who write code like that tend to be pretty incredulous and insulting when you explain what you're trying to do. "I mean, if you want to read a cert that isn't in a file, obviously I need to educate you until you stop wanting to do such a stupid thing. Not that it's stupid in itself, but you are probably solving your problem in such a dumb way that you should stop right now and start over when you understand what you're doing. If you're super-polite I might even help you." That kind of thing.

dchest 16y ago
It is not the first time I read that there's no way to read a certificate from memory in OpenSSL, but it's not true. Here's how to do it: http://gist.github.com/574388 http://gist.github.com/574388 I agree with the opinion, though, because it took me a lot of time to figure this out.
megrimlock 16y ago
There's a school of API design that believes you can guide users to reflect on what they are doing by deliberately excluding certain things, without explaining why they are absent. This springs from a prescriptive viewpoint that presumes the reader will inevitably arrive at the same conclusions as the author about the right way to do things. Since no one is always right, and API's are too often designed without considering what actual application usage would be like, this can be extremely frustrating. The best API's that somehow hit the sweet spot of minimal, efficient, and obvious are always extremely impressive.
hackermom 16y ago
The good old "I really know better than you. I even know better than you what it is that you want/need to do. Do it the way I tell you to, because that's how I would do it, and that matters more than what you want/need to do."-personality so incredibly prevalent in a certain sphere of programmers we are all very familiar with, who gladly spend 5 minutes pushing you down instead of spending 20 seconds to just kindly, sociably and helpfully answer your question.
- albertzeyer 16y ago
  The problem is that in many cases, there is not an answer that easy. And instead of admitting that the current design is too crippled to allow such a thing or makes such a thing pretty horrible and difficult, they try to defend the design by declaring that thing as something you don't want to do anyway normally.
e40 16y ago
Perfect recent example. Plex v9 doesn't allow one to delete a file after you watch it. v8 had the feature and LOTS of us used it. The forums are abuzz with people complaining, and the devs are just now coming around that they might implement it (though when is anyone's guess). In the mean time, the lot of us has gone back to v8.
bborud 16y ago
If it was only that. But the OpenSSL code is genuinely bad.

fleitz 16y ago

Having had to reverse engineer the way openssl generates keys from passphrases for certificates I can attest to the general f'ed upness of openssl. Why did I have to reverse engineer it? Because the documentation was both lacking and wrong.

baguasquirrel 16y ago
The only documentation I found useful at all was the RFC. It seems that if you want to use OpenSSL, you need to read both the RFC and the source. It's also preferable to have examples of its usage in other OSS code.
bborud 16y ago
...and the code is written in such a convoluted way that most people who have to deal with it never want to touch it again if they can help it.

jcroberts 16y ago

BIAS: I'm an editor on undeadly.org If anyone is interested, an overview/interview with Marco Peereboom was recently published on the OpenBSD news site. http://www.undeadly.org/cgi?action=article&sid=20100907204555&mode=expanded&count=2 http://www.undeadly.org/cgi?action=article&sid=201009072...

sbierwagen 16y ago
"In 2004, the head flying height was equivalent to a Boeing 747 airliner flying at 0.05 cm above the ground and travelling at 92 Km/h (7200 RPM drive)." That was in 2004. What would that translate into these days? The... same? At least, if you're using a 7200 RPM drive, but 10k and 15k drives existed in 2004 as well. Bit density has increased dramatically over the last six years, but rotational speed has remained constant, thanks to so some hard physical limits; and now that SSDs are no longer a howling black hole of suck on a dollar/GB basis, we won't see any more breakthroughs done with mechanical hard drives. Solid state drives are just better.

JeremyChase 16y ago

I'm not going to defend or support OpenSSL, but I find your tone extremely frustrating. For better or worse OpenSSL is an open library for you to use, and you should be thankful for that. Also, insulting the people working on it is no way to improve the situation.

_delirium 16y ago
Imagine my surprise at discovering that someone who writes with a tone like that is a prominent OpenBSD developer. ;-)
- jcroberts 16y ago
  Have you ever actually talked with any of the OpenBSD developers? Marco is a fantastic human being and absolutely hilarious. On top of all that, he's also an amazing programmer. Yes, I've met him in person, and we've traded emails and packages for years. In fact, there's a half a pallet of donated gear sitting behind me in need of being shipped out to him. --It should go without saying, but he's a friend and I have a strong bias. Getting frustrated by widely deployed but poorly written software should be expected. Just voicing said frustrations solves nothing and wastes time, but voicing frustrations while providing an alternative is actually beneficial.
  - _delirium 16y ago
    It wasn't intended as a comment on anyone's personal worth as a human being, or what they're like in person. I have spent a good amount of time following OpenBSD-related mailing lists, though, and I'd say "polite and collegial" is not the prevailing tone--- hyperbolically trashing other people's work and calling them stupid monkeys is more par for the course. Admittedly, they don't have a monopoly on that; plenty of GNU mailing lists are similar (esp. anything RMS or Ulrich Drepper regularly posts to).
    - jcroberts 16y ago
      The majority of open source projects, OpenBSD included, have difficulty differentiating between attacking a problem and attacking a person. If you read the recent HN article: "How to keep someone with you forever" http://news.ycombinator.com/item?id=1677013 http://news.ycombinator.com/item?id=1677013 And ponder it a bit, you'll see how it applies to open source projects, and interactions on mailing lists, or as the case may be, a homepage article by an open source developer. For me at least, the more fascinating question is why open source projects eventually degrade into "sick systems" of interaction? --I wish I had an answer, but the only speculation I have is it's the result of frustration.

gw 16y ago

I don't understand why he persisted in using OpenSSL. There may not have been a lot of alternatives ten years ago, but we developers are practically spoiled with choices now. I heartily recommend Botan, which is BSD licensed, written in C++, and has a clean API. I loved it so much I donated to the developer.

jcroberts 16y ago
You're only seeing the very first steps, but not noticing where the intent is. Building an entirely new replacement implementation will fight the battle of adoption. By wrapping initially an abstraction around the existing code, you can then go back and replace the original while at the same time, keeping everything relying on the original working. If you see Marco's initial work as a "shim" to enable replacement, it makes a lot more sense.

jrockway 16y ago

If there's one thing we don't hear enough in the programming world, it's "all this code I didn't write sucks!" Oh wait, that's about all we ever hear. It makes for boring reading (and boring comments), so I'm flagging the article. When you write about how you've rewritten OpenSSL to be more flexible and featureful, I will enjoy reading about that. This? It just makes me mad. (If you want to whine about a particular issue with a particular library, at least try to generalize it to something meaningful. The lesson here is "be careful writing a library that only one app uses and then calling it a library". A lot of C programs make this mistake; sure, they have a small app that calls libapp to do all the real work... but the functions that do the real work are things like read_config_file_and_then_handle_the_applications_foo_command, which is really unhelpful to everyone.) Anyway... OpenBSD guy whining about code he didn't write? Big surprise. On HN, I want valuable articles that show me something cool or teach me how not to be uncool. This is just a rant that has no real value.

dschobel 16y ago
In his defense, he does provide all of the code for how he ultimately solved his problem.
rjurney 16y ago
He did rewrite it to be more flexible. So while you are right, I'm down-voting you for not reading the fucking article ;)
- acqq 16y ago
  I've looked at his code. He didn't rewrite it, just wrote the wrappers to the API calls to make his usage cases more convenient for him, which is what every beginner would also do.
  - rjurney 16y ago
    He gave it a sane interface for a broader use case, with sane code that he published. Mission accomplished. Not just complaining.

manelito56 16y ago

Try gnutls

manelito56 16y ago
and there ares BIOs to read from memory so your point is wrong. The documentation is a mess though.

viraptor 16y ago

I have to wonder - if the API was so bad, but he didn't have to do anything that isn't available from openssl "the tool"... why did he write the code at all? Excerpt from openssl x509 help: usage: x509 args -in arg - input file - default stdin -out arg - output file - default stdout It does reading / writing via std{in,out} pipes - why not just spawn openssl with the right options and interact with it?

ciupicri 16y ago
Maybe he wanted something a bit more efficient.
- hapless 16y ago
  If you're generating new CAs in such volume that the (in)efficiency of a UNIX pipe becomes a problem, you should just stop right there. That should be a red flag.
  - FooBarWidget 16y ago
    It's not the pipes, it's fork() and exec() and re-main()-initialization. Try doing that a few ten thousand times and see how long it takes compare to doing everything in-process where you only initialize once.
    - CGamesPlay 16y ago
      The entire internet gets by on a few hundred CAs. If you are generating so many CAs, you are doing something wrong. Performance is a misdirected concern, here. Error-handling and reliability are better reasons to use the API over the tool.
  - ciupicri 16y ago
    While I'm not a fan of premature optimization, I don't see nothing wrong here even if I like the Unix philosophy of "Write programs that do one thing and do it well". Why run an external program when the same thing should be possible by calling functions from a (well written) library?

squanderingtime 16y ago

While I generally agree that the OpenSSL library is somewhat horrendous, this specific use-case addresses a very tiny subset of what the library does internally to do some real magic. I used to work for a very large public facing certificate authority and there was a period in which one of the public CA systems used OpenSSL, so we had to be able to call several of the signing functions via a JNI extension. Why? Because OpenSSL has fairly remarkable support fir internal context objects that allow you to designate PKCS#11-interfaced hardware security modules for private key operations. Yes, the levels of indirection are painful to read through and when you get down to the levels where you can actually do a signing operation you have multi-level points (something) of strange types, but it works almost magically when you need it to. It's also the combined with the intersection of handling/reading certificates. The IO interface has to correctly handle PEM and DER information. Even if you can parse those structures all you're left with is ASN.1 annotated structures and ASN.1 is an exercise in pain. So a library that can read a broad format of single-bit sensitive data, correctly decode and preserve the structure of ASN.1 entries, perform crypto operations, and then somehow make sense of it all across how many years is mostly likely going to be complicated as hell. Or at the very least, given my proficiency with C I think I would have a hard time coming up with a library that worked on as many platforms as transparently for as many years.

cicada 16y ago

Shouldn't there be a debate about the technical merits of the idea? I want my libraries and tools to make unwise things hard to do, and the wise easy and obvious. It's a framing issue, one that I think Marco's right about. But I'd like to hear what OpenSSL has to say about this and other, other, other things.

m4dc4p 16y ago

old

ChuckN 16y ago

It seems like he should use a higher level language to do this stuff: http://docs.python.org/dev/library/ssl.html http://docs.python.org/dev/library/ssl.html Very well documented and does most of what he wants.

16s 16y ago

If you do C++, try Crypto++ it's very nice.

deleted 16y ago

[deleted]

phenylene 16y ago

I'm using MatrixSSL (http://www.matrixssl.org http://www.matrixssl.org) in my product. It's cross platform, small footprint, has dual commercial / GPL license options and a sane API.

soitgoes 16y ago

Complex code is never good for security. In 2006, a small change was made to the Debian port of OpenSSL to do something like remove a few compiler warning messages. Unfortunately, it had the side effect of making the random number generator predictable. This security flaw wasn't discovered for over a year and in that time many weak keys were generated and used. See here for more info: http://wiki.debian.org/SSLkeys http://wiki.debian.org/SSLkeys

mike-cardwell 16y ago

I'd be interested to know how GNUTLS compares