4 ms·

Not sure the trade off line has been drawn in the correct place here. TLS should be implemented as a filter; the world isn't limited to files and (blocking) soc

Not sure the trade off line has been drawn in the correct place here. TLS should be implemented as a filter; the world isn't limited to files and (blocking) sockets. What about integration with event loops, for example? How does one test this API?

amelius 12y ago
Indeed. Functions like tls_connect and tls_connect_fds don't belong at this level. A TLS library should do only TLS, and behave like a black box, from which you pull data and into which you can push data (in a non-blocking way). Mixing in OS-related stuff only complicates the API.
- wereHamster 12y ago
  1) Optimizing for the most common use case. 2) You can create file descriptors not backed by any actual network connection or file on the harddrive (man 2 pipe and man 2 socketpair), you can still build an API on top of that which operates purely on buffers.
  - amelius 12y ago
    > 1) Optimizing for the most common use case My compiler is most commonly used together with an editor. Still I'm glad it didn't come with an editor built-in :) > 2) You can create file descriptors not backed by any actual network connection or file on the harddrive Indeed. It doesn't mean it is the preferred way to do it, from a code-quality or even performance point of view.
  - zAy0LfpBZLC8mAC 12y ago
    Yes, you can, if your goal is massive bloat. If you want to have reasonable performance and scalability, a blocking API (that is to say, at least one OS thread per connection) and loads of IPC syscall overhead is about the worst thing you could do.
  - zzzcpan 12y ago
    You are wrong. Blocking socket I/O is rather uncommon in networking software.
- adamtj 12y ago
  But, I don't want to do TLS. I'm not in the encryption business. Instead, I just want to send data across the network. The OS used to do the networking for me. It exposes the BSD sockets API, which is pretty simple and nice but is unfortunately no longer safe to use. Yet, the kernel doesn't offer anything else. So you're right, but you don't go far enough. It should all be in the kernel. Instead of tls_config_set_cert_file() we should be using setsockopt(2). Until we can, however, we'll have to rely on user-space libraries.
  - AnthonyMouse 12y ago
    > It exposes the BSD sockets API, which is pretty simple and nice but is unfortunately no longer safe to use. This is the second time you've said something like this today. Explain what you mean by it. > So you're right, but you don't go far enough. It should all be in the kernel. Instead of tls_config_set_cert_file() we should be using setsockopt(2). Until we can, however, we'll have to rely on user-space libraries. I think what you mean is that it should be in the standard library. There is no reason it should have to be in the kernel.
- geofft 12y ago
  It complicates the implementation, but I'd argue it simplifies the API... one of the things you want to get absolutely right is the binding between hostname and server certificate. If you separate the two, there's a good chance that the TLS code won't even know what hostname it's supposed to be talking to (and I think I've heard of two codebases that required some refactoring for the hostname to get to the right place). A tls_connect that takes a string forces you to do the right thing, and the primary design goal I want out of a TLS API is that it should force me to do the right thing.
parley 12y ago
I agree whole-heartedly, and wrote as much in another comment below. This should not be overlooked.
IgorPartola 12y ago
Why not both? A low level API where you create a context, then give it [clear|cipher]text and it gives you [cipher|clear]text, and and API that builds on top of it to provide socket communication? I don't see why the two are mutually exclusive. Now, your complaint about event loop integration is very valid. You should be able to use select/epoll/kqueue/etc. on TLS sockets without having to worry about what your libssl is doing. I wrote exactly one program that uses libssl in C directly. It is a special-purpose web server that can serve files or stream content over an HTTPS connection [1]. It was a huge pain to get my event loop to play nice with TLS because TLS does more reads/writes where a plain HTTP connection would do just reads or just writes. I also had to enable various non-defaults (SSL_MODE_ENABLE_PARTIAL_WRITE | SSL_MODE_RELEASE_BUFFERS) to get it to behave more like a regular socket. Having a more standard way to support stuff like this would be great, and it does exist in other places. For example, Python wraps libssl in such a way that you can just wrap a plain socket into a TLS socket, and it pretty much just works. It'd be great if stuff like this existed at a lower level where this new API seems to live. [1] https://github.com/ipartola/hawkeye https://github.com/ipartola/hawkeye
deleted 12y ago
[deleted]
drdaeman 12y ago
From a perfectionist viewpoint, the decision's almost correct: TLS work on streams and fds are streams in POSIX. Although an ANSI C's FILE* would be a more portable thus wiser choice. However, given that the world's flawed and FILEs (and fds) aren't proper objects that end-user can create and use with their own functions (like an implementation of read(2) that'd yield to an event loop until the data's there), guess it warrants yet another ad-hoc struct. Or a bunch of functions that work on some context struct and that you're supposed to pass data chunks by yourself. Like in classic OpenSSL API.
quotemstr 12y ago
Agreed. OpenSSL's BIO system is actually very nice and is one of the nicer things about the library. Removing it is short-sighted.
- pquerna 12y ago
  Yes, and BIOs are one of the parts that basically every major open source project uses: Apache, Nginx, Node.js, curl, etc -- they all use the BIO API to provide TLS as a filter over their internal streaming APIs. Having personally worked on some of these projects, the current libtls API is not sufficient at all. This isn't to say at all that OpenSSL's BIO API was great -- it could definitely use iteration all over -- and I hope libtls gets there, but it doesn't seem at all focused on realistic server workloads.
  - X-Istence 12y ago
    The only reason you need the BIO API is because it is the only way to push data into OpenSSL, there is no reason why this can't be done with buffers you manage. For example: http://funcptr.net/2012/04/08/openssl-as-a-filter-(or-non-blocking-openssl)/ http://funcptr.net/2012/04/08/openssl-as-a-filter-(or-non-bl... is a blog post I wrote wherein I show how to use SSL as a filter. The only reason I use a BIO is to temporarily store the same data I just got from the network and have in an std::string... it's duplication.
    - quotemstr 12y ago
      BIO is just a thin, generic interface for reading bytes. If you want to use std::string, you can easily write an adapter.
jameshart 12y ago
Not sure if this applies in this case, but perhaps this is an attempt to address a common problem with crypto, which is that you have to get a whole lot of stuff around the usage of the basic crypto primitives right to avoid inadvertently opening up sidechannel vulnerabilities, or weakening the crypto. By binding the crypto to particular ways of getting data in and out, perhaps it eliminates implementation risks?
adamtj 12y ago
I'm not sure there's only one line to be drawn. There is absolutely a place for an API like this -- but that place is not as a replacement for OpenSSL. The world has changed. We must now consider send(2) and recv(2) to be toxic. They are fundamentally broken and cannot be used safely. The problem is, what do we use in their place? Simple sockets should continue to be simple to use, even now that we need to layer encryption on top. OpenSSL is not a practical replacement for those functions. This new API is a pretty good attempt. It may not do everything that OpenSSL does, but it doesn't need to. It only needs to make the simple things easy. If you have a different use case, you should use a different high-level API. I think the way to understand this API is to see it as a "view", a wrapper, an abstraction. It's like a database view that hides the dozen tricky joins necessary to answer a frequently asked question. Also, like a database view, it is one of potentially many, each with a different purpose. As a wrapper, this API exposes only a subset of the full functionality, but it's a very useful subset. Perhaps 80% of the complexity of the OpenSSL API comes from only 20% of use cases. This API exposes only a small subset of functionality, but that's what makes it simple and easy to use and it still covers 80% of use cases. If you aren't in that 80%, then this API isn't for you. But that's ok! It seems reasonable to expect multiple layers of cryptography libraries. We ought to have a set of low-level libraries that implement the fundamental primitives in a general and orthogonal way. It would be a sort of cryptography "kernel". Most developers wouldn't use it in the same way that most don't directly make syscalls into the OS kernel, but instead use a high-lever wrapper library. To open a file, for example, we often use a wrapper like the functions in a libc, or a wrapper of a wrapper like libsqlite3. So, if an API doesn't cover your particular use case, that doesn't mean it's bad. You just need to find a different API. The uses for cryptography are myriad. It seems unreasonable to expect a single API to cover everything while remaining simple and easy to use.
- parley 12y ago
  > There is absolutely a place for an API like this -- but that place is not as a replacement for OpenSSL. I disagree. I think it is exactly the place for an OpenSSL API replacement. As is alluded to in this link (thanks cremno) [1], they want a critical mass of users before it makes sense to deprecate the libssl API. Deprecation seems necessary in order to perform the serious cleanup/reimplementation under the hood that they would like to do. While new projects may have an easier time, they admit that adoption by existing projects is significant work. However, so many and large projects already exist that it seems that many of them need to be won over in order for a deprecation to be practical. Many existing projects (see other comments in this post) use this transport layer flexibility in the OpenSSL API, and not offering it with the new API may slow adoption. Yes, like you say another API layer could be added, and it is a deeply subjective matter so let's agree to disagree... But composability is so important and only becomes more so as we tire of wheel reinventions leading to unnecessary bugs. Enabling modularity and reuse needs 1st class API attention. Being able to easily plug this new API as a source/sink onto the myriad of existing (and yet to be created) I/O frameworks would be a major win. This is my $.02 and humble appeal to the people doing the fantastic work on this lib. My thanks to all of you! [1] https://marc.info/?l=openbsd-tech&m=141524972826918&w=2 https://marc.info/?l=openbsd-tech&m=141524972826918&w=2

amelius 12y ago

Indeed. Functions like tls_connect and tls_connect_fds don't belong at this level. A TLS library should do only TLS, and behave like a black box, from which you pull data and into which you can push data (in a non-blocking way). Mixing in OS-related stuff only complicates the API.

wereHamster 12y ago
1) Optimizing for the most common use case. 2) You can create file descriptors not backed by any actual network connection or file on the harddrive (man 2 pipe and man 2 socketpair), you can still build an API on top of that which operates purely on buffers.
- amelius 12y ago
  > 1) Optimizing for the most common use case My compiler is most commonly used together with an editor. Still I'm glad it didn't come with an editor built-in :) > 2) You can create file descriptors not backed by any actual network connection or file on the harddrive Indeed. It doesn't mean it is the preferred way to do it, from a code-quality or even performance point of view.
- zAy0LfpBZLC8mAC 12y ago
  Yes, you can, if your goal is massive bloat. If you want to have reasonable performance and scalability, a blocking API (that is to say, at least one OS thread per connection) and loads of IPC syscall overhead is about the worst thing you could do.
- zzzcpan 12y ago
  You are wrong. Blocking socket I/O is rather uncommon in networking software.
adamtj 12y ago
But, I don't want to do TLS. I'm not in the encryption business. Instead, I just want to send data across the network. The OS used to do the networking for me. It exposes the BSD sockets API, which is pretty simple and nice but is unfortunately no longer safe to use. Yet, the kernel doesn't offer anything else. So you're right, but you don't go far enough. It should all be in the kernel. Instead of tls_config_set_cert_file() we should be using setsockopt(2). Until we can, however, we'll have to rely on user-space libraries.
- AnthonyMouse 12y ago
  > It exposes the BSD sockets API, which is pretty simple and nice but is unfortunately no longer safe to use. This is the second time you've said something like this today. Explain what you mean by it. > So you're right, but you don't go far enough. It should all be in the kernel. Instead of tls_config_set_cert_file() we should be using setsockopt(2). Until we can, however, we'll have to rely on user-space libraries. I think what you mean is that it should be in the standard library. There is no reason it should have to be in the kernel.
geofft 12y ago
It complicates the implementation, but I'd argue it simplifies the API... one of the things you want to get absolutely right is the binding between hostname and server certificate. If you separate the two, there's a good chance that the TLS code won't even know what hostname it's supposed to be talking to (and I think I've heard of two codebases that required some refactoring for the hostname to get to the right place). A tls_connect that takes a string forces you to do the right thing, and the primary design goal I want out of a TLS API is that it should force me to do the right thing.

parley 12y ago

I agree whole-heartedly, and wrote as much in another comment below. This should not be overlooked.

IgorPartola 12y ago

Why not both? A low level API where you create a context, then give it [clear|cipher]text and it gives you [cipher|clear]text, and and API that builds on top of it to provide socket communication? I don't see why the two are mutually exclusive. Now, your complaint about event loop integration is very valid. You should be able to use select/epoll/kqueue/etc. on TLS sockets without having to worry about what your libssl is doing. I wrote exactly one program that uses libssl in C directly. It is a special-purpose web server that can serve files or stream content over an HTTPS connection [1]. It was a huge pain to get my event loop to play nice with TLS because TLS does more reads/writes where a plain HTTP connection would do just reads or just writes. I also had to enable various non-defaults (SSL_MODE_ENABLE_PARTIAL_WRITE | SSL_MODE_RELEASE_BUFFERS) to get it to behave more like a regular socket. Having a more standard way to support stuff like this would be great, and it does exist in other places. For example, Python wraps libssl in such a way that you can just wrap a plain socket into a TLS socket, and it pretty much just works. It'd be great if stuff like this existed at a lower level where this new API seems to live. [1] https://github.com/ipartola/hawkeye https://github.com/ipartola/hawkeye

deleted 12y ago

[deleted]

drdaeman 12y ago

From a perfectionist viewpoint, the decision's almost correct: TLS work on streams and fds are streams in POSIX. Although an ANSI C's FILE* would be a more portable thus wiser choice. However, given that the world's flawed and FILEs (and fds) aren't proper objects that end-user can create and use with their own functions (like an implementation of read(2) that'd yield to an event loop until the data's there), guess it warrants yet another ad-hoc struct. Or a bunch of functions that work on some context struct and that you're supposed to pass data chunks by yourself. Like in classic OpenSSL API.

quotemstr 12y ago

Agreed. OpenSSL's BIO system is actually very nice and is one of the nicer things about the library. Removing it is short-sighted.

pquerna 12y ago
Yes, and BIOs are one of the parts that basically every major open source project uses: Apache, Nginx, Node.js, curl, etc -- they all use the BIO API to provide TLS as a filter over their internal streaming APIs. Having personally worked on some of these projects, the current libtls API is not sufficient at all. This isn't to say at all that OpenSSL's BIO API was great -- it could definitely use iteration all over -- and I hope libtls gets there, but it doesn't seem at all focused on realistic server workloads.
- X-Istence 12y ago
  The only reason you need the BIO API is because it is the only way to push data into OpenSSL, there is no reason why this can't be done with buffers you manage. For example: http://funcptr.net/2012/04/08/openssl-as-a-filter-(or-non-blocking-openssl)/ http://funcptr.net/2012/04/08/openssl-as-a-filter-(or-non-bl... is a blog post I wrote wherein I show how to use SSL as a filter. The only reason I use a BIO is to temporarily store the same data I just got from the network and have in an std::string... it's duplication.
  - quotemstr 12y ago
    BIO is just a thin, generic interface for reading bytes. If you want to use std::string, you can easily write an adapter.

jameshart 12y ago

Not sure if this applies in this case, but perhaps this is an attempt to address a common problem with crypto, which is that you have to get a whole lot of stuff around the usage of the basic crypto primitives right to avoid inadvertently opening up sidechannel vulnerabilities, or weakening the crypto. By binding the crypto to particular ways of getting data in and out, perhaps it eliminates implementation risks?

adamtj 12y ago

I'm not sure there's only one line to be drawn. There is absolutely a place for an API like this -- but that place is not as a replacement for OpenSSL. The world has changed. We must now consider send(2) and recv(2) to be toxic. They are fundamentally broken and cannot be used safely. The problem is, what do we use in their place? Simple sockets should continue to be simple to use, even now that we need to layer encryption on top. OpenSSL is not a practical replacement for those functions. This new API is a pretty good attempt. It may not do everything that OpenSSL does, but it doesn't need to. It only needs to make the simple things easy. If you have a different use case, you should use a different high-level API. I think the way to understand this API is to see it as a "view", a wrapper, an abstraction. It's like a database view that hides the dozen tricky joins necessary to answer a frequently asked question. Also, like a database view, it is one of potentially many, each with a different purpose. As a wrapper, this API exposes only a subset of the full functionality, but it's a very useful subset. Perhaps 80% of the complexity of the OpenSSL API comes from only 20% of use cases. This API exposes only a small subset of functionality, but that's what makes it simple and easy to use and it still covers 80% of use cases. If you aren't in that 80%, then this API isn't for you. But that's ok! It seems reasonable to expect multiple layers of cryptography libraries. We ought to have a set of low-level libraries that implement the fundamental primitives in a general and orthogonal way. It would be a sort of cryptography "kernel". Most developers wouldn't use it in the same way that most don't directly make syscalls into the OS kernel, but instead use a high-lever wrapper library. To open a file, for example, we often use a wrapper like the functions in a libc, or a wrapper of a wrapper like libsqlite3. So, if an API doesn't cover your particular use case, that doesn't mean it's bad. You just need to find a different API. The uses for cryptography are myriad. It seems unreasonable to expect a single API to cover everything while remaining simple and easy to use.

parley 12y ago
> There is absolutely a place for an API like this -- but that place is not as a replacement for OpenSSL. I disagree. I think it is exactly the place for an OpenSSL API replacement. As is alluded to in this link (thanks cremno) [1], they want a critical mass of users before it makes sense to deprecate the libssl API. Deprecation seems necessary in order to perform the serious cleanup/reimplementation under the hood that they would like to do. While new projects may have an easier time, they admit that adoption by existing projects is significant work. However, so many and large projects already exist that it seems that many of them need to be won over in order for a deprecation to be practical. Many existing projects (see other comments in this post) use this transport layer flexibility in the OpenSSL API, and not offering it with the new API may slow adoption. Yes, like you say another API layer could be added, and it is a deeply subjective matter so let's agree to disagree... But composability is so important and only becomes more so as we tire of wheel reinventions leading to unnecessary bugs. Enabling modularity and reuse needs 1st class API attention. Being able to easily plug this new API as a source/sink onto the myriad of existing (and yet to be created) I/O frameworks would be a major win. This is my $.02 and humble appeal to the people doing the fantastic work on this lib. My thanks to all of you! [1] https://marc.info/?l=openbsd-tech&m=141524972826918&w=2 https://marc.info/?l=openbsd-tech&m=141524972826918&w=2