3 ms·

Ask HN: Why is there no standard encryption file format?

When encrypting strings or files, application developers have to make choices about what cipher/mode to use, where to store the IV and how to represent all of that in the database or on disk. This is annoying at best, but also leads to errors. I've searched for RFCs but I couldn't find any standard, and not even a defacto standard. Why does this not exist? Am I missing something? If we had such a format, libraries could define a simple "encrypt()" function that would pick today's suggested cipher spec (e.g. AES128/GCM), but could provide backwards compatibility for the corresponding "decrypt()" function -- similar to hash mechanisms supporting the crypt syntax ($2y$...) such as PHP's password_hash(). I've defined such a format before for an open source project I worked on years ago [1], so I don't think it would be hard to define a standard like this. Am I overlooking something here? [1] scroll down to "crypto file format" - https://syncany.readthedocs.io/en/latest/security.html#encrypting-new-files

verdverm 7y ago
Implementation matters and the same cipher may be good in one language and broken in another. What happens when I cipher becomes exploitable? How would you re-encrypt with only a single en/decrypt functions? Which cipher was originally used?
- binwiederhier 7y ago
  Hm. I don't buy the "implementation matters" argument. If AES-128-GCM is broken in a library or language that'd be a pretty big deal. Cipher implementations should be interoperable, otherwise you'd have a big problem. Your other argument is totally valid. If a cipher becomes exploitable, you'd treat it the same way as you would with password hashes. You'd need to re-encrypt. password_hash() is designed to hash with today's best hash. My question was mainly about the file format though. The encrypt/decrypt functions would be a simple perk that becomes possible if we had that. Edit: encrypt->hash
  - verdverm 7y ago
    By broken, I don't mean incorrect, just poorly implemented and there for exploitable. This is true of libraries in the wild. File format is typically binary or blob. It's encrypted so you can't see or understand it's structure, only assume.
jolmg 7y ago
> I've searched for RFCs but I couldn't find any standard, and not even a defacto standard. How about RFC 4880 - OpenPGP Message Format[1]? [1] https://tools.ietf.org/html/rfc4880 https://tools.ietf.org/html/rfc4880
- insomniacity 7y ago
  This page explains some of the problems with the format: https://latacora.micro.blog/2019/07/16/the-pgp-problem.html https://latacora.micro.blog/2019/07/16/the-pgp-problem.html
  - binwiederhier 7y ago
    I remember reading this a while ago. Thanks for reposting. I did in faxt stumble upon the linked RFC, but I quickly realized that it's quite complex for the simple thing I would have expected. As the article nicely points out, it's trying to be the Swiss army knife but that makes things difficult.
badrabbit 7y ago
Been down this road. GPG exists for that purpose. Although not intednded for disk files,s/mime has cross platform support as well. To answer your question,interoperability with other applications is neither expected nor desired for most apps that store encrypted data in a file. Quite often there are additional implementation details to be included (for which AEAD supportig cipher modes account for). Let's say I encrypt outputs of my program and store them in a file,I will most likely serialize it,encrypt it and store it only caring about my app being able to read and deserialize it. There is mostly no need for interoperability for data-at-rest encryption since there are a lot of interoperable protocols for data-in-transit encryption. I do think it would not hurt to have something like .gpg but more srandardized and easy to implement.
- binwiederhier 7y ago
  You bring up a good point. In most cases interoperability is not required. However, assuming it find enough adoption, having a standard format would likely make many things easier, like switching languages, or moving parts of the application logic into the database layer. Think of json support in databases these days and how it enables versatile use across applications. Another benefit would IMHO be the possibility for a simplified API for developers. Many engineers don't know what AEAD or block cipher modes even are. Providing a simple encrypt()/decrypt() function that would work across applications would be awesome.
binwiederhier 7y ago
This is obviously not a spec, but it's as close as it gets to the API that I would like. And it's got Daniel Bleichenbacher listed as author so it's got my vote. I couldn't find any on disk spec though at all though which is a little odd: https://github.com/google/tink https://github.com/google/tink Very cool project.

verdverm 7y ago

Implementation matters and the same cipher may be good in one language and broken in another. What happens when I cipher becomes exploitable? How would you re-encrypt with only a single en/decrypt functions? Which cipher was originally used?

binwiederhier 7y ago
Hm. I don't buy the "implementation matters" argument. If AES-128-GCM is broken in a library or language that'd be a pretty big deal. Cipher implementations should be interoperable, otherwise you'd have a big problem. Your other argument is totally valid. If a cipher becomes exploitable, you'd treat it the same way as you would with password hashes. You'd need to re-encrypt. password_hash() is designed to hash with today's best hash. My question was mainly about the file format though. The encrypt/decrypt functions would be a simple perk that becomes possible if we had that. Edit: encrypt->hash
- verdverm 7y ago
  By broken, I don't mean incorrect, just poorly implemented and there for exploitable. This is true of libraries in the wild. File format is typically binary or blob. It's encrypted so you can't see or understand it's structure, only assume.

jolmg 7y ago

> I've searched for RFCs but I couldn't find any standard, and not even a defacto standard. How about RFC 4880 - OpenPGP Message Format[1]? [1] https://tools.ietf.org/html/rfc4880 https://tools.ietf.org/html/rfc4880

insomniacity 7y ago
This page explains some of the problems with the format: https://latacora.micro.blog/2019/07/16/the-pgp-problem.html https://latacora.micro.blog/2019/07/16/the-pgp-problem.html
- binwiederhier 7y ago
  I remember reading this a while ago. Thanks for reposting. I did in faxt stumble upon the linked RFC, but I quickly realized that it's quite complex for the simple thing I would have expected. As the article nicely points out, it's trying to be the Swiss army knife but that makes things difficult.

badrabbit 7y ago

Been down this road. GPG exists for that purpose. Although not intednded for disk files,s/mime has cross platform support as well. To answer your question,interoperability with other applications is neither expected nor desired for most apps that store encrypted data in a file. Quite often there are additional implementation details to be included (for which AEAD supportig cipher modes account for). Let's say I encrypt outputs of my program and store them in a file,I will most likely serialize it,encrypt it and store it only caring about my app being able to read and deserialize it. There is mostly no need for interoperability for data-at-rest encryption since there are a lot of interoperable protocols for data-in-transit encryption. I do think it would not hurt to have something like .gpg but more srandardized and easy to implement.

binwiederhier 7y ago
You bring up a good point. In most cases interoperability is not required. However, assuming it find enough adoption, having a standard format would likely make many things easier, like switching languages, or moving parts of the application logic into the database layer. Think of json support in databases these days and how it enables versatile use across applications. Another benefit would IMHO be the possibility for a simplified API for developers. Many engineers don't know what AEAD or block cipher modes even are. Providing a simple encrypt()/decrypt() function that would work across applications would be awesome.

binwiederhier 7y ago

This is obviously not a spec, but it's as close as it gets to the API that I would like. And it's got Daniel Bleichenbacher listed as author so it's got my vote. I couldn't find any on disk spec though at all though which is a little odd: https://github.com/google/tink https://github.com/google/tink Very cool project.