8 ms·

LavaBit's Architecture (2009)

140 points by yankcrime 13y ago

chayesfss 13y ago
That might be the best article/review I've ever read.
- tehwalrus 13y ago
  It is taken directly from the Lavabit "about us" pages* - I read them all a few weeks ago when I signed up and paid for pro! :) * - the pages are gone now, which is why they needed to be mirrored somewhere.
  - toddh 13y ago
    Ladar was kind enough to answer the questions in an architecture template I have for High Scalability (http://highscalability.com/architecture-template-advice-needed http://highscalability.com/architecture-template-advice-need...). I guess they also used it for their about page, which makes sense given all the work that went into it. He's a very nice guy, a shame to see how things went down.
    - tehwalrus 13y ago
      I'm still hopeful they'll all emigrate to Iceland (or somewhere) and start up again. Slightly hopeful, anyway.
captn3m0 13y ago
Just after the PRISM scandal exploded, I came up with the idea of encrypted-storage based email [1]. Unfortunately, within a little while of thinking of this, I read about lavabit and realized that they had the exact same security model that I was thinking of. Now that lavabit's gone, I think there is a good opportunity for someone in a sane country with good data-protection laws. [1]: http://firespotting.com/item?id=1632 http://firespotting.com/item?id=1632
- option_greek 13y ago
  It's very interesting. Something like this is really required. How ever, if this is a private project then people may not have absolute trust on it. If it is open source, independent verification is possible which will improve user trust. Also when malicious unnamed agents come after you through legal channels, you can drop the service like lavabit but some one else can setup an independent server and do a direct import of user data from your servers with user permission.
  - captn3m0 13y ago
    My plan was to keep it open source, and have free accounts, as far as possible. The monetization could have come from extra storage, like GMail, or custom domains.
    - Xurinos 13y ago
      Every time I see the phrase "open source", I mentally add "but not open binary"; how do I know that the service that is run uses binaries that are compiled from that open source alone? with trusted libraries? After all, anyone can employ a site-specific patch in their build process that adds additional "features" to their normally open source project. Is there a reasonable way to have trusted individuals intermittently audit the service?
      - muyuu 13y ago
        A no-trust-required system would consist of taking these sources and hosting your mail yourself. Which pretty much defeats the purpose, since what you want to avoid is hosting your own email (another alternative). Honestly I lend more towards phasing out email completely. Maybe providing gateways to other protocols.
- vader1 13y ago
  For a couple of weeks now, I've been brainstorming about how to create a system like this. I'd very much like to use this sort of setup for my own emails, but it simply doesn't seem to exist, especially not as open source (while lavabit claimed to store your emails encrypted, there was no way for anyone to actually vet their approach). The most viable option I came up with so far is modding eCryptfs to allow "write only" mounts by supplying it directly with the derived public key for the passphrase instead of the passphrase itself. This would allow you to use vanilla components such as postfix & dovecot instead of having to write a custom application server. Using the maildir format, incoming mails could be written to the write-only mount without having to read any files from the mount. On IMAP login, the mail daemon would use a passphrase derived from the login password to mount the maildir as read-write, using unshare to ensure that the mount is visible to that process alone, and automatically unmounting it when the IMAP user logs out. Thoughts and ideas are welcome :)
walden42 13y ago
I believe that the LavaBit fiasco just proves the need for a decentralized system with no central points of failure. Although it's still in the beginning stages, bitmessage (https://bitmessage.org/wiki/Main_Page https://bitmessage.org/wiki/Main_Page) offers a solution similar to the bitcoin protocol. I would more people to start using solutions such as these instead of email. Although email has served its purposed for a long time, it was never designed around security and privacy in the first place.
- zokier 13y ago
  > I believe that the LavaBit fiasco just proves the need for a decentralized system with no central points of failure That's probably the only thing e-mail actually does really well. Centralization is definitely not the problem with email. Lack of end-to-end encryption and meta-data leakage are the real problems.
  - VikingCoder 13y ago
    In theory, sure - but in practice... Most of my friends are @yahoo.com, @gmail.com, or their ISP. Meaning, in practice they're centralized.
    - zokier 13y ago
      From a technical point of view, they are not centralized. Gmail is a massively decentralized system in itself, a feature partially enabled by emails flexibility. Even if gmail would go down, (other) emails would still continue flowing. And when gmail would come back online, most messages would find their destination due the store-and-forward nature of email. The fact that even you mention several providers shows that federation does work. Currently the major pain point in the architecture of email arises when people bind their online identities to singular email providers, even when email has nice facilities of not doing so by having a domain independent from the mail provider and setting appropriate MX records.
      - VikingCoder 13y ago
Tloewald 13y ago
I don't understand why the service provider needs to store or provide any keys. Let the customer generate a private and public key, store the private key anyway he/she chooses, and then broadcast an address/public-key combo to would-be contacts. You can communicate using a totally insecure file-sharing service -- e.g. usenet, anonymous FTP, or mailinator.
- tptacek 13y ago
  Unless you're shipping the customer a persistent component that doesn't automatically update itself every time they connect to the service, you might as well just store the keys, because (as LavaBit apparently discovered) if a court decides you need to cough up your users info, they'll probably just get you to install whatever code is needed to make that happen.
  - conductor 13y ago
    If I understand correctly, the decryption code was in the server, so the user was providing the password for the private key, and the code from the server was doing the encryption/decryption (by server-side or client-side execution, I don't know). This scheme could be backdoored by altering that server-side code. So, why not keep the code on the client-side (like signed, verified browser add-on or native application)? Then there will be no way to get the password of the private key from the user (except by installing a malware with some 0-day browser vuln).
    - tptacek 13y ago
      Note that Chrome extensions can suffer from essentially the same problem, since they can autoupdate. The reason nobody does this, though, is that end-users don't want to install software on their machines. Unfortunately, there is a fundamental conflict between desire to run software off of other people's machines rather than your own and desire for cryptographic security.
    - bradleyjg 13y ago
      The client would have to be open source in order for that to work. Otherwise they can just force the provider to put a backdoor in the client code. Probably open source with no downloadable binaries would be best. Maybe even better would be something like only pulling off a source control system so there's an easily accessible audit trail. The idea being not necessarily to prevent an evil adversary, but to convince a judge that there's no way to convincingly be an evil adversary.
santry 13y ago
> * How do you backup and restore your system? > All critical servers use RAID 5. Unfortunately RAID 5 (or any other flavor) is not a backup solution. It doesn't account for user error, RAID controller failure and numerous other scenarios. It seems LavaBit had an interesting product, but this seems like a pretty critical design failure.
- pktgen 13y ago
  It's much harder to delete things from backups though. Lavabit promised that when you deleted a message, it was actually deleted. Redundancy against hardware failure but not against user error, software error, etc. was probably part of their security-convenience tradeoff.
- jonknee 13y ago
  Lavabit also flipped the bit and destroyed everyone's emails (without much complaint, at least that I have read). The goal was security, not availability. If security is the goal it's better to fail hard than fail [possibly] insecurely. POP/IMAP email also backs up itself--you retain a copy on your own machine.
yarapavan 13y ago
Old article, circa March 2009! Look at the monthly visits mentioned, and also this highscalability article- http://highscalability.com/blog/2009/3/30/lavabit-architecture-creating-a-scalable-email-service.html http://highscalability.com/blog/2009/3/30/lavabit-architectu... I wonder how much of this architecture still exits and how much has changed! Do they still host servers, use CentOS+Apache+MySQL+Memcache or was there an architectural change with more apt software components?
negativity 13y ago
It would be really awesome if the LavaBit team released all their trade secrets, including their C code for their custom daemons, and other substantial details, as an open source project. ...that is, if there's truly no hope, and they've shuttered the business for good, and never plan on re-opening their doors and returning to business. Hardware and Service Provider expenses aside, this would represent an opportunity for many people to create their own systems and participate in a distributed, federated network of professionally vetted, (ostensibly) secure e-mail services, based on field tested architectural standards. If the company is dead, why not turn lavabit.com into the homepage of a project spawned from the ashes of the original business.
caycep 13y ago
actually, I'm wondering if something like this could fit the bill for a HIPAA compliant medical communication system?
wenming 13y ago
ok?
wildgift 13y ago
I'm guessing that the government wants Lavabit's private keys. Maybe they already sniffed the traffic and now need keys to decrypt it. What do you all think?

140 points by yankcrime 13y ago

chayesfss 13y ago

That might be the best article/review I've ever read.

tehwalrus 13y ago
It is taken directly from the Lavabit "about us" pages* - I read them all a few weeks ago when I signed up and paid for pro! :) * - the pages are gone now, which is why they needed to be mirrored somewhere.
- toddh 13y ago
  Ladar was kind enough to answer the questions in an architecture template I have for High Scalability (http://highscalability.com/architecture-template-advice-needed http://highscalability.com/architecture-template-advice-need...). I guess they also used it for their about page, which makes sense given all the work that went into it. He's a very nice guy, a shame to see how things went down.
  - tehwalrus 13y ago
    I'm still hopeful they'll all emigrate to Iceland (or somewhere) and start up again. Slightly hopeful, anyway.

captn3m0 13y ago

Just after the PRISM scandal exploded, I came up with the idea of encrypted-storage based email [1]. Unfortunately, within a little while of thinking of this, I read about lavabit and realized that they had the exact same security model that I was thinking of. Now that lavabit's gone, I think there is a good opportunity for someone in a sane country with good data-protection laws. [1]: http://firespotting.com/item?id=1632 http://firespotting.com/item?id=1632

option_greek 13y ago
It's very interesting. Something like this is really required. How ever, if this is a private project then people may not have absolute trust on it. If it is open source, independent verification is possible which will improve user trust. Also when malicious unnamed agents come after you through legal channels, you can drop the service like lavabit but some one else can setup an independent server and do a direct import of user data from your servers with user permission.
- captn3m0 13y ago
  My plan was to keep it open source, and have free accounts, as far as possible. The monetization could have come from extra storage, like GMail, or custom domains.
  - Xurinos 13y ago
    Every time I see the phrase "open source", I mentally add "but not open binary"; how do I know that the service that is run uses binaries that are compiled from that open source alone? with trusted libraries? After all, anyone can employ a site-specific patch in their build process that adds additional "features" to their normally open source project. Is there a reasonable way to have trusted individuals intermittently audit the service?
    - muyuu 13y ago
      A no-trust-required system would consist of taking these sources and hosting your mail yourself. Which pretty much defeats the purpose, since what you want to avoid is hosting your own email (another alternative). Honestly I lend more towards phasing out email completely. Maybe providing gateways to other protocols.
vader1 13y ago
For a couple of weeks now, I've been brainstorming about how to create a system like this. I'd very much like to use this sort of setup for my own emails, but it simply doesn't seem to exist, especially not as open source (while lavabit claimed to store your emails encrypted, there was no way for anyone to actually vet their approach). The most viable option I came up with so far is modding eCryptfs to allow "write only" mounts by supplying it directly with the derived public key for the passphrase instead of the passphrase itself. This would allow you to use vanilla components such as postfix & dovecot instead of having to write a custom application server. Using the maildir format, incoming mails could be written to the write-only mount without having to read any files from the mount. On IMAP login, the mail daemon would use a passphrase derived from the login password to mount the maildir as read-write, using unshare to ensure that the mount is visible to that process alone, and automatically unmounting it when the IMAP user logs out. Thoughts and ideas are welcome :)

walden42 13y ago

I believe that the LavaBit fiasco just proves the need for a decentralized system with no central points of failure. Although it's still in the beginning stages, bitmessage (https://bitmessage.org/wiki/Main_Page https://bitmessage.org/wiki/Main_Page) offers a solution similar to the bitcoin protocol. I would more people to start using solutions such as these instead of email. Although email has served its purposed for a long time, it was never designed around security and privacy in the first place.

zokier 13y ago
> I believe that the LavaBit fiasco just proves the need for a decentralized system with no central points of failure That's probably the only thing e-mail actually does really well. Centralization is definitely not the problem with email. Lack of end-to-end encryption and meta-data leakage are the real problems.
- VikingCoder 13y ago
  In theory, sure - but in practice... Most of my friends are @yahoo.com, @gmail.com, or their ISP. Meaning, in practice they're centralized.
  - zokier 13y ago
    From a technical point of view, they are not centralized. Gmail is a massively decentralized system in itself, a feature partially enabled by emails flexibility. Even if gmail would go down, (other) emails would still continue flowing. And when gmail would come back online, most messages would find their destination due the store-and-forward nature of email. The fact that even you mention several providers shows that federation does work. Currently the major pain point in the architecture of email arises when people bind their online identities to singular email providers, even when email has nice facilities of not doing so by having a domain independent from the mail provider and setting appropriate MX records.
    - VikingCoder 13y ago

Tloewald 13y ago

I don't understand why the service provider needs to store or provide any keys. Let the customer generate a private and public key, store the private key anyway he/she chooses, and then broadcast an address/public-key combo to would-be contacts. You can communicate using a totally insecure file-sharing service -- e.g. usenet, anonymous FTP, or mailinator.

tptacek 13y ago
Unless you're shipping the customer a persistent component that doesn't automatically update itself every time they connect to the service, you might as well just store the keys, because (as LavaBit apparently discovered) if a court decides you need to cough up your users info, they'll probably just get you to install whatever code is needed to make that happen.
- conductor 13y ago
  If I understand correctly, the decryption code was in the server, so the user was providing the password for the private key, and the code from the server was doing the encryption/decryption (by server-side or client-side execution, I don't know). This scheme could be backdoored by altering that server-side code. So, why not keep the code on the client-side (like signed, verified browser add-on or native application)? Then there will be no way to get the password of the private key from the user (except by installing a malware with some 0-day browser vuln).
  - tptacek 13y ago
    Note that Chrome extensions can suffer from essentially the same problem, since they can autoupdate. The reason nobody does this, though, is that end-users don't want to install software on their machines. Unfortunately, there is a fundamental conflict between desire to run software off of other people's machines rather than your own and desire for cryptographic security.
  - bradleyjg 13y ago
    The client would have to be open source in order for that to work. Otherwise they can just force the provider to put a backdoor in the client code. Probably open source with no downloadable binaries would be best. Maybe even better would be something like only pulling off a source control system so there's an easily accessible audit trail. The idea being not necessarily to prevent an evil adversary, but to convince a judge that there's no way to convincingly be an evil adversary.

santry 13y ago

> * How do you backup and restore your system? > All critical servers use RAID 5. Unfortunately RAID 5 (or any other flavor) is not a backup solution. It doesn't account for user error, RAID controller failure and numerous other scenarios. It seems LavaBit had an interesting product, but this seems like a pretty critical design failure.

pktgen 13y ago
It's much harder to delete things from backups though. Lavabit promised that when you deleted a message, it was actually deleted. Redundancy against hardware failure but not against user error, software error, etc. was probably part of their security-convenience tradeoff.
jonknee 13y ago
Lavabit also flipped the bit and destroyed everyone's emails (without much complaint, at least that I have read). The goal was security, not availability. If security is the goal it's better to fail hard than fail [possibly] insecurely. POP/IMAP email also backs up itself--you retain a copy on your own machine.

yarapavan 13y ago

Old article, circa March 2009! Look at the monthly visits mentioned, and also this highscalability article- http://highscalability.com/blog/2009/3/30/lavabit-architecture-creating-a-scalable-email-service.html http://highscalability.com/blog/2009/3/30/lavabit-architectu... I wonder how much of this architecture still exits and how much has changed! Do they still host servers, use CentOS+Apache+MySQL+Memcache or was there an architectural change with more apt software components?

negativity 13y ago

It would be really awesome if the LavaBit team released all their trade secrets, including their C code for their custom daemons, and other substantial details, as an open source project. ...that is, if there's truly no hope, and they've shuttered the business for good, and never plan on re-opening their doors and returning to business. Hardware and Service Provider expenses aside, this would represent an opportunity for many people to create their own systems and participate in a distributed, federated network of professionally vetted, (ostensibly) secure e-mail services, based on field tested architectural standards. If the company is dead, why not turn lavabit.com into the homepage of a project spawned from the ashes of the original business.

caycep 13y ago

actually, I'm wondering if something like this could fit the bill for a HIPAA compliant medical communication system?

wenming 13y ago

ok?

wildgift 13y ago

I'm guessing that the government wants Lavabit's private keys. Maybe they already sniffed the traffic and now need keys to decrypt it. What do you all think?