9 ms·

ZeroDB, an end-to-end encrypted database

102 points by mwilkison 12y ago

rubbingalcohol 12y ago
This is an amazing promise, but I was sad this is just a beta signup. I would really love to play around with something like this, and would also like to know how it works. Don't play with my heart, ZeroDB. Show us what you've got!
- michwill 12y ago
  We will release it open source once it's ready to be used, together with a whitepaper. Which, I hope, will happen pretty soon!
  - incomethax 12y ago
    This could be a really BIG deal for healthcare if you do it well. I would love to have a completely encrypted db that I could setup dynamic failover and replication. Also an amazing additional feature would be if it contained built-in access logging. If you were to do that, HIPAA becomes MUCH easier for other healthcare IT startups using your db.
    - michwill 12y ago
      Thanks a lot for the hints! :-)
  - jkot 12y ago
    Hi Michael, if you go opensource way, I would be happy to integrate this into my apache licensed project (see my desc). That would give you head start on Java and Java Collections.
    - michwill 12y ago
      Sure. Will keep in touch!
akerl_ 12y ago
This feels like a hollow announcement, given that there's no code or design details to look at. I am curious how they intend to let a client run queries against a dataset that the server cannot read without the server having to send all the encrypted data over the wire, or at least an index of all the encrypted data. Which sounds limiting for large datasets.
- onezeno 12y ago
  Perhaps by using homomorphic encryption? http://en.wikipedia.org/wiki/Homomorphic_encryption http://en.wikipedia.org/wiki/Homomorphic_encryption
  - taariqlewis 12y ago
    It was my understanding that homomorphic encryption was not yet ready for deployment. I didn't see that mentioned in the too brief blog post.
  - michwill 12y ago
    No. It is possible to do it without HE! Full HE is 10^12 slower than normal computations, so impractical yet. In our case, the actual computations are done on the client
    - davidw 12y ago
      So if I have, say, 10,000 patients, and I want to sort them by name, or search for those whose names begin with 'D', I need to fetch them all first?
      - michwill 12y ago
        To be precise, when you dataset is large, you fetch about log(index_size)
        tmd83 12y ago
        More than the size of the data read (which I'm not sure how big would actually be) I'm more concerned about the latency. Isn't a lot of normal db server side operation now a fetch-from-db + do-calculation-on-client type operation? And a compromise client can still pull data by running query against the server right? So is the primary improvement that one cannot use database tools to easily export the full dataset? If I know correctly isn't database with encryption support also do dumps and other full db operations encrypted too?
bcg1 12y ago
I realize that your post was titled "Hello World" so I wouldn't expect too much substance, but a couple quick questions (honest ones, not being sarcastic): What is the use case for something like this? Is this f/oss ... similarly, what are the licensing terms? Quick comment: Please don't misuse the word "hack" when you actually mean "security breach". Thanks!
- mwilkison 12y ago
  Yes, it will be open-source, we're still figuring out the correct license. Thanks for the "hack" -> "security breach" correction.
- michwill 12y ago
  The use-cases could be, for example, payment processors storing customer data, encrypted webmail (where you're able to search w/o downloading all your emails), "encrypted evernote" :-) Yes, it will be open source once available
  - bcg1 12y ago
    Oh, sounds good. So even though you can't get at the data on the server, you are able verify that it is unchanged with correct timestamps, etc. Sounds smart, useful. I presume then that you can support searching on the client side by downloading an encrypted index or something like that? Thanks for the good work guys, best of luck with your project and (ad)venture.
    - michwill 12y ago
      We support client-side search, that' the whole point. But we don't even have to download all the index, only log(index_size)
      - mcintyre1994 12y ago
        That's for full text search? This sounds really cool! Excuse the not-very-sophisticated question because I'm not really familiar with the data structures a database typically uses, how do your index sizes compare to a typical sql database or similar indexed similarly for the same sort of queries?
alimoeeny 12y ago
Some technical detail would be much appreciated, like the language you are using on the server side, any dependencies? Road map for when you are open sourcing it (I assume you will do)...
michaelmachine 12y ago
How does this compare to CryptDB http://css.csail.mit.edu/cryptdb/ http://css.csail.mit.edu/cryptdb/ ?
superobserver 12y ago
Interesting. I wonder how it will turn out to compare with ProtonMail's solution.
- michwill 12y ago
  Yes, interesting. I wonder how did they implement search over there
axx 12y ago
Sorry for asking, but if the private keys are stored client-side, how do handle users with multiple computers? Let the user handle it by hand? (i'm no encryption expert, just curious)
- hasenj 12y ago
  I think the "client" here is the application server, not the web browser.
  - axx 12y ago
    Then where's the point in this?
    - michwill 12y ago
      Your [pretty long] passphrase can be your key. Or you can take your key file with you. Having the key derived from your password is also possible, but I think not really that secure. SpiderOak and Mega deal with this problem as they have e2e encrypted file storage
    - hasenj 12y ago
      So that someone getting access to your database's data storage doesn't compromise your data.
bobofettfett 12y ago
1. Good, all DB data needs to be encrypted 2. That said, the largest security risk is applications (backends) that enable mass access to customer data and allow mass leaks of customer data.
BinaryIdiot 12y ago
Alright even though it's light on information you've certainly caught my attention. Is there a GitHub page setup yet that I can follow?
elchief 12y ago
Database encryption doesn't make a whole lot of sense to me. Proper row and column security, and using real database user authentication (not one single, pooled web server user) is real security. A db on its own box, in its own network zone, physically controller by the data owner. What's the threat here? SQL Injection? Encryption won't help. Use parameterized queries and least privilege. Evil admin? They can just monitor the web server instead of the db.
- michwill 12y ago
  Appications should be architected so that security-critical logic happens on the client. See how Mega and SpiderOak do that. This way, the attacker has to hack every computer of every client instead of one single server. Of course, this requires making sure that the code which the client executes is not malicious
  - zobzu 12y ago
    yeah there are no perfect trade offs: if you control the application server then you just instruct clients to decrypt/give away their data/encryption key still its more effort and more opportunistic (client needs to actually connect that day) so it is safer/reduces likelihood. But a lot of work and issues for the security gain compared to db encryption and proper design (even thus, you know, its not a great trade off either!) In the end I guess we need a truly client-initiated and controlled data handling. HTTP isnt exactly made for that. It tells the client how to handle and control the data, no choice is really made client-side.
  - undefined0 12y ago
    I can see this being useful for the example you gave, Mega. Currently, the list of files you upload are all downloaded to the client in order to do a search. If the search could happen on the server but the file list remains encrypted, that would make the website less costly for the client.
- floatboth 12y ago
  I think this project could be used to secure against evil admins. Especially if you make CouchDB-style apps (communicate directly with the DB from the browser or desktop/mobile app)
hasenj 12y ago
Kind of interesting but I would like to see an explanation of the idea and how it works. The demo video doesn't seem to show any sign of encryption.
peterboo 12y ago
hi there. this concept has already been developed at : http://spot-on.sf.net http://spot-on.sf.net and is also deployed in http://goldbug.sf.net http://goldbug.sf.net

102 points by mwilkison 12y ago

This is an amazing promise, but I was sad this is just a beta signup. I would really love to play around with something like this, and would also like to know how it works. Don't play with my heart, ZeroDB. Show us what you've got!

michwill 12y ago
We will release it open source once it's ready to be used, together with a whitepaper. Which, I hope, will happen pretty soon!
- incomethax 12y ago
  This could be a really BIG deal for healthcare if you do it well. I would love to have a completely encrypted db that I could setup dynamic failover and replication. Also an amazing additional feature would be if it contained built-in access logging. If you were to do that, HIPAA becomes MUCH easier for other healthcare IT startups using your db.
  - michwill 12y ago
    Thanks a lot for the hints! :-)
- jkot 12y ago
  Hi Michael, if you go opensource way, I would be happy to integrate this into my apache licensed project (see my desc). That would give you head start on Java and Java Collections.
  - michwill 12y ago
    Sure. Will keep in touch!

akerl_ 12y ago

This feels like a hollow announcement, given that there's no code or design details to look at. I am curious how they intend to let a client run queries against a dataset that the server cannot read without the server having to send all the encrypted data over the wire, or at least an index of all the encrypted data. Which sounds limiting for large datasets.

onezeno 12y ago
Perhaps by using homomorphic encryption? http://en.wikipedia.org/wiki/Homomorphic_encryption http://en.wikipedia.org/wiki/Homomorphic_encryption
- taariqlewis 12y ago
  It was my understanding that homomorphic encryption was not yet ready for deployment. I didn't see that mentioned in the too brief blog post.
- michwill 12y ago
  No. It is possible to do it without HE! Full HE is 10^12 slower than normal computations, so impractical yet. In our case, the actual computations are done on the client
  - davidw 12y ago
    So if I have, say, 10,000 patients, and I want to sort them by name, or search for those whose names begin with 'D', I need to fetch them all first?
    - michwill 12y ago
      To be precise, when you dataset is large, you fetch about log(index_size)
      - tmd83 12y ago
        More than the size of the data read (which I'm not sure how big would actually be) I'm more concerned about the latency. Isn't a lot of normal db server side operation now a fetch-from-db + do-calculation-on-client type operation? And a compromise client can still pull data by running query against the server right? So is the primary improvement that one cannot use database tools to easily export the full dataset? If I know correctly isn't database with encryption support also do dumps and other full db operations encrypted too?

bcg1 12y ago

I realize that your post was titled "Hello World" so I wouldn't expect too much substance, but a couple quick questions (honest ones, not being sarcastic): What is the use case for something like this? Is this f/oss ... similarly, what are the licensing terms? Quick comment: Please don't misuse the word "hack" when you actually mean "security breach". Thanks!

mwilkison 12y ago
Yes, it will be open-source, we're still figuring out the correct license. Thanks for the "hack" -> "security breach" correction.
michwill 12y ago
The use-cases could be, for example, payment processors storing customer data, encrypted webmail (where you're able to search w/o downloading all your emails), "encrypted evernote" :-) Yes, it will be open source once available
- bcg1 12y ago
  Oh, sounds good. So even though you can't get at the data on the server, you are able verify that it is unchanged with correct timestamps, etc. Sounds smart, useful. I presume then that you can support searching on the client side by downloading an encrypted index or something like that? Thanks for the good work guys, best of luck with your project and (ad)venture.
  - michwill 12y ago
    We support client-side search, that' the whole point. But we don't even have to download all the index, only log(index_size)
    - mcintyre1994 12y ago
      That's for full text search? This sounds really cool! Excuse the not-very-sophisticated question because I'm not really familiar with the data structures a database typically uses, how do your index sizes compare to a typical sql database or similar indexed similarly for the same sort of queries?

alimoeeny 12y ago

Some technical detail would be much appreciated, like the language you are using on the server side, any dependencies? Road map for when you are open sourcing it (I assume you will do)...

michaelmachine 12y ago

How does this compare to CryptDB http://css.csail.mit.edu/cryptdb/ http://css.csail.mit.edu/cryptdb/ ?

superobserver 12y ago

Interesting. I wonder how it will turn out to compare with ProtonMail's solution.

michwill 12y ago
Yes, interesting. I wonder how did they implement search over there

axx 12y ago

Sorry for asking, but if the private keys are stored client-side, how do handle users with multiple computers? Let the user handle it by hand? (i'm no encryption expert, just curious)

hasenj 12y ago
I think the "client" here is the application server, not the web browser.
- axx 12y ago
  Then where's the point in this?
  - michwill 12y ago
    Your [pretty long] passphrase can be your key. Or you can take your key file with you. Having the key derived from your password is also possible, but I think not really that secure. SpiderOak and Mega deal with this problem as they have e2e encrypted file storage
  - hasenj 12y ago
    So that someone getting access to your database's data storage doesn't compromise your data.

bobofettfett 12y ago

1. Good, all DB data needs to be encrypted 2. That said, the largest security risk is applications (backends) that enable mass access to customer data and allow mass leaks of customer data.

BinaryIdiot 12y ago

Alright even though it's light on information you've certainly caught my attention. Is there a GitHub page setup yet that I can follow?

elchief 12y ago

Database encryption doesn't make a whole lot of sense to me. Proper row and column security, and using real database user authentication (not one single, pooled web server user) is real security. A db on its own box, in its own network zone, physically controller by the data owner. What's the threat here? SQL Injection? Encryption won't help. Use parameterized queries and least privilege. Evil admin? They can just monitor the web server instead of the db.

michwill 12y ago
Appications should be architected so that security-critical logic happens on the client. See how Mega and SpiderOak do that. This way, the attacker has to hack every computer of every client instead of one single server. Of course, this requires making sure that the code which the client executes is not malicious
- zobzu 12y ago
  yeah there are no perfect trade offs: if you control the application server then you just instruct clients to decrypt/give away their data/encryption key still its more effort and more opportunistic (client needs to actually connect that day) so it is safer/reduces likelihood. But a lot of work and issues for the security gain compared to db encryption and proper design (even thus, you know, its not a great trade off either!) In the end I guess we need a truly client-initiated and controlled data handling. HTTP isnt exactly made for that. It tells the client how to handle and control the data, no choice is really made client-side.
- undefined0 12y ago
  I can see this being useful for the example you gave, Mega. Currently, the list of files you upload are all downloaded to the client in order to do a search. If the search could happen on the server but the file list remains encrypted, that would make the website less costly for the client.
floatboth 12y ago
I think this project could be used to secure against evil admins. Especially if you make CouchDB-style apps (communicate directly with the DB from the browser or desktop/mobile app)

hasenj 12y ago

Kind of interesting but I would like to see an explanation of the idea and how it works. The demo video doesn't seem to show any sign of encryption.

peterboo 12y ago

hi there. this concept has already been developed at : http://spot-on.sf.net http://spot-on.sf.net and is also deployed in http://goldbug.sf.net http://goldbug.sf.net