7 ms·
MongoDB Releases Queryable Encryption Preview
- rafaelturk 4y agoThis looks really cool. Albeit feels that it is actually a feature implemented in the driver (client side) so my initial impression is that is not a meanignfull innovation on the server side. This can be implemented with any Database, even with current MongoDBs
- rafaelturk 4y agoWe use Mongoose, for sensitive data we have a wrapper around the .pre Save() method da encrypts it before sending data to the downstream db. Feels that MongoDB implemented that, in a more elegant structured code.
- gqewogpdqa 4y agoNope it’s implemented on the server side. I think that they are going to talk more about it at a session and maybe even in a keynote
- 8jy89hui 4y ago> This can be implemented with any Database, even with current MongoDBs Is it really all client side? How could they do things like substring matching without sending the entire index back and forth to the client? The graphic seems to show the query being executed solely on the server (although graphics often lie).
- jayd16 4y agoPerhaps encrypted trigrams (or some such thing) are sent during insert and search. Then it's just a matter of counting matching trigrams/chunks. The server doesn't need to know how to read the trigrams.
- claudiug 4y agocan this be done in postgres via client or via server? I found it really nice
- eknkc 4y agoI didn't know this was a thing. The article mentions it can do equality, range, prefix, suffix and substring queries. Does this mean that the encryption scheme creates sortable 1:1 mapped results after encryption? Kind of like a shift cipher?
- tyingq 4y agoThey mention this: "Queryable Encryption was designed by MongoDB’s Advanced Cryptography Research Group, headed by Seny Kamara and Tarik Moataz" Some related papers with those two as authors: https://eprint.iacr.org/2016/453.pdf https://eprint.iacr.org/2016/453.pdf https://cs.brown.edu/people/seny/pubs/sgx.pdf https://cs.brown.edu/people/seny/pubs/sgx.pdf
- uberdru 4y agoseriously did not think we would see homomorphic encryption productized for a few more years. pretty impressive!
- 8jy89hui 4y ago> Some of the existing tools, such as homomorphic encryption or secure enclaves have performance unsuited to scalable encrypted search, require proprietary hardware, or have uncertain security properties. I don't think this is exactly homomorphic. I hope they put out a whitepaper so researchers can properly evaluate its security.
- uberdru 4y agoNice catch, I was scanning for homomorphic encryption, but missed this. Have no idea how else they would implement this.
- samwillis 4y agoHomomorphic encryption allows you to modify the encrypted data without decrypting it or even knowing the the content. I don’t think this is homomorphic encryption. If they are able to do this without decrypting the data then I think you could describe this as a somewhat week encryption that exposes some data attributes as queryable. You could not implement this with strong encryption without at least decrypting for indexing.
- dandraper 4y agoIts not Homomorphic but "structural encryption". Less useful than HE but faster.
- SkyPuncher 4y agoThis is a really neat technology, but I don't understand it's use case. I've worked in HealthTech and currently in the compliance space. I'm skeptical of Mongo's claims (and their familiarity with compliance laws). Kind of feels like a solution in search of a problem. "In use" implies that you have a need to process that data. It doesn't matter if the end client is submitting queries in plain text (protected in transit) or this fancy encryption, the client (or server) still needs to be authorized to query that data. Translating from plain-text to encryption does not add additional protections from a compliance perspective.
- redwood 4y agoI see this as more about fundamental trust.. confidentiality from the service providers, not compliance
- xhkkffbf 4y agoIn one of the books about the general idea, _Translucent Databases_, the idea is to save the costs of securing the raw data. Someone might break into the database server (or listen on the wire) and find only encrypted values. This can make many different architectural use cases easier to deliver. In the most extreme cases, the unencrypted values never leave the client. The database can concentrate on delivering storage and fast query answers without paying much attention to issues of security. Clients don't need to trust the database because they control the encryption.
- giaour 4y ago> It doesn't matter if the end client is submitting queries in plain text (protected in transit) or this fancy encryption It's not just the query that is encrypted in this case, but the data being queried. From MongoDB's description, the server never receives or stores plaintext data, and the query results can only be decrypted by a client who has the same key that was used to encrypt the data in the first place. From a compliance perspective, that's amazing if it works. It means the server is never storing or processing anything but ciphertext.
- gkop 4y ago
- api 4y agoIs this actually possible? Couldn't you make many repeated queries and slowly decrypt the text by e.g. slowly narrowing the range?
- Diggsey 4y agoYeah the article is very thin on technical details. To make this work as they describe, it must not be possible for any client to "forge" queries, or else they could trivially decode the content by sending prefix queries of increasing length. It's also difficult to see how this could work on the server side without exposing some information about the encrypted fields. For example, if all documents have a value that begins with "a", then there must exist a prefix query that matches all those documents. I would expect it to be possible to figure out whether such a query is possible or not, only given access to the encrypted data, but even if that's not possible, the simple fact that a prefix query was issued that matched all documents gives away that information.
- robmccoll 4y agoYou could have a larger range than domain and throw in some noise. Exact match queries would need to become range queries that are de-noised at decryption.
- robmccoll 4y agoThis is possible. The goal is that the server knows as little as possible, while the client has full information. It's order revealing encryption. The server side knows the ordering of the values, but doesn't know any specific value. When queried, it is always getting prefixes (or exact matches) following the same encryption scheme, so it can compare those to the corpus and select results since the query parameters fall into the same ordering. The server doesn't have access to the keys needed to generate query parameters, so in theory it would be difficult for the server to perform narrowing queries on its own. Over time the server could gather statistical results that may reveal more about the data it's holding. Also, these schemes may need to produce the same cipher text for the same input, so frequency distributions can be used to reveal information.
- 4y ago
- dandraper 4y agoThis feature is a result of MongoDB's acquisition of Aroki. It looks like a good product but we actually beat them to it with https://cipherstash.com/activestash https://cipherstash.com/activestash CipherStash works with any Database and also supports Range queries and sorting/ordering. We do it in the application layer. Only supports Ruby so far but C#, Java, Python, Rust are in the works.
- metadat 4y agoWhat about Go, or even Tcl, and Ocaml? Do you have pointers to docs that'd help OSS efforts in this department?
- dandraper 4y agoNot yet but that's a good suggestion! The core client code is Rust so additional languages are (mostly) just native bindings to Rust. We will be releasing the Rust SDK publicly soon and welcome contributions!
- Redsquare 4y agoIf it is going to the likes of aws kms everytime it will blow budgets
- GTP 4y agoThe problem is: is also the full query encrypted or just some values that are considered sensitive? I remember a research form some years ago showing that if an attacker is still able to see the SQL code can recover the content of the database by looking at the queries, the responses and "putting the pieces together". Now, if the target was to get the exact values inside the database (think about employees wages) it still required to observe a very big number of queries, but if you were interested in getting a reasonable interval for each value then the number of queries needed become small enough to be doable in practice. Unfortunately I don't seem too be able to find this again, but a quick search turned out two papers that say that just encrypting your db isn't enough: [0], [1]. In particualr [1] doesn't seem to go into the details of how you could recover the data, but mentions that many operations as performed by "normal" databases leak information if performed over encrypted data. Maybe someone that is more familiar with Queryable Encryption can comment on this? [0] https://www.cs.cornell.edu/~shmat/shmat_hotos17.pdf https://www.cs.cornell.edu/~shmat/shmat_hotos17.pdf [1] https://www.microsoft.com/en-us/research/wp-content/uploads/2012/01/final-submission.pdf https://www.microsoft.com/en-us/research/wp-content/uploads/...
- ihucos 4y agoI like it. There is always a way to hack something. This is an additional layer of security that yes, can be also broken.
- mahmoudimus 4y agoYou're on the right track. I work in the data security space and while this is a cool release, it's not novel[0] and has been around for a while[1]. As a general rule of thumb, the first thing to check is if the provider is asking you to pass in your query in plain text AND without a local client (very important, because if you're sending data in plaintext, the threat model is now transitioning to a honest-but-curious model). This is obviously not that. They're encrypting locally. However, Simon Oya & Dr. Kerschbaum's paper, https://arxiv.org/abs/2010.03465 https://arxiv.org/abs/2010.03465, demonstrate a fantastic efficient attack to recover keywords on most constructions without a lot of queries. It is yet to be seen how effective MongoDB's implementation will be. This is a very interesting space but structural encryption is the right way to put the theory into good use. Most of the other encryption mechanisms such as homomorphic, partially homomorphic, etc. are just too impractical or require very specific niche use cases to be useful. There are other misnamed technology I've seen in marketing such as "polymorphic encryption" or "vaultless" - but most of these haven't had real research or cryptanalysis behind it. [0] https://info.ionic.com/hubfs/IonicDotCom/Resources/Assets/Securing%20the%20Cloud%20with%20Client-Side%20Encryption.pdf https://info.ionic.com/hubfs/IonicDotCom/Resources/Assets/Se... [1] https://eprint.iacr.org/2017/111.pdf https://eprint.iacr.org/2017/111.pdf
- throwaway2016a 4y agoHelp me understand this... It says it will support prefix search, substring search, and the like. Can anyone point me in the right direction on what the algorithm may be here? I don't get how you could do those things without making the encryption less secure and/or decrypting every record the fly. Another interesting use case I found that isn't mentioned here is sort. I've had customers ask me to be able to sort the results by PII and we tell them... no, we can't do that because the field is encrypted.
- hapiri 4y agoIt is less secure than your standard symmetric encryption. I guess they would use deterministic encryption in which 2 entries with same email address will have the same record string ( this leaks information to attacker ). Prefix search & sort can be achieved by using order preserving encryption. Not really sure about sub-string though.
- throwaway2016a 4y agoI've researched order preserving encryption before but the tradeoffs (mainly that the attacker can tell the order and use that to narrow the search space) always seemed like high risk.
- sangel 4y agoHigh risk compared to what? The alternative is absolutely no privacy (status quo) or no/limited functionality (not very useful). Seems like strictly better than having no privacy.
- bawolff 4y agoUsing fake encryption is much riskier than no encryption, because if you think you are safe you will do unsafe things with your data. If you know you are unsafe then you will take appropriate precautions.
- throwaway2016a 4y ago
- bincyber 4y agoThis is really neat. Recently I explored similar functionality for relational databases and only got as far as implementing column-level encryption [0] in this Go library [1], but without support for querying the encrypted data. HashiCorp Vault's transit secrets engine supports Convergent Encryption [2] which provides limited ability to query the encrypted data, but I haven't yet experimented with it. If anyone is doing something like this in production, would love to hear about your experience. [0]: https://en.wikipedia.org/wiki/Column_Level_Encryption https://en.wikipedia.org/wiki/Column_Level_Encryption [1]: https://github.com/bincyber/go-sqlcrypter https://github.com/bincyber/go-sqlcrypter [2]: https://www.vaultproject.io/docs/secrets/transit#convergent-encryption https://www.vaultproject.io/docs/secrets/transit#convergent-...
- muchpir 4y agoThe MuchPIR project (https://github.com/ReverseControl/MuchPIR https://github.com/ReverseControl/MuchPIR) implements Information-Theoretic Private Information Retrieval (IT-PIR) in Postgresql; In addition to the demo there is a high performance version available for commercial use.
- bawolff 4y agoI call bullshit. So let me get this right - its encrypted but you cansearch prefix and suffix? So all the attacker has to do is do it one letter at a time, see if it starts with A, B, C, once they figure that out, go to the next letter and so on. (I presume that the DB is not supposed to be trusted since they make such a big fuss about only being decryptable on the client side) Also there doesn't seem to be a whitepaper detailing algorithms or their threat model. Bitcoin scams try harder then this.
- mushi 4y agoIt’s already been mentioned that “Queryable Encryption was designed by MongoDB’s Advanced Cryptography Research Group, headed by Seny Kamara and Tarik Moataz" - are you calling bullshit on their work? What are your qualifications?
- bawolff 4y agoSo long as whatever system they designed has not been published and reviewed by independent experts, then yes. I don't have to be an expert in this space to recognize what the norms are for making new production ready cryptosystems are, and that this doesn't remotely meet them. Designing secure cryptosystems is hard. Experts fail at it all the time. The lack of technical details is a major red flag. Not to mention the distinct possibility that even if this group made a secure system, the mongodb marketing dept may very well be misrepresenting its security/limitations.
- winrid 4y agoThe use case you're outlining is someone already has access to the database. They can just do a find() in that case and get everything, no query required. You're basically describing an lz77 SSL hack that's like 20 years old, I'm pretty sure they would think of this. The use case here is just "advanced encryption at rest". Encrypting at rest is one thing, but this means people are less likely to see PII by accident, for example.
- bawolff 4y ago
- winrid 4y agoNeat. Did they fix their blog's pagination yet? If you hit next enough times you may or may not be able to take down the site, don't ask me how I know. (their pagination is implemented just by increasing the limit parameter).