3 ms·

I'm not deep into sandstorm or know how search is/will be implemented there but these thoughts might help understanding alternatives. It seems you see the sear

I'm not deep into sandstorm or know how search is/will be implemented there but these thoughts might help understanding alternatives. It seems you see the search indexer as a process which has accesses all data from the outside in order to index it. What if each grain itself exposes its inner index via an interface that can be accessed by the global search indexer? This way the search indexer only sees what it shall see but not data meant to be hidden (like passwords). Also by exposing it via a security managed interface not every process/grain gets access to every other grains data. This method surely complicates things a bit compared to file system access but the minimum implementation of the grain presenting the search the whole internal file tree. Is no worse than the traditional way.

kentonv 11y ago
While having each grain implement a search API would be an elegant approach, it probably isn't practical. With the fine-grained model, we expect users to have hundreds, maybe thousands of grains, and possibly access to (and therefore searchability over) thousands more. Just replicating the query would probably be too expensive. Also keep in mind that a grain normally isn't running unless it is actively being used, and so such a search query would need to start up thousands of grains all at once. On the other hand, having a single central search index is a pretty big security liability. A bug in search would potentially give you read access to the contents of all documents on the server. Moreover, fine-grained encryption breaks down: since everyone can use the search index, it would need to be encrypted using a key that everyone has access to. Instead, each user should have a separate search index covering their own private data. This search index itself should run as a grain, to get the same protections / encryption as other grains. The search index grain should not be allowed to hold capabilities to any other grain. Communications between the search index and the grains being indexed should be one-way. The only two-way communication is the search query API, which should usually be accessible only to the user. As an (important) optimization, grains that are public or accessible to a large number of users should be indexed in shared indexes, not per-user.
- static_noise 11y ago
  We were thinking about two/three different things what the grain does: 1) My idea was that the grain exposes selected data to a search indexer. 2) Your idea was that the grain performs the search itself. I see why 2) would not work very well in the current implementation. But 1) should work well and make it possible to only expose selected data. (Also with central/user specific search.) Both ideas could by combined via per-grain-search-grains which only hold the local index and are accessed via a central search interface. The problem, however, remains, that searchable data is exposed data. To me it seems useful that some applications can have fine-grained control about what to expose. (Think of List of Establishments where each has a List of Customers with Credit Card information. You might make one or two things searchable but not everything.)

kentonv 11y ago

While having each grain implement a search API would be an elegant approach, it probably isn't practical. With the fine-grained model, we expect users to have hundreds, maybe thousands of grains, and possibly access to (and therefore searchability over) thousands more. Just replicating the query would probably be too expensive. Also keep in mind that a grain normally isn't running unless it is actively being used, and so such a search query would need to start up thousands of grains all at once. On the other hand, having a single central search index is a pretty big security liability. A bug in search would potentially give you read access to the contents of all documents on the server. Moreover, fine-grained encryption breaks down: since everyone can use the search index, it would need to be encrypted using a key that everyone has access to. Instead, each user should have a separate search index covering their own private data. This search index itself should run as a grain, to get the same protections / encryption as other grains. The search index grain should not be allowed to hold capabilities to any other grain. Communications between the search index and the grains being indexed should be one-way. The only two-way communication is the search query API, which should usually be accessible only to the user. As an (important) optimization, grains that are public or accessible to a large number of users should be indexed in shared indexes, not per-user.

static_noise 11y ago
We were thinking about two/three different things what the grain does: 1) My idea was that the grain exposes selected data to a search indexer. 2) Your idea was that the grain performs the search itself. I see why 2) would not work very well in the current implementation. But 1) should work well and make it possible to only expose selected data. (Also with central/user specific search.) Both ideas could by combined via per-grain-search-grains which only hold the local index and are accessed via a central search interface. The problem, however, remains, that searchable data is exposed data. To me it seems useful that some applications can have fine-grained control about what to expose. (Think of List of Establishments where each has a List of Customers with Credit Card information. You might make one or two things searchable but not everything.)