3 ms·

I get the grain-per-document thing. Sounds interesting for at least a subset of applications, and disagree with the grandparent that it's common practice. Firs

I get the grain-per-document thing. Sounds interesting for at least a subset of applications, and disagree with the grandparent that it's common practice. First question that comes to mind is how the data source is sharded like that. You say in another post that you don't mandate a tech stack. So let's say I want to store my data in MySQL. Are you saying each document literally has its own private database instance? I have many more questions, but they will depend on your answer to that question. :)

kentonv 10y ago
> Are you saying each document literally has its own private database instance? Yes. Each "grain" (our catch-all term; could mean "an Etherpad document" or "a Gitlab repo" or "a Wekan board", etc.) gets a slice of filesystem to do whatever it wants with. Of course, MySQL is totally overkill for this task. We generally encourage people to use sqlite instead. But there are a couple apps that do actually run MySQL in the container, mostly because switching to sqlite would have been too much work. There are also several apps that use Mongo, but it turns out Mongo 3's WiredTiger engine is reasonably efficient for small data sets, so it works out. The model definitely is friendlier to some kinds of apps than others. Productivity apps with clear data boundaries -- Etherpad (document editor), Wekan (Trello clone), Rocket.Chat (Slack clone), Gitlab, etc. -- work very well, and are our main focus. "Big data processing" apps are not well-suited to this model. We're happy to cede those, as there are tons of people doing great work in that domain.
- simplehuman 10y ago
  In the case of Rocket.Chat, is each channel a grain or is the entire chat a grain? It appears to me it should be a grain but I am not sure.
  - kentonv 10y ago
    Currently, the whole Rocket.Chat instance is a grain. However, it would be "more Sandstorm-y" if each channel were a separate grain. For that to work well, though, we need some UI features to emulate what Rocket.Chat already does internally: * The ability for grains to highlight themselves in the sidebar when there is activity or when you've been mentioned, so that you know to check them. This API actually just launched last week! So now the apps are being updated to use it. * Some way to create "private message" grains between two people. We haven't done this yet but we have some ideas. Soon, I hope to see Rocket.Chat add a "single-channel grain" mode where its own sidebar is removed and it relies on Sandstorm's sidebar. This would provide better security for channels which should only be accessible to a subset of people. Probably, the multi-channel mode will stick around too, for cases where you want the same people to have access to many channels.
Nullabillity 10y ago
> Are you saying each document literally has its own private database instance? Yup. On the other hand, those containers are garbage-collected when nobody is viewing them.
- kentonv 10y ago
  To be clear: The container shuts down, but the storage stays. Since the data set is small, apps start back up quickly on-demand. Moreover, the storage for app assets (code, binaries, images, etc.) is shared read-only among all instances, so the marginal storage usage of each grain is only the per-grain data, which is usually very small.

kentonv 10y ago

> Are you saying each document literally has its own private database instance? Yes. Each "grain" (our catch-all term; could mean "an Etherpad document" or "a Gitlab repo" or "a Wekan board", etc.) gets a slice of filesystem to do whatever it wants with. Of course, MySQL is totally overkill for this task. We generally encourage people to use sqlite instead. But there are a couple apps that do actually run MySQL in the container, mostly because switching to sqlite would have been too much work. There are also several apps that use Mongo, but it turns out Mongo 3's WiredTiger engine is reasonably efficient for small data sets, so it works out. The model definitely is friendlier to some kinds of apps than others. Productivity apps with clear data boundaries -- Etherpad (document editor), Wekan (Trello clone), Rocket.Chat (Slack clone), Gitlab, etc. -- work very well, and are our main focus. "Big data processing" apps are not well-suited to this model. We're happy to cede those, as there are tons of people doing great work in that domain.

simplehuman 10y ago
In the case of Rocket.Chat, is each channel a grain or is the entire chat a grain? It appears to me it should be a grain but I am not sure.
- kentonv 10y ago
  Currently, the whole Rocket.Chat instance is a grain. However, it would be "more Sandstorm-y" if each channel were a separate grain. For that to work well, though, we need some UI features to emulate what Rocket.Chat already does internally: * The ability for grains to highlight themselves in the sidebar when there is activity or when you've been mentioned, so that you know to check them. This API actually just launched last week! So now the apps are being updated to use it. * Some way to create "private message" grains between two people. We haven't done this yet but we have some ideas. Soon, I hope to see Rocket.Chat add a "single-channel grain" mode where its own sidebar is removed and it relies on Sandstorm's sidebar. This would provide better security for channels which should only be accessible to a subset of people. Probably, the multi-channel mode will stick around too, for cases where you want the same people to have access to many channels.

Nullabillity 10y ago

> Are you saying each document literally has its own private database instance? Yup. On the other hand, those containers are garbage-collected when nobody is viewing them.

kentonv 10y ago
To be clear: The container shuts down, but the storage stays. Since the data set is small, apps start back up quickly on-demand. Moreover, the storage for app assets (code, binaries, images, etc.) is shared read-only among all instances, so the marginal storage usage of each grain is only the per-grain data, which is usually very small.