4 ms·

One thing I appreciated about the Sandstorm architecture, and I wonder if you now consider this a mistake, is that each grain holds its own state, using ordinar

One thing I appreciated about the Sandstorm architecture, and I wonder if you now consider this a mistake, is that each grain holds its own state, using ordinary files and possibly in memory (since IIUC, there's one instance of a grain running at any given time). But I would have liked to use an architecture like this in a SaaS application. We may wish that applications were self-hosted, but SaaS is what people are used to, and there's a lot of money in it. An architecture for SaaS applications based on many small, stateful units, like Sandstorm grains, would be a refreshing alternative to the current norm of trying to make everything stateless and web-scale. And I think it's something you could have sold. I remember proposing something like this to you on an HN comment thread a few years ago, but maybe I didn't articulate it well. The reason I wonder if you now see the statefulness of grains as a mistake is that Cloudflare Workers are going even further in the other direction. Whereas Sandstorm grains can be long-running, Cloudflare Workers are very short-lived. Of course, I realize that you aren't explicitly developing Cloudflare Workers as the successor to Sandstorm; it's a new product with its own requirements. Still, I wonder if your thinking on that part of Sandstorm has changed.

kentonv 7y ago
I still very much believe in the fine-grained instances model. Cloudflare Workers is a long, long way from done. Stay tuned... ;)
solipsism 7y ago
An architecture for SaaS applications based on many small, stateful units, like Sandstorm grains, would be a refreshing alternative to the current norm itof trying to make everything stateless and web-scale Can you expand on this? The way I see things, the more of my stack that's stateless the better. That goes equally at the application level and at the distributed system level. There are so many advantages to designing things that way that it's hard to imagine going back to the stateful "bad old days". So I'm curious to read a treatise about this alternative small, stateful units model you describe.
- mwcampbell 7y ago
  For starters, I'll refer you to Sandstorm's own "how it works" page [1], which explains it quite well IMO. Basically, the Sandstorm modl works well for applications that consist of many small things -- documents, chat rooms, project boards, etc. -- each of which can easily run in a single process. Each grain can run in a self-contained process, with its live state being stored in memory, and using the filesystem for durability. When things happen within the grain, it can easily send out notifications to all connected clients, e.g. through WebSockets, because all clients are communicating with that one process (indirectly through a proxy, of course). Consider how much easier it is to implement real-time things, like chat and collaborative document editing, this way, compared to using message brokers, distributed pubsub systems, etc. As for scaling and high availability, as the page I linked explains, the platform takes care of that. [1]: https://sandstorm.io/how-it-works https://sandstorm.io/how-it-works
- kentonv 7y ago
  Statelessness is desirable in large-scale cloud architecture because it means you can auto-scale by just running more instances of the thing and load balancing, and you can fix any problem by just nuking the whole instance and making a new one. Also, storing data reliably in a large distributed system is an extraordinarily difficult problem which most people have no hope of getting right, so it's best to rely on an off-the-shelf database written by people who know what they're doing. But these are problems that only exist because you're trying to create a mega-scale centralized application. In Sandstorm's granular model, you don't have mega-scale applications. You have many distributed small instances. The programming model ends up being much more like desktop or mobile apps. In those environments, state has never been that big a deal. You don't need a database that can store and index petabytes of data; you can use something simple like sqlite or maybe even flat files. You don't need to think about machine failure, backups, etc.; that is the OS and/or device owner's problem. (Meanwhile, small-scale instances lead to clear data ownership which means that the owner can take responsibility for backups, and a distributed OS can potentially migrate grains around as needed to work around machine failures, transparently to the app.) Another big problem with the stateless approach to big web apps is that off-the-shelf databases are often a terrible fit for specific use cases, but typically changing the database to suit your needs is not an option. Databases tend to be particularly bad at real-time updates, e.g. like in Google Docs where you can see other people typing. To implement Google Docs reasonably, you need a stateful server; you need for all users of the same document to land on the same server instance so that coordination and routing can occur in-memory. That's incredibly difficult to achieve in the stateless cloud architecture orthodoxy, but incredibly easy to achieve in Sandstorm's model.

kentonv 7y ago

I still very much believe in the fine-grained instances model. Cloudflare Workers is a long, long way from done. Stay tuned... ;)

solipsism 7y ago

An architecture for SaaS applications based on many small, stateful units, like Sandstorm grains, would be a refreshing alternative to the current norm itof trying to make everything stateless and web-scale Can you expand on this? The way I see things, the more of my stack that's stateless the better. That goes equally at the application level and at the distributed system level. There are so many advantages to designing things that way that it's hard to imagine going back to the stateful "bad old days". So I'm curious to read a treatise about this alternative small, stateful units model you describe.

mwcampbell 7y ago
For starters, I'll refer you to Sandstorm's own "how it works" page [1], which explains it quite well IMO. Basically, the Sandstorm modl works well for applications that consist of many small things -- documents, chat rooms, project boards, etc. -- each of which can easily run in a single process. Each grain can run in a self-contained process, with its live state being stored in memory, and using the filesystem for durability. When things happen within the grain, it can easily send out notifications to all connected clients, e.g. through WebSockets, because all clients are communicating with that one process (indirectly through a proxy, of course). Consider how much easier it is to implement real-time things, like chat and collaborative document editing, this way, compared to using message brokers, distributed pubsub systems, etc. As for scaling and high availability, as the page I linked explains, the platform takes care of that. [1]: https://sandstorm.io/how-it-works https://sandstorm.io/how-it-works
kentonv 7y ago
Statelessness is desirable in large-scale cloud architecture because it means you can auto-scale by just running more instances of the thing and load balancing, and you can fix any problem by just nuking the whole instance and making a new one. Also, storing data reliably in a large distributed system is an extraordinarily difficult problem which most people have no hope of getting right, so it's best to rely on an off-the-shelf database written by people who know what they're doing. But these are problems that only exist because you're trying to create a mega-scale centralized application. In Sandstorm's granular model, you don't have mega-scale applications. You have many distributed small instances. The programming model ends up being much more like desktop or mobile apps. In those environments, state has never been that big a deal. You don't need a database that can store and index petabytes of data; you can use something simple like sqlite or maybe even flat files. You don't need to think about machine failure, backups, etc.; that is the OS and/or device owner's problem. (Meanwhile, small-scale instances lead to clear data ownership which means that the owner can take responsibility for backups, and a distributed OS can potentially migrate grains around as needed to work around machine failures, transparently to the app.) Another big problem with the stateless approach to big web apps is that off-the-shelf databases are often a terrible fit for specific use cases, but typically changing the database to suit your needs is not an option. Databases tend to be particularly bad at real-time updates, e.g. like in Google Docs where you can see other people typing. To implement Google Docs reasonably, you need a stateful server; you need for all users of the same document to land on the same server instance so that coordination and routing can occur in-memory. That's incredibly difficult to achieve in the stateless cloud architecture orthodoxy, but incredibly easy to achieve in Sandstorm's model.