5 ms·

Sandstorm, Tempest, and the Future

150 points by ocdtrekkie 3y ago

abdullahkhalids 3y ago
The call for donation, for this very very excellent project, are at the end of the post. > Unfortunately, as we hope to continue to support Sandstorm long-term, we’ve had an increasingly challenging problem with the legacy MongoDB database underneath, as Meteor has dropped support for the version we use. Ensuring a seamless migration on end-user servers has proven to be a challenging problem. Dependency upgrades are not fun and this has become a fairly unpleasant project to undertake for unpaid open source contributors. > Over the past few weeks, we have spoken to a number of potential outside parties about taking on this project and building a migration so that we can update the databases of existing Sandstorm servers and allow us to continue moving forward. > We are looking to raise the funding for this project through our OpenCollective. Please take a look at the long list of supported apps [1], which are basically one click install. [1] https://apps.sandstorm.io/ https://apps.sandstorm.io/
kentonv 3y ago
Just in case anyone is wondering, sandstorm.org represents the continuation of the Sandstorm community project, as opposed to sandstorm.io which was the startup that I co-founded. The company has been dissolved, and the community project runs under Open Source Collective. We felt it was important to make this change in continuity clear via a change in domain name (and release signing keys) as well. I plan to update the old sandstorm.io domain with a blog post and a top banner marking it as a historical site soon, and linking to sandstorm.org. For my part, I am no longer heavily involved in the community project -- mostly because I just don't have any free time. :/ But I trust Jake and co.
- hiAndrewQuinn 3y ago
  It's cool that you handed over the project to the open source community when you decided to fold the startup. I know absolutely nothing about Sandstorm and my interest has now skyrocketed in both it, and you.
  - kentonv 3y ago
    Thanks. FWIW Sandstorm was always an open source project, even as a startup, so technically my blessing wasn't actually required here. :)
_a_a_a_ 3y ago
From the website Sandstorm is a self-hostable web productivity suite.
em-bee 3y ago
these upgrade challenges are a very damning verdict for meteor and mongodb.
- ocdtrekkie 3y ago
  To be fair to Mongo, I've seen an enterprise application with an embedded Postgres backend choke on itself every single time it's upgraded the database. Of course, I'm a sysadmin and we had an enterprise support contract so that was not a big deal. MongoDB is perfectly fine at what it's designed for: Enterprise data infrastructure. But while needing a sysadmin or a support call may be no big deal for an enterprise platform, for an open source project without a customer support team or any product telemetry, the risk of shipping a less than perfect database upgrade is extreme. Tempest, or course, uses SQLite, which is far more suited to the role here with ten years of hindsight.
  - phyzome 3y ago
    Having seen MongoDB in use in an enterprise context, I may have to disagree with you about its fitness for purpose there. :-/
apitman 3y ago
Really sorry to hear about Ian. Sandstorm is an excellent project. I sort of consider it the platonic ideal of self-hosting software. It does everything the "right" way, especially with respect to security, but ultimately isn't compatible enough with the broader ecosystem for general adoption. I've been wondering if a lot of self-hosting platforms are targeting the wrong level of scale. On one extreme you have single user instances (SUI hosting). The other extreme is centralized platforms like Google. I used to think the ideal for most people is the "nerdy cousin" approach, where each family has someone capable of running something like Sandstorm or Nextcloud. The problem is that I would much rather trust Google with my private data than anyone close to me. There's so much more incentive for them to go looking through my stuff. This makes me think SUI might be the way to go. Or maybe large federated providers, similar to what we have for email.
- ocdtrekkie 3y ago
  The biggest thing we have to accomplish for single user hosting is what Sandstorm excels at: Removing security footguns. Self-hosting can't just be for folks who can sysadmin. Ultimately, it'd be nice to have Sandstorm available preloaded on a box someone can just plug into their home router and set up from a mobile app... but we're a long way from there yet.
  - apitman 3y ago
    But for that to work Sandstorm itself has to be as easy to install as an app on your phone, and there needs to be way more apps.
    - abdullahkhalids 3y ago
      What particular killer app do you think is missing?
      - palata 3y ago
        Can Sandstorm host a Matrix server? Or a Mastodon server?
mynegation 3y ago
I wish server applications reached the streamlined experience of desktop or phone applications where I can go to some App Store, install application with one click and then have a consistent experience, authentication handled by OS and (in case of mobile operating systems), strong data protection, notification and privacy controls etc. There are several attempts hoping to get there (Umbrel, CasaOS, Yunohost, Synology etc), but in my opinion, Sandstorm is (was?) the most consistent, daring, and well-engineered stab at it. What did not work for me was the per-document security model which I think is an audacious and advanced idea, but was a bit too much friction.
- apitman 3y ago
  Don't forget networking. Having to deal with DNS, NAT, port mapping, TLS, firewalls, etc etc is a complete non starter for basically everyone. IMO we need a nice simple OAuth2-based protocol for setting up something like Cloudflare tunnels.
- ocdtrekkie 3y ago
  The per document model is actually super easy to write apps for outright, just unfortunately fairly painful to package existing apps into. Ideally if everyone was writing apps specifically for Sandstorm it would be a non-issue. :) It's also extremely hard to get a consistent experience across apps without it: When every app has it's own concept of file management, it's impossible to build a unified experience. I think Sandstorm as a startup was, if anything, just too early. (I remember people questioning why anyone wouldn't trust Google back when it started!) I suspect in several years everything comparable that isn't Sandstorm will look a lot more like Sandstorm than Docker and such.
- the_common_man 3y ago
  Have you tried cloudron? It's pretty close to phone apps.
- ilaksh 3y ago
  Does the new version Tempest share that model?
dfc 3y ago
Does anyone remember a hw+SW company from the late 90s that had an application server that was sort of similar to this? I thought it was RackOS but I'm having zero luck. You could install/add different applications to the box and make them available on your local network.
- wmf 3y ago
  Cobalt Qube and RaQ. https://en.wikipedia.org/wiki/Cobalt_Qube https://en.wikipedia.org/wiki/Cobalt_Qube
  - dfc 3y ago
    Thank you! I tried every rack-like word I could think of.
ngmc 3y ago
I found this helpful in getting started with Sandstorm: https://youtu.be/y6120QOlsfU https://youtu.be/y6120QOlsfU
- ocdtrekkie 3y ago
  As a project that's been around for a decade, this joke has gotten very old. This old: https://github.com/sandstorm-io/sandstorm-website/pull/19 https://github.com/sandstorm-io/sandstorm-website/pull/19
  - ngmc 3y ago
    And here I was feeling cheeky. In all seriousness, I just learned about Sandstorm and think it could be a great fit for a local co-op I’m building. It’s a shame about the startup, but I hope we’ll get a chance to support the open collective.
chillpenguin 3y ago
I've been following this project for a few years now. Always thought it was a great idea.
haolez 3y ago
I think they should also target enterprise users as well as hobbyists. I've evaluated it for my company (I'm the CTO of a small Fintech of 100 people) and I was kinda lost on how to share data between a team and not just individuals. Felt like a big gap to me, even if I invested on contributing what I needed. With the support from a business case, things could get easier for the hobbyists. This is just constructive criticism. I sincerely think this is the coolest open source project around.
- ocdtrekkie 3y ago
  Targeting enterprise users was indeed the startup's focus! Unfortunately I think at the time a lot of businesses were perfectly comfortable giving all their data to Google, and some of the core apps weren't really good enough to justify the switch. I think a business building Sandstorm would do better today, but right now our nonprofit is focused around the self-hosting community first and foremost. I've done a lot of thinking about how to prep Sandstorm for different business verticals though and what that would look like. The Collections app is currently the best way to share "a group of permissions", which you can in-turn share with a bunch of users, but there's definitely a missing feature there in terms of being able to share to groups of users or especially share some grains by default to new users. For Tempest, Ian and I had discussed doing a lot more with "keyrings" of capabilities, and I absolutely would like to be able to do that someday.
traverseda 3y ago
Sandstorm has some nice features, especially in regards to single-sign-on, automatically setting up a reverse proxy, automatically handling SSL certificates, etc. But I really don't understand what the sandstorm runtime is bringing over just having a docker container. It's trying to reinvent docker, which seems weirdly out of scope for the problems it's trying to solve. Or even if docker isn't suitable you can still use OCI-compatible container images instead of a custom format?
- kentonv 3y ago
  Sandstorm and Docker are really totally different things. https://sandstorm.io/news/2014-08-19-why-not-run-docker-apps https://sandstorm.io/news/2014-08-19-why-not-run-docker-apps
  - traverseda 3y ago
    >We do often use Docker’s build tools in the process of building Sandstorm packages, simply because they provide a structured and reproducible way to gather an app and its dependencies into a chroot environment, which we can then package up. It sounds like "we do use OCI-compatible images, but we wrap them in our own filetype". Which I suppose is fine.
RecycledEle 3y ago
ELI5: Sandstorm is an open source project to make it easy to run open source web applications. Source: https://sandstorm.org/ https://sandstorm.org/

150 points by ocdtrekkie 3y ago

The call for donation, for this very very excellent project, are at the end of the post. > Unfortunately, as we hope to continue to support Sandstorm long-term, we’ve had an increasingly challenging problem with the legacy MongoDB database underneath, as Meteor has dropped support for the version we use. Ensuring a seamless migration on end-user servers has proven to be a challenging problem. Dependency upgrades are not fun and this has become a fairly unpleasant project to undertake for unpaid open source contributors. > Over the past few weeks, we have spoken to a number of potential outside parties about taking on this project and building a migration so that we can update the databases of existing Sandstorm servers and allow us to continue moving forward. > We are looking to raise the funding for this project through our OpenCollective. Please take a look at the long list of supported apps [1], which are basically one click install. [1] https://apps.sandstorm.io/ https://apps.sandstorm.io/

kentonv 3y ago

Just in case anyone is wondering, sandstorm.org represents the continuation of the Sandstorm community project, as opposed to sandstorm.io which was the startup that I co-founded. The company has been dissolved, and the community project runs under Open Source Collective. We felt it was important to make this change in continuity clear via a change in domain name (and release signing keys) as well. I plan to update the old sandstorm.io domain with a blog post and a top banner marking it as a historical site soon, and linking to sandstorm.org. For my part, I am no longer heavily involved in the community project -- mostly because I just don't have any free time. :/ But I trust Jake and co.

hiAndrewQuinn 3y ago
It's cool that you handed over the project to the open source community when you decided to fold the startup. I know absolutely nothing about Sandstorm and my interest has now skyrocketed in both it, and you.
- kentonv 3y ago
  Thanks. FWIW Sandstorm was always an open source project, even as a startup, so technically my blessing wasn't actually required here. :)

_a_a_a_ 3y ago

From the website Sandstorm is a self-hostable web productivity suite.

em-bee 3y ago

these upgrade challenges are a very damning verdict for meteor and mongodb.

ocdtrekkie 3y ago
To be fair to Mongo, I've seen an enterprise application with an embedded Postgres backend choke on itself every single time it's upgraded the database. Of course, I'm a sysadmin and we had an enterprise support contract so that was not a big deal. MongoDB is perfectly fine at what it's designed for: Enterprise data infrastructure. But while needing a sysadmin or a support call may be no big deal for an enterprise platform, for an open source project without a customer support team or any product telemetry, the risk of shipping a less than perfect database upgrade is extreme. Tempest, or course, uses SQLite, which is far more suited to the role here with ten years of hindsight.
- phyzome 3y ago
  Having seen MongoDB in use in an enterprise context, I may have to disagree with you about its fitness for purpose there. :-/

apitman 3y ago

Really sorry to hear about Ian. Sandstorm is an excellent project. I sort of consider it the platonic ideal of self-hosting software. It does everything the "right" way, especially with respect to security, but ultimately isn't compatible enough with the broader ecosystem for general adoption. I've been wondering if a lot of self-hosting platforms are targeting the wrong level of scale. On one extreme you have single user instances (SUI hosting). The other extreme is centralized platforms like Google. I used to think the ideal for most people is the "nerdy cousin" approach, where each family has someone capable of running something like Sandstorm or Nextcloud. The problem is that I would much rather trust Google with my private data than anyone close to me. There's so much more incentive for them to go looking through my stuff. This makes me think SUI might be the way to go. Or maybe large federated providers, similar to what we have for email.

ocdtrekkie 3y ago
The biggest thing we have to accomplish for single user hosting is what Sandstorm excels at: Removing security footguns. Self-hosting can't just be for folks who can sysadmin. Ultimately, it'd be nice to have Sandstorm available preloaded on a box someone can just plug into their home router and set up from a mobile app... but we're a long way from there yet.
- apitman 3y ago
  But for that to work Sandstorm itself has to be as easy to install as an app on your phone, and there needs to be way more apps.
  - abdullahkhalids 3y ago
    What particular killer app do you think is missing?
    - palata 3y ago
      Can Sandstorm host a Matrix server? Or a Mastodon server?

mynegation 3y ago

I wish server applications reached the streamlined experience of desktop or phone applications where I can go to some App Store, install application with one click and then have a consistent experience, authentication handled by OS and (in case of mobile operating systems), strong data protection, notification and privacy controls etc. There are several attempts hoping to get there (Umbrel, CasaOS, Yunohost, Synology etc), but in my opinion, Sandstorm is (was?) the most consistent, daring, and well-engineered stab at it. What did not work for me was the per-document security model which I think is an audacious and advanced idea, but was a bit too much friction.

apitman 3y ago
Don't forget networking. Having to deal with DNS, NAT, port mapping, TLS, firewalls, etc etc is a complete non starter for basically everyone. IMO we need a nice simple OAuth2-based protocol for setting up something like Cloudflare tunnels.
ocdtrekkie 3y ago
The per document model is actually super easy to write apps for outright, just unfortunately fairly painful to package existing apps into. Ideally if everyone was writing apps specifically for Sandstorm it would be a non-issue. :) It's also extremely hard to get a consistent experience across apps without it: When every app has it's own concept of file management, it's impossible to build a unified experience. I think Sandstorm as a startup was, if anything, just too early. (I remember people questioning why anyone wouldn't trust Google back when it started!) I suspect in several years everything comparable that isn't Sandstorm will look a lot more like Sandstorm than Docker and such.
the_common_man 3y ago
Have you tried cloudron? It's pretty close to phone apps.
ilaksh 3y ago
Does the new version Tempest share that model?

dfc 3y ago

Does anyone remember a hw+SW company from the late 90s that had an application server that was sort of similar to this? I thought it was RackOS but I'm having zero luck. You could install/add different applications to the box and make them available on your local network.

wmf 3y ago
Cobalt Qube and RaQ. https://en.wikipedia.org/wiki/Cobalt_Qube https://en.wikipedia.org/wiki/Cobalt_Qube
- dfc 3y ago
  Thank you! I tried every rack-like word I could think of.

ngmc 3y ago

I found this helpful in getting started with Sandstorm: https://youtu.be/y6120QOlsfU https://youtu.be/y6120QOlsfU

ocdtrekkie 3y ago
As a project that's been around for a decade, this joke has gotten very old. This old: https://github.com/sandstorm-io/sandstorm-website/pull/19 https://github.com/sandstorm-io/sandstorm-website/pull/19
- ngmc 3y ago
  And here I was feeling cheeky. In all seriousness, I just learned about Sandstorm and think it could be a great fit for a local co-op I’m building. It’s a shame about the startup, but I hope we’ll get a chance to support the open collective.

chillpenguin 3y ago

I've been following this project for a few years now. Always thought it was a great idea.

haolez 3y ago

I think they should also target enterprise users as well as hobbyists. I've evaluated it for my company (I'm the CTO of a small Fintech of 100 people) and I was kinda lost on how to share data between a team and not just individuals. Felt like a big gap to me, even if I invested on contributing what I needed. With the support from a business case, things could get easier for the hobbyists. This is just constructive criticism. I sincerely think this is the coolest open source project around.

ocdtrekkie 3y ago
Targeting enterprise users was indeed the startup's focus! Unfortunately I think at the time a lot of businesses were perfectly comfortable giving all their data to Google, and some of the core apps weren't really good enough to justify the switch. I think a business building Sandstorm would do better today, but right now our nonprofit is focused around the self-hosting community first and foremost. I've done a lot of thinking about how to prep Sandstorm for different business verticals though and what that would look like. The Collections app is currently the best way to share "a group of permissions", which you can in-turn share with a bunch of users, but there's definitely a missing feature there in terms of being able to share to groups of users or especially share some grains by default to new users. For Tempest, Ian and I had discussed doing a lot more with "keyrings" of capabilities, and I absolutely would like to be able to do that someday.

traverseda 3y ago

Sandstorm has some nice features, especially in regards to single-sign-on, automatically setting up a reverse proxy, automatically handling SSL certificates, etc. But I really don't understand what the sandstorm runtime is bringing over just having a docker container. It's trying to reinvent docker, which seems weirdly out of scope for the problems it's trying to solve. Or even if docker isn't suitable you can still use OCI-compatible container images instead of a custom format?

kentonv 3y ago
Sandstorm and Docker are really totally different things. https://sandstorm.io/news/2014-08-19-why-not-run-docker-apps https://sandstorm.io/news/2014-08-19-why-not-run-docker-apps
- traverseda 3y ago
  >We do often use Docker’s build tools in the process of building Sandstorm packages, simply because they provide a structured and reproducible way to gather an app and its dependencies into a chroot environment, which we can then package up. It sounds like "we do use OCI-compatible images, but we wrap them in our own filetype". Which I suppose is fine.

RecycledEle 3y ago

ELI5: Sandstorm is an open source project to make it easy to run open source web applications. Source: https://sandstorm.org/ https://sandstorm.org/