7 ms·

Show HN: Sandstorm Personal Cloud Platform Demo

88 points by kentonv 12y ago

wmf 12y ago
Wired article explains the background: http://www.wired.com/2014/07/out-in-the-open-sandstorm-makes-it-easy-to-control-your-apps-in-the-cloud/ http://www.wired.com/2014/07/out-in-the-open-sandstorm-makes...
nobodysfool 12y ago
That's pretty amazing how quickly I can install and use an app. It's instant as far as I can tell. This makes me wonder - how difficult would it be to take a sandstorm app and migrate it to the internet? And how difficult is it to migrate an app to your platform?
- kentonv 12y ago
  Sandstorm uses a native-code sandbox on Linux which can run standard Linux binaries, so generally porting an app to Sandstorm is pretty easy. Most of the work is actually in removing the app's existing login system and relying on Sandstorm's authentication instead. David touched on this in his post about porting ShareLaTeX last week: https://blog.sandstorm.io/news/2014-07-25-sharelatex.html https://blog.sandstorm.io/news/2014-07-25-sharelatex.html Most of our existing apps are actually ports of existing open source Linux-based servers. Most were ported by Jason Paryani over the last month, which tells you how easy it is. This includes apps written in PHP, Python, Node, etc. Note that the dev tools (and all the platform source code) are available to anyone. There are some WIP docs on the wiki: https://github.com/sandstorm-io/sandstorm/wiki https://github.com/sandstorm-io/sandstorm/wiki It's probably actually harder to port _away_ from Sandstorm simply because you have to add a login system and other such features. Adding is a lot harder than removing. :)
  - nobodysfool 12y ago
    But I see on some I can download data - couldn't I just port away by copying the data over? Anyway, this is wonderful. One more question - web2py?
    - kentonv 12y ago
      Oh, maybe I misunderstood. I thought your question was about porting code, not data. It's actually incredibly easy to move your data between Sandstorm hosts. Unlike, say, Google apps, Sandstorm apps are not tied to the hosting, so you can change hosts while still using exactly the same app. You can also run Sandstorm on your own machine if you want (it's open source). Porting data from a Sandstorm version of an app to a non-Sandstorm version is also relatively easy. If you click the "download a backup" button, it's just a zip file that contains the app's database in whatever format it uses. But, how exactly to set up the app outside of Sandstorm depends entirely on the app. Regarding web2py: I don't know much about it, but I assume it runs on Linux so it should work fine for writing a Sandstorm app...
      - nobodysfool 12y ago
        Web2Py is a python-based website, like django, but it's a bit different. Since you say you are passing an authentication header, I don't see how it could work with Sandstorm, since it wouldn't really be used as a one-user app anyway.
  - chatmasta 12y ago
    So the apps on the platform are essentially forked versions of open source apps. If you're editing the authentication system, which I imagine can introduce some heavy dependencies, how do you plan on handling updates from the original developers? Presumably there will be manual labor involved with merging the origin master branch into the Sandstorm app. Do you plan to do this yourself? Nice implementation.
    - kentonv 12y ago
      > If you're editing the authentication system, which I imagine can introduce some heavy dependencies... Not really true. Sandstorm acts as a proxy in front of the apps and adds headers which identify the user. It takes very little code on the app side to read these headers and use them rather than use the app's own authentication. That said, we do have some changes which we'll have to rebase from time to time. Eventually we hope that app developers will be interested in maintaining these changes themselves and explicitly targeting Sandstorm, but that is probably a ways away still. In the meantime, maintaining the edits doesn't seem to painful.
AceJohnny2 12y ago
Just to tie a few other things together, kentonv was the primary author of Google's Protocol Buffers, and current author of open-source Cap'n Proto [1]. While at Google, he initiated the Jeff Dean Facts prank [2], and in his spare time part-designed his own LAN-party-optimized house. [3] So, you know, if he thinks he can provide a differentiating hosting platform, I'd have a glance. [1] http://kentonv.github.io/capnproto/ http://kentonv.github.io/capnproto/ [2] https://plus.google.com/+KentonVarda/posts/TSDhe5CvaFe https://plus.google.com/+KentonVarda/posts/TSDhe5CvaFe [3] http://kentonsprojects.blogspot.com/2011/12/lan-party-optimized-house.html http://kentonsprojects.blogspot.com/2011/12/lan-party-optimi...
- kentonv 12y ago
  Oh my. Thanks! Just need to correct one thing: I didn't invent Protobufs (Jeff Dean and his colleague Sanjay Ghemawat did), but I did write most of version 2 and was responsible for open sourcing it.
  - AceJohnny2 12y ago
    Thanks for the correction! In a way, it's kind of demoralizing that it's actually yet another product of the Jeff & Sanjay Brain Trust. It highlights how much they're in a league of their own, unattainable. That said, my googler friends say they much prefer ProtoBufs v2 ;)
    - kentonv 12y ago
      Part of the reason Jeff and Sanjay invented half the tech at Google is that they're incredibly smart. The other part is that they were there very early on, and basically have had free reign to spend their time however they want. Of course they spend it all inventing things, which means they output a lot of awesome stuff. Certainly not just any hacker can produce the kind of quality work they do, but I think there are a lot of other very smart people at Google and elsewhere who could be doing similar things if given the same deal. But, most of those other people didn't bet on Google as early as Jeff and Sanjay did, so...
    - deleted 12y ago
kentonv 12y ago
Thanks for the load test, HN! :) Several hundred users spinning up instances of apps yet the server hasn't broken a sweat... We still need your help to get this into production. Please check out the crowdfunding campaign: http://igg.me/at/sandstorm http://igg.me/at/sandstorm
- e12e 12y ago
  Great stuff. Could you share some data on the server and the traffic? Ram/cores/uplink? This looks like the kind of stuff I've been thinking about for a long time - will have look at the code and test it on a server. Is everything in place to run a clone of your demo server easily?
  - kentonv 12y ago
    Sure! Over the course of yesterday we had 747 demo users launch 952 app instances. Currently we're on GCE for price reasons, though we're looking for good non-Google, non-Amazon options for our eventual release. Sandstorm tends to be memory-bottlenecked since we're launching separate instances of each app for each user, although we make up a lot of it by shutting down instances when they aren't in active use. I was super-paranoid about the demo totally breaking under load (having not tried this before), so I upped the server to GCE's largest-memory instance at 104GB of RAM. That turned out to be completely unnecessary. It never used more than 6GB (and CPU was mostly negligible). Doh. Oh well, better to have spent too much than too little. We also have a separate front-end server which does our SSL, so that our private keys don't reside on the same server as apps, just as an extra precaution. That machine is just a regular n1-standard-1 instance. CPU usage was in the single digit percentages there as well. You can indeed run your own Sandstorm instance. There are installation instructions in our Github readme: https://github.com/sandstorm-io/sandstorm https://github.com/sandstorm-io/sandstorm Feel free to send me questions.
    - e12e 12y ago
      Thanks for the detailed reply. I had some trouble building the whoke thing under Debian stable (with a 3.14 kernel from backports, trying to force everything to work under my user id in $HOME/opt. Not too surprising. Meteor serms to be part of the problem (got it working with its self-contained binary install) - as well as the apparent need for a newer libfuse and some c++ headers I couldn't quite figure out. Although I was building with clang-3.5). I'll have a go at getting the vagrant box up, and then trying again in a Debian testing vm/chroot. If that fails, I'll have to go with an ubuntu vm/chroot :-) At any rate, sounds like my 4 core gbuplink hetzner box with 16gb ram should have no problem working with this (obviously overkill for a personal playpen, but its good to know I have lots of headroom...). Are you planning to stay with meteor going forward? The other components seem much less complicated to get up and running, and more sys.adm. friendly. An example being the need (?) for node/npm and meteor's bundled node/npm...
techaddict009 12y ago
I was trying out Demo and found that you do not have it wordpress in sandstrom apps list. Hope to see it soon.
- kentonv 12y ago
  Ah, the 800lb Gorilla of open source web apps. We wanted to highlight some of the little guys first, but... it's coming.
etherealG 12y ago
oh wow, the demo setup 'just works' and is instant to install apps. this just keeps getting better and better.
swah 12y ago
A cool project in this "own your data" space is https://camlistore.org/ https://camlistore.org/ which IIUC could be the storage system of a personal cloud. OT: I always take a closer look when a computer screen appears in a video - in this case I couldn't recognize what you're running there. My guess would be a a tiling window manager? But the other screen shows Windows so I'm curious. The code on the right window looks like Haskell due to the one liners. https://dl.dropboxusercontent.com/u/19746944/kv.png https://dl.dropboxusercontent.com/u/19746944/kv.png (Also, its kinda surprising/refresing that Sandstorm isn't written in Go or Node... )
- deleted 12y ago
  [deleted]
- kentonv 12y ago
  > https://dl.dropboxusercontent.com/u/19746944/kv.png https://dl.dropboxusercontent.com/u/19746944/kv.png That's me running Qt Creator (my personal favorite IDE) on Linux. You may be disappointed to learn that the code is C++. :) Sandstorm's lower layers are all C++ since we interface closely with Linux syscalls, though I think in that shot I'm actually editing Cap'n Proto (which Sandstorm uses). All of the machines in the video are running Linux, with GNOME 3 as the window manager. > Camlistore Yep, I've talked to Bradfitz about Camlistore a few times. Sandstorm actually gives each app instance its own directory on the filesystem for storage, so that it can use any storage format it wants. So an app could use Camlistore. It may also make sense for Camlistore to act as an independent app which other apps connect to. But in any case, Sandstorm is agnostic to these things.

88 points by kentonv 12y ago

wmf 12y ago

Wired article explains the background: http://www.wired.com/2014/07/out-in-the-open-sandstorm-makes-it-easy-to-control-your-apps-in-the-cloud/ http://www.wired.com/2014/07/out-in-the-open-sandstorm-makes...

nobodysfool 12y ago

That's pretty amazing how quickly I can install and use an app. It's instant as far as I can tell. This makes me wonder - how difficult would it be to take a sandstorm app and migrate it to the internet? And how difficult is it to migrate an app to your platform?

kentonv 12y ago
Sandstorm uses a native-code sandbox on Linux which can run standard Linux binaries, so generally porting an app to Sandstorm is pretty easy. Most of the work is actually in removing the app's existing login system and relying on Sandstorm's authentication instead. David touched on this in his post about porting ShareLaTeX last week: https://blog.sandstorm.io/news/2014-07-25-sharelatex.html https://blog.sandstorm.io/news/2014-07-25-sharelatex.html Most of our existing apps are actually ports of existing open source Linux-based servers. Most were ported by Jason Paryani over the last month, which tells you how easy it is. This includes apps written in PHP, Python, Node, etc. Note that the dev tools (and all the platform source code) are available to anyone. There are some WIP docs on the wiki: https://github.com/sandstorm-io/sandstorm/wiki https://github.com/sandstorm-io/sandstorm/wiki It's probably actually harder to port _away_ from Sandstorm simply because you have to add a login system and other such features. Adding is a lot harder than removing. :)
- nobodysfool 12y ago
  But I see on some I can download data - couldn't I just port away by copying the data over? Anyway, this is wonderful. One more question - web2py?
  - kentonv 12y ago
    Oh, maybe I misunderstood. I thought your question was about porting code, not data. It's actually incredibly easy to move your data between Sandstorm hosts. Unlike, say, Google apps, Sandstorm apps are not tied to the hosting, so you can change hosts while still using exactly the same app. You can also run Sandstorm on your own machine if you want (it's open source). Porting data from a Sandstorm version of an app to a non-Sandstorm version is also relatively easy. If you click the "download a backup" button, it's just a zip file that contains the app's database in whatever format it uses. But, how exactly to set up the app outside of Sandstorm depends entirely on the app. Regarding web2py: I don't know much about it, but I assume it runs on Linux so it should work fine for writing a Sandstorm app...
    - nobodysfool 12y ago
      Web2Py is a python-based website, like django, but it's a bit different. Since you say you are passing an authentication header, I don't see how it could work with Sandstorm, since it wouldn't really be used as a one-user app anyway.
- chatmasta 12y ago
  So the apps on the platform are essentially forked versions of open source apps. If you're editing the authentication system, which I imagine can introduce some heavy dependencies, how do you plan on handling updates from the original developers? Presumably there will be manual labor involved with merging the origin master branch into the Sandstorm app. Do you plan to do this yourself? Nice implementation.
  - kentonv 12y ago
    > If you're editing the authentication system, which I imagine can introduce some heavy dependencies... Not really true. Sandstorm acts as a proxy in front of the apps and adds headers which identify the user. It takes very little code on the app side to read these headers and use them rather than use the app's own authentication. That said, we do have some changes which we'll have to rebase from time to time. Eventually we hope that app developers will be interested in maintaining these changes themselves and explicitly targeting Sandstorm, but that is probably a ways away still. In the meantime, maintaining the edits doesn't seem to painful.

AceJohnny2 12y ago

Just to tie a few other things together, kentonv was the primary author of Google's Protocol Buffers, and current author of open-source Cap'n Proto [1]. While at Google, he initiated the Jeff Dean Facts prank [2], and in his spare time part-designed his own LAN-party-optimized house. [3] So, you know, if he thinks he can provide a differentiating hosting platform, I'd have a glance. [1] http://kentonv.github.io/capnproto/ http://kentonv.github.io/capnproto/ [2] https://plus.google.com/+KentonVarda/posts/TSDhe5CvaFe https://plus.google.com/+KentonVarda/posts/TSDhe5CvaFe [3] http://kentonsprojects.blogspot.com/2011/12/lan-party-optimized-house.html http://kentonsprojects.blogspot.com/2011/12/lan-party-optimi...

kentonv 12y ago
Oh my. Thanks! Just need to correct one thing: I didn't invent Protobufs (Jeff Dean and his colleague Sanjay Ghemawat did), but I did write most of version 2 and was responsible for open sourcing it.
- AceJohnny2 12y ago
  Thanks for the correction! In a way, it's kind of demoralizing that it's actually yet another product of the Jeff & Sanjay Brain Trust. It highlights how much they're in a league of their own, unattainable. That said, my googler friends say they much prefer ProtoBufs v2 ;)
  - kentonv 12y ago
    Part of the reason Jeff and Sanjay invented half the tech at Google is that they're incredibly smart. The other part is that they were there very early on, and basically have had free reign to spend their time however they want. Of course they spend it all inventing things, which means they output a lot of awesome stuff. Certainly not just any hacker can produce the kind of quality work they do, but I think there are a lot of other very smart people at Google and elsewhere who could be doing similar things if given the same deal. But, most of those other people didn't bet on Google as early as Jeff and Sanjay did, so...
  - deleted 12y ago

kentonv 12y ago

Thanks for the load test, HN! :) Several hundred users spinning up instances of apps yet the server hasn't broken a sweat... We still need your help to get this into production. Please check out the crowdfunding campaign: http://igg.me/at/sandstorm http://igg.me/at/sandstorm

e12e 12y ago
Great stuff. Could you share some data on the server and the traffic? Ram/cores/uplink? This looks like the kind of stuff I've been thinking about for a long time - will have look at the code and test it on a server. Is everything in place to run a clone of your demo server easily?
- kentonv 12y ago
  Sure! Over the course of yesterday we had 747 demo users launch 952 app instances. Currently we're on GCE for price reasons, though we're looking for good non-Google, non-Amazon options for our eventual release. Sandstorm tends to be memory-bottlenecked since we're launching separate instances of each app for each user, although we make up a lot of it by shutting down instances when they aren't in active use. I was super-paranoid about the demo totally breaking under load (having not tried this before), so I upped the server to GCE's largest-memory instance at 104GB of RAM. That turned out to be completely unnecessary. It never used more than 6GB (and CPU was mostly negligible). Doh. Oh well, better to have spent too much than too little. We also have a separate front-end server which does our SSL, so that our private keys don't reside on the same server as apps, just as an extra precaution. That machine is just a regular n1-standard-1 instance. CPU usage was in the single digit percentages there as well. You can indeed run your own Sandstorm instance. There are installation instructions in our Github readme: https://github.com/sandstorm-io/sandstorm https://github.com/sandstorm-io/sandstorm Feel free to send me questions.
  - e12e 12y ago
    Thanks for the detailed reply. I had some trouble building the whoke thing under Debian stable (with a 3.14 kernel from backports, trying to force everything to work under my user id in $HOME/opt. Not too surprising. Meteor serms to be part of the problem (got it working with its self-contained binary install) - as well as the apparent need for a newer libfuse and some c++ headers I couldn't quite figure out. Although I was building with clang-3.5). I'll have a go at getting the vagrant box up, and then trying again in a Debian testing vm/chroot. If that fails, I'll have to go with an ubuntu vm/chroot :-) At any rate, sounds like my 4 core gbuplink hetzner box with 16gb ram should have no problem working with this (obviously overkill for a personal playpen, but its good to know I have lots of headroom...). Are you planning to stay with meteor going forward? The other components seem much less complicated to get up and running, and more sys.adm. friendly. An example being the need (?) for node/npm and meteor's bundled node/npm...

techaddict009 12y ago

I was trying out Demo and found that you do not have it wordpress in sandstrom apps list. Hope to see it soon.

kentonv 12y ago
Ah, the 800lb Gorilla of open source web apps. We wanted to highlight some of the little guys first, but... it's coming.

etherealG 12y ago

oh wow, the demo setup 'just works' and is instant to install apps. this just keeps getting better and better.

swah 12y ago

A cool project in this "own your data" space is https://camlistore.org/ https://camlistore.org/ which IIUC could be the storage system of a personal cloud. OT: I always take a closer look when a computer screen appears in a video - in this case I couldn't recognize what you're running there. My guess would be a a tiling window manager? But the other screen shows Windows so I'm curious. The code on the right window looks like Haskell due to the one liners. https://dl.dropboxusercontent.com/u/19746944/kv.png https://dl.dropboxusercontent.com/u/19746944/kv.png (Also, its kinda surprising/refresing that Sandstorm isn't written in Go or Node... )

deleted 12y ago
[deleted]
kentonv 12y ago
> https://dl.dropboxusercontent.com/u/19746944/kv.png https://dl.dropboxusercontent.com/u/19746944/kv.png That's me running Qt Creator (my personal favorite IDE) on Linux. You may be disappointed to learn that the code is C++. :) Sandstorm's lower layers are all C++ since we interface closely with Linux syscalls, though I think in that shot I'm actually editing Cap'n Proto (which Sandstorm uses). All of the machines in the video are running Linux, with GNOME 3 as the window manager. > Camlistore Yep, I've talked to Bradfitz about Camlistore a few times. Sandstorm actually gives each app instance its own directory on the filesystem for storage, so that it can use any storage format it wants. So an app could use Camlistore. It may also make sense for Camlistore to act as an independent app which other apps connect to. But in any case, Sandstorm is agnostic to these things.