5 ms·

No. There are a number of some pretty serious restrictions. I would _love_ something like Heroku for WSGI applications. Ian Bicking is working on something simi

by jokull 17y ago

No. There are a number of some pretty serious restrictions. I would _love_ something like Heroku for WSGI applications. Ian Bicking is working on something similar, not as high level with his Silver Lining project. http://bitbucket.org/ianb/silverlining/ http://bitbucket.org/ianb/silverlining/

njl 17y ago
A couple of months ago I explored building a Django-friendly Heroku clone. I white-boarded out the architecture, estimated how much work would be needed for all the pieces, and built a spreadsheet revenue model. It's an interesting and attractive business, but I'm only one man and my C skills are rusty (really, really rusty). I couldn't get myself to a point where I could deliver a product for at least a year, and I want to be ramen profitable before that. My current focus is to look at the plethora of excellent SaaS applications that help provide the whole Rails ecosystem, then turn around and do the same for Django, with an eye towards expanding to handle other Python frameworks, and then into the Java space.
- rgrieselhuber 17y ago
  I see a lot of looking and no doing. ;-)
  - njl 17y ago
    Hah! Well, I extracted myself from my previous commitments about two weeks ago, then spent a week building a co-founder-finder (http://amb.itio.us/ http://amb.itio.us/). I've been spending this week doing mock-ups and filling out the rough architecture of the app I'm building. Give me a month or two.
- thegoleffect 17y ago
  I ended up writing a lot of similar functionality for the higut.com team. It was setup to auto deploy Tornado apps instead of Django but its probably not too much of a stretch to do WSGI in general. Didn't know there'd be interest. I could factor it out and release it? It did deploy on commit/push, bundled eggs, worked with mysql or sqlite and redis. Dunno if that sounds interesting to others though.
  - njl 17y ago
    If you haven't had a chance to mull over Heroku's architecture, I'd suggest giving it a look. http://heroku.com/how/architecture http://heroku.com/how/architecture Basically, nginx fronts a bunch of Varnish caches, which are then in front of a "routing mesh", which queues up requests for individual "dynos". A dyno is a single-threaded instance of your application. When one dies or is migrated, another is deployed in its place. Multiple dynos are automatically spread across multiple machines. It's a very neat solution to automatically scaling hundreds or thousands of apps. In other words, bundling the apps and pushing them out is just the beginning of a pretty fascinating architecture.
    - thegoleffect 17y ago
      Yeah, I meant those were additional features. We used a similar setup on Rackspace Cloud, though not as refined (or as secure imo). Tornado really lends itself well to a dyno type arrangement. We used nginx for load balancing and automatic frontend resolution, probably in a similar way as Heroku. Thanks for the tip though, interesting read. Talking to some other Pythonistas, there didn't seem to be much desire for a Heroku-equivalent system though - most ppl enjoy rolling their own deployments.
- rcoder 17y ago
  I think you could get well into the "people are paying money for this service" stage of a hosted Django hosting platform without ever having to touch C. Python is fast enough to do most of the systems-level programming you'd need for such a beast, especially since so much of the standard library (socket, struct, asyncore, subprocess) sits directly on top of the fast POSIX syscalls you'd be invoking to do the same work in C anyway. Going down to the metal may help squeeze the last 10-20% out of your hardware, but the the really interesting and challenging work in coming up with a scalable hosting platform is elsewhere: security, monitoring, process spawning/reaping, deployment, et. al. If working in Python gives you a time-to-market advantage, then go for it. You can hire a C hacker when you have enough business to make the improvement in your hardware utilization efficiency pay off.
  - njl 17y ago
    Well, some effort needs to go into some plugins for nginx and varnish. nginx needs to route incoming requests to the right varnish, and some custom cache invalidation stuff should happen on varnish as well. Other than that? Lots and lots of python, of course ;) I'm only one guy, and there are three or four big moving parts that need to be written. Something to build eggs and push them, something to monitor and manage processes on individual servers, something to dispatch requests to the appropriate servers, and something to monitor and manage the database servers. Even pulling back and looking at a minimum viable product, I'm still doing most of that work, just far simpler versions. Succinctly, it's too big for me to tackle on my own in the next six months.
    - jokull 17y ago
      I'd do without eggs and provide common Python libraries that need compiling on the server (lxml, Crypto etc.). You'd then simply accept Python code over the live for deployments. You'd also want to handle rollback, schema migration, key-value services and a DB per instance, some sort of dumping/loading of data. If you stick with certain django conventions (namely settings-as-a-module and apps/libs on PYTHONPATH) you could get away with a really sweet and simple deployment procedure. A pretty smart client to handle all this is a lot of work too. Have a look at Silver Lining. Just got EC2 working on my fork ;)

by jokull 17y ago

njl 17y ago

A couple of months ago I explored building a Django-friendly Heroku clone. I white-boarded out the architecture, estimated how much work would be needed for all the pieces, and built a spreadsheet revenue model. It's an interesting and attractive business, but I'm only one man and my C skills are rusty (really, really rusty). I couldn't get myself to a point where I could deliver a product for at least a year, and I want to be ramen profitable before that. My current focus is to look at the plethora of excellent SaaS applications that help provide the whole Rails ecosystem, then turn around and do the same for Django, with an eye towards expanding to handle other Python frameworks, and then into the Java space.

rgrieselhuber 17y ago
I see a lot of looking and no doing. ;-)
- njl 17y ago
  Hah! Well, I extracted myself from my previous commitments about two weeks ago, then spent a week building a co-founder-finder (http://amb.itio.us/ http://amb.itio.us/). I've been spending this week doing mock-ups and filling out the rough architecture of the app I'm building. Give me a month or two.
thegoleffect 17y ago
I ended up writing a lot of similar functionality for the higut.com team. It was setup to auto deploy Tornado apps instead of Django but its probably not too much of a stretch to do WSGI in general. Didn't know there'd be interest. I could factor it out and release it? It did deploy on commit/push, bundled eggs, worked with mysql or sqlite and redis. Dunno if that sounds interesting to others though.
- njl 17y ago
  If you haven't had a chance to mull over Heroku's architecture, I'd suggest giving it a look. http://heroku.com/how/architecture http://heroku.com/how/architecture Basically, nginx fronts a bunch of Varnish caches, which are then in front of a "routing mesh", which queues up requests for individual "dynos". A dyno is a single-threaded instance of your application. When one dies or is migrated, another is deployed in its place. Multiple dynos are automatically spread across multiple machines. It's a very neat solution to automatically scaling hundreds or thousands of apps. In other words, bundling the apps and pushing them out is just the beginning of a pretty fascinating architecture.
  - thegoleffect 17y ago
    Yeah, I meant those were additional features. We used a similar setup on Rackspace Cloud, though not as refined (or as secure imo). Tornado really lends itself well to a dyno type arrangement. We used nginx for load balancing and automatic frontend resolution, probably in a similar way as Heroku. Thanks for the tip though, interesting read. Talking to some other Pythonistas, there didn't seem to be much desire for a Heroku-equivalent system though - most ppl enjoy rolling their own deployments.
rcoder 17y ago
I think you could get well into the "people are paying money for this service" stage of a hosted Django hosting platform without ever having to touch C. Python is fast enough to do most of the systems-level programming you'd need for such a beast, especially since so much of the standard library (socket, struct, asyncore, subprocess) sits directly on top of the fast POSIX syscalls you'd be invoking to do the same work in C anyway. Going down to the metal may help squeeze the last 10-20% out of your hardware, but the the really interesting and challenging work in coming up with a scalable hosting platform is elsewhere: security, monitoring, process spawning/reaping, deployment, et. al. If working in Python gives you a time-to-market advantage, then go for it. You can hire a C hacker when you have enough business to make the improvement in your hardware utilization efficiency pay off.
- njl 17y ago
  Well, some effort needs to go into some plugins for nginx and varnish. nginx needs to route incoming requests to the right varnish, and some custom cache invalidation stuff should happen on varnish as well. Other than that? Lots and lots of python, of course ;) I'm only one guy, and there are three or four big moving parts that need to be written. Something to build eggs and push them, something to monitor and manage processes on individual servers, something to dispatch requests to the appropriate servers, and something to monitor and manage the database servers. Even pulling back and looking at a minimum viable product, I'm still doing most of that work, just far simpler versions. Succinctly, it's too big for me to tackle on my own in the next six months.
  - jokull 17y ago
    I'd do without eggs and provide common Python libraries that need compiling on the server (lxml, Crypto etc.). You'd then simply accept Python code over the live for deployments. You'd also want to handle rollback, schema migration, key-value services and a DB per instance, some sort of dumping/loading of data. If you stick with certain django conventions (namely settings-as-a-module and apps/libs on PYTHONPATH) you could get away with a really sweet and simple deployment procedure. A pretty smart client to handle all this is a lot of work too. Have a look at Silver Lining. Just got EC2 working on my fork ;)