7 ms·
Ask HN: Node.js+Redis+CouchApp+BigCouch+CouchDB+TurnkeyLinux = Nirvana?
We've been very happy with AppEngine, especially since I hate system administration and maintenance, but we've got features coming up that simply can't be done on appEngine. I've also got some "hacker time" coming up, about two months with no deliverables so I can work on a new infrastructure.... And I'm ruminating on a solution which ill share below. I'm keen to know if any of this is clueless or foolish, as I have not worked with any of these new hot technologies before, though I do know erlang and JavaScript pretty well.
All these technologies are to be assembled into a pre-built appliance. Most of it is the set of apps described in the title, but the real magic comes from a couple relatively static applications written in node.js. To increase capacity, we add more nodes. Should be zero configuration.
Under node.js there are services I have to write-
cluster management-- which is catch all for connecting new nodes to the cluster, or migrating off of nodes going away.
authoritativeDNS-- the cluster is it's own name servers. This say load can be balanced, and we can dynamically add customers custom domains to our app and support them right away,
web platform with caching-- this will run couch apps, custom JavaScript stored in the couch and but run outside under node.js
Simple queue service, and cron service--- using node.js and redis.
This is tied into redis to cache dns records, static or unchanged pages using the etag couch provides, etc. The cluster manager reports the local nodes CPU load in redis as well, and so the dns knows to return address to direct traffic to nodes in the cluster that are less busy. This way apps can add customer dns so that our neophyte customers can get set up by merely putting out name servers into their registrar.
Most of the product functionality will be written in CouchApps, except for places where couchapp is limited..... Like showing multiple views on a page, and checking for uservauthentication. Don't want one customer to be able to see anthers private info. Well do that, and cron like tasks in node.js.
All of this is stored in couchdb which is distributed dynamo style by big couch. Like couchapp we have JavaScript in design documents, this makes deployment really easy. Part of the node. Web platform will support a/b testing and beta/production splits.
In this way, new node.js level functionality can be added by putting functions in JavaScript into documents and adding urls to the URL routing scheme that point to these handlers, all loaded from the couch, but executed in node.js. So, deployment still is just a push into the couch.
The reason for doing things this way is to make continuous deployment easier, but also to never have to reboot or change e servers much. The node.js services will be well tested and unchanging.
All of this will be built up using turnkey linux into an appliance. This can then be deployed on amazon or vps or wherever without the need for a load balancer in front. Since it does it's own dns and the authoritative dns is the cluster itself.
This should reduce maintenance to the bare minimum,and I can push out a new set of appliances over time and slowly migrate trafic to them once or twice a year when I need to update core funcationality.
I might go with web machine and raik instead of node.js and couch if I can find a compelling reason to, but the big holdup there is th question of how much of ahassle it would be to attach beam files to documents and get web machine to run them.
I know this is unconventional. I'm up for criticisms and suggestions, especially if there is a way to avoid writing all this code in node.js. Please don't tell me it is wrong because it's unconventional or that I don't need this flexibility. I want to just focus on the business and the product, if I could get this kind of capability from a hosted solution, i would, but I can't. (for instance, I need to download zip files nightly, unpack them, feed the data into a database, something I can't do on app engine, for instance, or cloudant, etc.)
Thanks. What do You think?
- smoody 16y agoI don't have an answer for you, but I do have a question: What types of features are you implementing that can't be done with AppEngine? Thanks!
- lzw 16y agoHere's a list of things that may be related to features were going to do thatbwe can't do in app engine. Whate features we actually do will depend on customer response, but were increasingly needing to do stuff that app engine doesn't support. But also, I'm uncomfortable with app engines lockin and odd performance limitations that require a fair bit of optimization. Couchdb makes seems like I will write things efficiently the first time. Anyway it is impossible ornery hard to: -- support new domains for app engine apps. Only way to do it is to manually add them to your google apps account which takes from 48 hours to 48 weeks, and requires control over the domain, which is hard when it is a customers domain. -- apple puch notification sevice, or any kind of long standing process -- big data processing is difficult you can do map, but not reduce on app engine, but you can't unzip a 25gb file and then process the results. Much ok,f the influence here and the focus on easy deployment are inspired by app engine, though.
- jchrisa 16y agoThis sounds like a good fit to me. I'd watch the complexity, though, and see what you can get done with the simplest thing that could possibly work, then build up, always with the end-goal in mind.