4 ms·

Seems like a lot of reinvention to me - cluster management, DNS service, caching, queue service, etc. Reinventing cluster-management infrastructure, for exampl

Seems like a lot of reinvention to me - cluster management, DNS service, caching, queue service, etc. Reinventing cluster-management infrastructure, for example, is just a bad idea plain and simple. It will take a lot of time, and that time won't be very enjoyable. You seem rather proud of having been an engineer longer than some commenters have been alive, but just about everybody who could say that about me is retired or dead by now and I actually have done production-level work in some of these areas so maybe you'll have to find a different excuse for dismissing my advice. Although it's possible to determine that the large pile of bleeding-edge technology and almost-as-large pile of reinvented wheels that you've described is almost certainly the wrong solution, it's hard to tell what the right solution might be without any description of what you're trying to achieve. At least part of it looks like an attempt to clone DynDNS, so perhaps one starting point would be to look at what they're doing and how it might be improved. Using something like BigCouch that's already clustered seems like a good start, though I'd contend that something like Riak/Voldemort/Cassandra does that particular kind of clustering better. Adding Redis to the mix seems to add very little, since it's inherently a SPOF and you don't mention anything that would use its unique capabilities. Similarly, why not measure the effectiveness of the caching already inherent in your primary Couch/Riak/whatever store before you consider adding yet another caching layer? My advice would be to use BigCouch/Riak or similar as your authoritative data store, and not add Redis or separate caching. That leaves your node.js front ends as the only thing that could possibly benefit from your home-grown cluster management, and if they're written properly so that they're not coupled to one another (bad for availability anyway) then you won't need that either. If you try that and still find that you need to build something else then fine, but I very seriously doubt that you would except as an artifact of implementing the simpler architecture poorly.

deleted 16y ago
[deleted]
lzw 16y ago
You mentioned raik/Cassandra etc do the clustering better, and this may be true, but a big part of the attraction of couchdb is couchApps and the built in map reduce which does the data crunching we need to do in a way well suited to our problem. Will have to investigate raik more, just haven't quite gotten my head around walking links. Or how vwebmachine fits in.... Whether it is comparable to couch apps. If you want a context in which to think of our product, think of it as a recommendation engine or search engine. A large amount of data comes in, some of it realtime, and it needs to be processed and queries of various types need to be handled, some using customer specific information. Eg: someone who likes shoes will see more shoes in their results. If we're at amazon, well use amazons load balancer. But it looks like we'll be at linode or somewhere like that initially... And since we need to host a dns server anyway, figured round robin dns is the way to go. I suspect it would take more coding effort to get powerDNS pulling records out of the couch an it would to adapt a pre-written authoritative server for node to our needs. But since node seems like it wont be ready in time... I'm not planning any wheel reinventing, and a big part of the reason for posting here was for someone to say "hey, here's a wheel you can buy off the shelf!" not "hur hur, you're stupid because you're assembling a web stack, fail!". Already had decided, if you'll notice the followups below, to replace redis with memcached, though I don't see how it would be a single point of failure if was just using it as a cache, which is all i was using it for. I just saw it had support for some nice data structures and figured it would be better... But changed my mind when i realized it didn't have an LRU. As for cluster management, turn key Linux provides some of that, and otherwise it was just going to be keeping queues and load readings in the distributed cache so that large jobs could be distributed to less busy machines. This doesn't seem like a big deal to me, because it wasn't even something we needed in a big way, though I didn't mention that. But if someone has already done it feel free to let me know. I also figured I'd need to write some scripts to have nodes join and leave the clusters, mostly calling the Apis built into the components of the stack.... Figure everyone has to do this. I know there are some open source projects for managing cloud clusters, but not very familiar and would appreciate recommendations if there are some that would be a good fit.
- CPlatypus 16y ago
  >and a big part of the reason for posting here was for someone to say "hey, here's a wheel you can buy off the shelf!" not "hur hur, you're stupid because you're assembling a web stack, fail!". I don't think assembling a web stack is automatic fail, but I think not buying a wheel you don't need is even better than buying one off the shelf. ;) >Already had decided, if you'll notice the followups below, to replace redis with memcached, though I don't see how it would be a single point of failure if was just using it as a cache If you're just using it as a cache then the SPOF doesn't matter, you're right, but if you're just using it as a cache then I'd say it's the wrong tool. It's great at doing fancy stuff with data structures in memory, but memory on a single node is all you get. (Dumping memory to disk improves durability, but does nothing for capacity.) If you want more cache than that, then you'll have to add your own clustering. Memcached itself is not any better, actually. At the very least you'll need to add libketama to consistent-hash across several memcached instances. Even better would be to use something like Kumofs or Voldemort which provide roughly the same semantics - Kumofs even uses the same protocol - while also handling server-set changes and such transparently. That brings me to the point about whether to implement a separate caching layer at all. Why? Is there some known or inevitable performance need that wouldn't be solved by having enough cache on the nodes serving the authoritative data? If not, then implementing interfaces to two different layers - with all of the consistency problems and impedance mismatches involved - will just be a waste. My general advice is to never add a tier unless you're absolutely and empirically sure that the pain your gaining won't outweigh the pain you're avoiding. >I also figured I'd need to write some scripts to have nodes join and leave the clusters, mostly calling the Apis built into the components of the stack Writing those scripts can be a significant part of implementing your own cluster infrastructure. I'm sure we could have an interesting discussion about whether writing those scripts would be easy or hard, but IMO the more pressing question is whether they need to be written at all. If your storage layer (e.g. BigCouch) already does its own clustering, then you need to do practically nothing at that level. If you then keep your front-end servers as stateless as possible (i.e. not storing any persistent/shared data other than in the primary data store) then adding/removing those servers should be a complete no-brainer. Just give them some addresses and credentials, and boom - no cluster magic necessary. It's not components I'm recommending so much as a pattern that avoids the need for so many components, but I hope it's still useful. Good luck.

deleted 16y ago

[deleted]

lzw 16y ago

You mentioned raik/Cassandra etc do the clustering better, and this may be true, but a big part of the attraction of couchdb is couchApps and the built in map reduce which does the data crunching we need to do in a way well suited to our problem. Will have to investigate raik more, just haven't quite gotten my head around walking links. Or how vwebmachine fits in.... Whether it is comparable to couch apps. If you want a context in which to think of our product, think of it as a recommendation engine or search engine. A large amount of data comes in, some of it realtime, and it needs to be processed and queries of various types need to be handled, some using customer specific information. Eg: someone who likes shoes will see more shoes in their results. If we're at amazon, well use amazons load balancer. But it looks like we'll be at linode or somewhere like that initially... And since we need to host a dns server anyway, figured round robin dns is the way to go. I suspect it would take more coding effort to get powerDNS pulling records out of the couch an it would to adapt a pre-written authoritative server for node to our needs. But since node seems like it wont be ready in time... I'm not planning any wheel reinventing, and a big part of the reason for posting here was for someone to say "hey, here's a wheel you can buy off the shelf!" not "hur hur, you're stupid because you're assembling a web stack, fail!". Already had decided, if you'll notice the followups below, to replace redis with memcached, though I don't see how it would be a single point of failure if was just using it as a cache, which is all i was using it for. I just saw it had support for some nice data structures and figured it would be better... But changed my mind when i realized it didn't have an LRU. As for cluster management, turn key Linux provides some of that, and otherwise it was just going to be keeping queues and load readings in the distributed cache so that large jobs could be distributed to less busy machines. This doesn't seem like a big deal to me, because it wasn't even something we needed in a big way, though I didn't mention that. But if someone has already done it feel free to let me know. I also figured I'd need to write some scripts to have nodes join and leave the clusters, mostly calling the Apis built into the components of the stack.... Figure everyone has to do this. I know there are some open source projects for managing cloud clusters, but not very familiar and would appreciate recommendations if there are some that would be a good fit.

CPlatypus 16y ago
>and a big part of the reason for posting here was for someone to say "hey, here's a wheel you can buy off the shelf!" not "hur hur, you're stupid because you're assembling a web stack, fail!". I don't think assembling a web stack is automatic fail, but I think not buying a wheel you don't need is even better than buying one off the shelf. ;) >Already had decided, if you'll notice the followups below, to replace redis with memcached, though I don't see how it would be a single point of failure if was just using it as a cache If you're just using it as a cache then the SPOF doesn't matter, you're right, but if you're just using it as a cache then I'd say it's the wrong tool. It's great at doing fancy stuff with data structures in memory, but memory on a single node is all you get. (Dumping memory to disk improves durability, but does nothing for capacity.) If you want more cache than that, then you'll have to add your own clustering. Memcached itself is not any better, actually. At the very least you'll need to add libketama to consistent-hash across several memcached instances. Even better would be to use something like Kumofs or Voldemort which provide roughly the same semantics - Kumofs even uses the same protocol - while also handling server-set changes and such transparently. That brings me to the point about whether to implement a separate caching layer at all. Why? Is there some known or inevitable performance need that wouldn't be solved by having enough cache on the nodes serving the authoritative data? If not, then implementing interfaces to two different layers - with all of the consistency problems and impedance mismatches involved - will just be a waste. My general advice is to never add a tier unless you're absolutely and empirically sure that the pain your gaining won't outweigh the pain you're avoiding. >I also figured I'd need to write some scripts to have nodes join and leave the clusters, mostly calling the Apis built into the components of the stack Writing those scripts can be a significant part of implementing your own cluster infrastructure. I'm sure we could have an interesting discussion about whether writing those scripts would be easy or hard, but IMO the more pressing question is whether they need to be written at all. If your storage layer (e.g. BigCouch) already does its own clustering, then you need to do practically nothing at that level. If you then keep your front-end servers as stateless as possible (i.e. not storing any persistent/shared data other than in the primary data store) then adding/removing those servers should be a complete no-brainer. Just give them some addresses and credentials, and boom - no cluster magic necessary. It's not components I'm recommending so much as a pattern that avoids the need for so many components, but I hope it's still useful. Good luck.