4 ms·

>and a big part of the reason for posting here was for someone to say "hey, here's a wheel you can buy off the shelf!" not "hur hur, you're stupid because you'r

>and a big part of the reason for posting here was for someone to say "hey, here's a wheel you can buy off the shelf!" not "hur hur, you're stupid because you're assembling a web stack, fail!". I don't think assembling a web stack is automatic fail, but I think not buying a wheel you don't need is even better than buying one off the shelf. ;) >Already had decided, if you'll notice the followups below, to replace redis with memcached, though I don't see how it would be a single point of failure if was just using it as a cache If you're just using it as a cache then the SPOF doesn't matter, you're right, but if you're just using it as a cache then I'd say it's the wrong tool. It's great at doing fancy stuff with data structures in memory, but memory on a single node is all you get. (Dumping memory to disk improves durability, but does nothing for capacity.) If you want more cache than that, then you'll have to add your own clustering. Memcached itself is not any better, actually. At the very least you'll need to add libketama to consistent-hash across several memcached instances. Even better would be to use something like Kumofs or Voldemort which provide roughly the same semantics - Kumofs even uses the same protocol - while also handling server-set changes and such transparently. That brings me to the point about whether to implement a separate caching layer at all. Why? Is there some known or inevitable performance need that wouldn't be solved by having enough cache on the nodes serving the authoritative data? If not, then implementing interfaces to two different layers - with all of the consistency problems and impedance mismatches involved - will just be a waste. My general advice is to never add a tier unless you're absolutely and empirically sure that the pain your gaining won't outweigh the pain you're avoiding. >I also figured I'd need to write some scripts to have nodes join and leave the clusters, mostly calling the Apis built into the components of the stack Writing those scripts can be a significant part of implementing your own cluster infrastructure. I'm sure we could have an interesting discussion about whether writing those scripts would be easy or hard, but IMO the more pressing question is whether they need to be written at all. If your storage layer (e.g. BigCouch) already does its own clustering, then you need to do practically nothing at that level. If you then keep your front-end servers as stateless as possible (i.e. not storing any persistent/shared data other than in the primary data store) then adding/removing those servers should be a complete no-brainer. Just give them some addresses and credentials, and boom - no cluster magic necessary. It's not components I'm recommending so much as a pattern that avoids the need for so many components, but I hope it's still useful. Good luck.

lzw 16y ago
Thanks for your thoughts, and I don't disagree with the general philosophy you're pitching. I think, though, you may be imagining something different than I'm trying to describe. It seems that you're thinking in terms of front end and back end servers and a cache that is seperate severs, which is why you suggest that the cache in the nodes serving the authoritative data should be sufficient. I'm describing a system whereby there is only one server configuration and it contains the whole stack and thus to increase capacity you add more servers, which means more cache, more database capacity, etc. This may be non-traditional, but it is amenable to an ec2 type of setup where you scale the cluster up or down without having to keep ratios of types of servers and their connections coordinated. I don't intend to change the cluster size, except to slowly add more servers over time. But having different types means different configurations, and keeping it coordinated. The idea here is, one type, it is already configured, it is actually an image and you just clone it. Didn't realize that memecahced worked that way, I do need to look into it more, and thanks for the alternative recommendations. There are some elements, like images that appear on every page, or CSS files, that will be served much more than others, I can store all these elements in the couch, and maybe they will live in ram due to the systems virtual memory, making explicit caching pointless. I will need to do tests here, you are right. But I was going by some previous experience and common "best practices" advice to try and explicitly keep some of this in ram. Further, I do want ot keep some operational state information in ram, so that it can be written often, even though it doesn't need to be persisted, but does need to be read by multiple machines in the cluster. Anyway, I appreciate your suggestion, and agree with the desire to keep complexity down. One of the ways I'm doing this is to have a single machine type, thus no complex heirarchy of machines that have to be configured to work with each other. Just add additional nodes to add capacity.
lzw 16y ago
Btw, one thing ive been looking for is a relatively high level queue/worker service, that is already written and distributed. I'd like to be able to write workers in JavaScript or python, and found one the other night in erlang that does this, called DISCO. However, it seems a bit heavyweight. Just need to spawn periodic tasks that will gather data and feed it into the couch. I could do it with cron, but something that divides them over each machine in the cluster seems like something people would have written already. Maybe DISCO is what I need but it seems a bit heavyweight, while cron is too little.

lzw 16y ago

Thanks for your thoughts, and I don't disagree with the general philosophy you're pitching. I think, though, you may be imagining something different than I'm trying to describe. It seems that you're thinking in terms of front end and back end servers and a cache that is seperate severs, which is why you suggest that the cache in the nodes serving the authoritative data should be sufficient. I'm describing a system whereby there is only one server configuration and it contains the whole stack and thus to increase capacity you add more servers, which means more cache, more database capacity, etc. This may be non-traditional, but it is amenable to an ec2 type of setup where you scale the cluster up or down without having to keep ratios of types of servers and their connections coordinated. I don't intend to change the cluster size, except to slowly add more servers over time. But having different types means different configurations, and keeping it coordinated. The idea here is, one type, it is already configured, it is actually an image and you just clone it. Didn't realize that memecahced worked that way, I do need to look into it more, and thanks for the alternative recommendations. There are some elements, like images that appear on every page, or CSS files, that will be served much more than others, I can store all these elements in the couch, and maybe they will live in ram due to the systems virtual memory, making explicit caching pointless. I will need to do tests here, you are right. But I was going by some previous experience and common "best practices" advice to try and explicitly keep some of this in ram. Further, I do want ot keep some operational state information in ram, so that it can be written often, even though it doesn't need to be persisted, but does need to be read by multiple machines in the cluster. Anyway, I appreciate your suggestion, and agree with the desire to keep complexity down. One of the ways I'm doing this is to have a single machine type, thus no complex heirarchy of machines that have to be configured to work with each other. Just add additional nodes to add capacity.

lzw 16y ago

Btw, one thing ive been looking for is a relatively high level queue/worker service, that is already written and distributed. I'd like to be able to write workers in JavaScript or python, and found one the other night in erlang that does this, called DISCO. However, it seems a bit heavyweight. Just need to spawn periodic tasks that will gather data and feed it into the couch. I could do it with cron, but something that divides them over each machine in the cluster seems like something people would have written already. Maybe DISCO is what I need but it seems a bit heavyweight, while cron is too little.