4 ms·

I love posts like this because, love it or hate it, it gives me a checkmark of thoughts to compare my choices to. I see three possible approaches, all with the

by nirvana 15y ago

I love posts like this because, love it or hate it, it gives me a checkmark of thoughts to compare my choices to. I see three possible approaches, all with their advantages and disadvantages. (Of course people may fit between them with a mixture of attributes.) 1. Monolithic - Build it all yourself, purpose built and high performance. This is why mailinator and plenty-of-fish are able to produce high thruput on a surprisingly small number of machines. 2. Confederated - Completely distributed. Each machine is its own monolithic platform with everything from DNS to database, including web server on that node, but a cluster of nodes gives you scalability, and workload is distributed across the cluster. (I'm not aware of any examples of this, which is why I'm building Nirvana.) 3. COGS: You build your cluster of machines by architecting a system whereby you minimize (but not eliminate) single points of failure. You have N web server machines and X database machines and you seek out really high performance open source cogs to keep the number of machines low (e.g.: Redis, MongoDB, etc.) The COGS approach is often taken with the idea that we need something really fast. MongoDB being fast (and "SQL") are the reasons its often chosen. Redis being fast is given as a key advantage (which is relevant for an in memory database, sure.) Node.js is often chosen for similar reasons. But the ends result of the COGS approach is a brittle architecture. You may have multiple redundant web servers but the thing that distributes loads is a SPF. More specifically the architecture is complicated- each machine has a different configuration, etc. With Monolithic, you get performance, and save hosting costs, and you can probably scale pretty well because you know your system really well, and you've squeezed out a lot of the inefficiencies that come from being generic (in the cogs approach) such that you can interoperate. What I think we should see more of is confederated- no machine is a unique snowflake. Every machine is identical to every other machine. This way configuration becomes dead simple-- just replicate your model node, bring it up and data and load starts going to it. This can be done with cogs- but they have to be fully distributed cogs. An example is Riak (Which hit 1.1 yesterday) which is open source and written in erlang and probably loses to nodeDB in every single single node benchmark you can come up with (not that the Basho people have designed it to be slow, quite the contrary.) But where's the fully distributed web platform for such an architecture? (If you have an answer, please make this question non-rhetorical. I'm putting a lot of time into building one because I couldn't find one.) An interesting thing about the confederated approach is, because each machine is identical, it could be built in a monolithic fashion. Thus super optimized for its purpose. I'm using a sorta cogs approach because there are many good erlang cogs to use in my project. But I think the big mistake is to focus on single node performance these days. Servers are relatively cheap, and you need more than one anyway for redundancy, so might as well have a cluster and no single points of failure.

nirvana 15y ago
For the life of me, I can't edit the above post. 5 attempts, all eaten by the HN monster. When talking about benchmarks, I meant to say: "and probably loses to nodeDB in every single one-machine benchmark you can come up with (not that the Basho people have designed it to be slow, quite the contrary.)" Finally the conclusion of the post: Further, the only inter-machine communication should be at the database layer. (very little needed elsewhere) so as a result: 1. Confederation of nodes that talk to each other to keep the databases consistent. With RIAK this means that one update only affects three nodes (with n=3). 2. The whole layer cake, DNS to Database, including web server, application services, queues, etc, is present on every machine. No single point of failure (if you use DNS round robin to spread load, otherwise your load balancer is the SPF). 3. Further, this layer cake, while composed of erlang cogs is mostly monolithic, as far as the programmer is concerned. You write your code in javascript or coffeescript and push it into the database. Each node is like a google App Engine, and you write handlers, and those handlers are run concurrently across the system. 4. Want more performance? Add servers, or upgrade some of your servers. Node went down in the middle of the night? Go back to sleep. (Or write a simple client test you can run from your iPhone to make sure there wasn't some sort of cascading failure and the service is still up.) 5. Almost zero operations, zero cluster architecture engineering work. No worrying about EBS being terribly slow (go dedicated machines for these nodes, btw) or any of that hassles. The engineer just writes javascript handlers and worries about CSS and javascript for the browser, etc. For me, that idea is nirvana. Nirvana may not be right for you... but confederation has to be the future.
- NyxWulf 15y ago
  The systems I work on are pretty sizable in scale, right now we are doing about 120 million calls a day on one of core clusters. I don't claim to be the end all authority on scaling, but I do have some observations based on what I've dealt with. First off, that type of architecture is brittle to logic changes. If you have a completely static architecture that you don't need to change it may be fine, but deploying those changes to every machine in the cluster is problematic. Second, not all components have the same underlying machine requirements. For instance, our nginx servers don't need much ram, but the HAProxy load balancers with nginx that terminate SSL need a lot of Ram and a good chunk of CPU. Hadoop works better with JBOD (Just a bunch of disks), whereas cassandra seems to work better with a raid 0 configuration. Certain layers like the Nginx through certain paths have real time requirements which means ultra low latency. Other things need to operate against massive data sets and compute answers in a few hours. So, not every machine in your architecture can have all of the services required by every other part of your architecture. A lot of it depends on workload types and what the underlying requirements are for your system. There are many more reasons, but I'll leave it at those for now. Ultimately the post is right that a single machine can work very well, but it's also misguided in just dismissing the HN commenter. There are many other cases where distributed architectures are required. Guaranteeing robustness and performance in the face of service and machine failures is very difficult, and is essentially impossible on a single machine. Which approach you apply depends, it depends on the unique situation and the unique constraints you have. Using a single model to solve all problems seems to be worse than using no model. Learn multiple models, learn how and when to apply each (yes - for those of you in the class, I'm taking the model thinking class :) ).

by nirvana 15y ago

nirvana 15y ago

For the life of me, I can't edit the above post. 5 attempts, all eaten by the HN monster. When talking about benchmarks, I meant to say: "and probably loses to nodeDB in every single one-machine benchmark you can come up with (not that the Basho people have designed it to be slow, quite the contrary.)" Finally the conclusion of the post: Further, the only inter-machine communication should be at the database layer. (very little needed elsewhere) so as a result: 1. Confederation of nodes that talk to each other to keep the databases consistent. With RIAK this means that one update only affects three nodes (with n=3). 2. The whole layer cake, DNS to Database, including web server, application services, queues, etc, is present on every machine. No single point of failure (if you use DNS round robin to spread load, otherwise your load balancer is the SPF). 3. Further, this layer cake, while composed of erlang cogs is mostly monolithic, as far as the programmer is concerned. You write your code in javascript or coffeescript and push it into the database. Each node is like a google App Engine, and you write handlers, and those handlers are run concurrently across the system. 4. Want more performance? Add servers, or upgrade some of your servers. Node went down in the middle of the night? Go back to sleep. (Or write a simple client test you can run from your iPhone to make sure there wasn't some sort of cascading failure and the service is still up.) 5. Almost zero operations, zero cluster architecture engineering work. No worrying about EBS being terribly slow (go dedicated machines for these nodes, btw) or any of that hassles. The engineer just writes javascript handlers and worries about CSS and javascript for the browser, etc. For me, that idea is nirvana. Nirvana may not be right for you... but confederation has to be the future.

NyxWulf 15y ago
The systems I work on are pretty sizable in scale, right now we are doing about 120 million calls a day on one of core clusters. I don't claim to be the end all authority on scaling, but I do have some observations based on what I've dealt with. First off, that type of architecture is brittle to logic changes. If you have a completely static architecture that you don't need to change it may be fine, but deploying those changes to every machine in the cluster is problematic. Second, not all components have the same underlying machine requirements. For instance, our nginx servers don't need much ram, but the HAProxy load balancers with nginx that terminate SSL need a lot of Ram and a good chunk of CPU. Hadoop works better with JBOD (Just a bunch of disks), whereas cassandra seems to work better with a raid 0 configuration. Certain layers like the Nginx through certain paths have real time requirements which means ultra low latency. Other things need to operate against massive data sets and compute answers in a few hours. So, not every machine in your architecture can have all of the services required by every other part of your architecture. A lot of it depends on workload types and what the underlying requirements are for your system. There are many more reasons, but I'll leave it at those for now. Ultimately the post is right that a single machine can work very well, but it's also misguided in just dismissing the HN commenter. There are many other cases where distributed architectures are required. Guaranteeing robustness and performance in the face of service and machine failures is very difficult, and is essentially impossible on a single machine. Which approach you apply depends, it depends on the unique situation and the unique constraints you have. Using a single model to solve all problems seems to be worse than using no model. Learn multiple models, learn how and when to apply each (yes - for those of you in the class, I'm taking the model thinking class :) ).