> That's true in theory. In practice I've never seen anyone max out the amount of RAM possible in off the shelf x86 servers. I'm currently experimenting with a

> That's true in theory. In practice I've never seen anyone max out the amount of RAM possible in off the shelf x86 servers. I'm currently experimenting with a 4 node cluster running Apache Drill (on top of MapR's Hadoop distro), with 500GB RAM, 96 cores, and 28TB of SSD. For some analytic queries that span multiple date partitions, its actually easy to see 70% RAM capacity being utilized, and 80% of the CPU. The size of the data set is currently about 7TB, but that replicates for both data locality and redundancy. We are a Postgres shop, and have tested with some of the options that run a "single node cluster" but one of the limiting factors is that RAM is divided between those DB processes, and performance is impacted. Plus we then need to implement some form of replication to obtain redundancy, fault tolerance, or HA. Quite simply we are not using a cluster because our data won't fit, but because we can get better performance and reliability on 4 nodes. We likely won't have the need for 400. There are cases where for non-interactive data processing we definitely don't need a cluster. At its simplest, multiple copies of Awk scripts can be run using GNU Parallel, Python scripts can be parallelized with the Multiprocessing module, and Go has been useful for concurrency (especially to us Python programmers).