3 ms·
Not the parent but I'll pitch in. For any non-trivial cluster, you'll want 30GB of RAM for Elasticsearch(no more), plus whatever you can spare as OS cache. 64G
by outworlder 7y ago
Not the parent but I'll pitch in.
For any non-trivial cluster, you'll want 30GB of RAM for Elasticsearch(no more), plus whatever you can spare as OS cache. 64GB is common. Plus a lot of storage, preferably on fast disks.
CPU consumption really depends on what you are doing but both ingestion/indexing and searches are costly. Ingestion is mostly I/O bound, but still requires a surprising amount of CPU.
Since we are talking about hot-warm, retention is the problem. You'll need enough nodes with capacity to hold all the required data. Every open index uses some amount of memory for each shard. A shard should ideally not exceed 50GB, so if you have big indices you'll have lots of shards (or you have lots of small indices but still lots of shards).
You could have really enormous disks, but you shouldn't. If there is a problem with any given node (and at scale, there will be), now your cluster will want to move ridiculous amounts of data if it detects a node failure. You can mitigate this, but you don't really want to turn of shard relocation.
Now multiply the storage requirements by two, as I assume you'll be using at least one replica per shard.
So now you have a bunch of nodes doing ingestion, and a bunch of other nodes storing data and answering search queries (in many clusters, they are the same, but you can specialize). It's not uncommon to end up with hundreds of nodes. At this scale, network communication becomes a bottleneck too, as the cluster is trying to send enormous state updates. And now masters will have trouble keeping up, so you'll end up scaling them as well.
It's a snowball. Small ES clusters are trivial to manage, large ones are a full time job, even with automation. If this solution can help offload data from the primary ingest or hot cluster, it will be a godsend.
- __blockcipher__ 7y agoLots of small indices is an anti-pattern due to performance overhead. I’ve seen such a setup wreck cluster stability. By sticking to shards that are 10-100GB, and only using i3 instances, you can get great performance. My company ingests about 2 TB per day and we use less than 10 i3 instances.