3 ms·

Could you elaborate on what of ES becomes so costly? I see it alludes to it in the article but I'm in the dark honestly. Is it the storage of the data since it

Could you elaborate on what of ES becomes so costly? I see it alludes to it in the article but I'm in the dark honestly. Is it the storage of the data since it mentions S3? Is it memory bound or CPU bound?

whalesalad 7y ago
In most situations it’s memory bound. In hyper cost savings situations it comes down to the fact that you need to run 3+ nodes at the very least in order to have some guarantees about data resilience. In reality you usually want more than 3 and you even want multiple clusters of more than 3 nodes each. So it can become costly. Most people just run one node. It works fine until the node falls over and comes back with zero indexes.
- outworlder 7y ago
  > Most people just run one node. That's completely unacceptable for any non-toy workloads. If you want a single node, deploy something like PG. You aren't using any of the "Elastic" capabilities, so why bother? If the volume of data is that small, it's better to run 3 small nodes than a single, larger one.
outworlder 7y ago
Not the parent but I'll pitch in. For any non-trivial cluster, you'll want 30GB of RAM for Elasticsearch(no more), plus whatever you can spare as OS cache. 64GB is common. Plus a lot of storage, preferably on fast disks. CPU consumption really depends on what you are doing but both ingestion/indexing and searches are costly. Ingestion is mostly I/O bound, but still requires a surprising amount of CPU. Since we are talking about hot-warm, retention is the problem. You'll need enough nodes with capacity to hold all the required data. Every open index uses some amount of memory for each shard. A shard should ideally not exceed 50GB, so if you have big indices you'll have lots of shards (or you have lots of small indices but still lots of shards). You could have really enormous disks, but you shouldn't. If there is a problem with any given node (and at scale, there will be), now your cluster will want to move ridiculous amounts of data if it detects a node failure. You can mitigate this, but you don't really want to turn of shard relocation. Now multiply the storage requirements by two, as I assume you'll be using at least one replica per shard. So now you have a bunch of nodes doing ingestion, and a bunch of other nodes storing data and answering search queries (in many clusters, they are the same, but you can specialize). It's not uncommon to end up with hundreds of nodes. At this scale, network communication becomes a bottleneck too, as the cluster is trying to send enormous state updates. And now masters will have trouble keeping up, so you'll end up scaling them as well. It's a snowball. Small ES clusters are trivial to manage, large ones are a full time job, even with automation. If this solution can help offload data from the primary ingest or hot cluster, it will be a godsend.
- __blockcipher__ 7y ago
  Lots of small indices is an anti-pattern due to performance overhead. I’ve seen such a setup wreck cluster stability. By sticking to shards that are 10-100GB, and only using i3 instances, you can get great performance. My company ingests about 2 TB per day and we use less than 10 i3 instances.

whalesalad 7y ago

In most situations it’s memory bound. In hyper cost savings situations it comes down to the fact that you need to run 3+ nodes at the very least in order to have some guarantees about data resilience. In reality you usually want more than 3 and you even want multiple clusters of more than 3 nodes each. So it can become costly. Most people just run one node. It works fine until the node falls over and comes back with zero indexes.

outworlder 7y ago
> Most people just run one node. That's completely unacceptable for any non-toy workloads. If you want a single node, deploy something like PG. You aren't using any of the "Elastic" capabilities, so why bother? If the volume of data is that small, it's better to run 3 small nodes than a single, larger one.

outworlder 7y ago

Not the parent but I'll pitch in. For any non-trivial cluster, you'll want 30GB of RAM for Elasticsearch(no more), plus whatever you can spare as OS cache. 64GB is common. Plus a lot of storage, preferably on fast disks. CPU consumption really depends on what you are doing but both ingestion/indexing and searches are costly. Ingestion is mostly I/O bound, but still requires a surprising amount of CPU. Since we are talking about hot-warm, retention is the problem. You'll need enough nodes with capacity to hold all the required data. Every open index uses some amount of memory for each shard. A shard should ideally not exceed 50GB, so if you have big indices you'll have lots of shards (or you have lots of small indices but still lots of shards). You could have really enormous disks, but you shouldn't. If there is a problem with any given node (and at scale, there will be), now your cluster will want to move ridiculous amounts of data if it detects a node failure. You can mitigate this, but you don't really want to turn of shard relocation. Now multiply the storage requirements by two, as I assume you'll be using at least one replica per shard. So now you have a bunch of nodes doing ingestion, and a bunch of other nodes storing data and answering search queries (in many clusters, they are the same, but you can specialize). It's not uncommon to end up with hundreds of nodes. At this scale, network communication becomes a bottleneck too, as the cluster is trying to send enormous state updates. And now masters will have trouble keeping up, so you'll end up scaling them as well. It's a snowball. Small ES clusters are trivial to manage, large ones are a full time job, even with automation. If this solution can help offload data from the primary ingest or hot cluster, it will be a godsend.

__blockcipher__ 7y ago
Lots of small indices is an anti-pattern due to performance overhead. I’ve seen such a setup wreck cluster stability. By sticking to shards that are 10-100GB, and only using i3 instances, you can get great performance. My company ingests about 2 TB per day and we use less than 10 i3 instances.