4 ms·

Pretty interesting stuff. I'm amazed that there are only 14 nodes in the cluster with 1TB SSD each; message size must be fairly small on disk (less than 295 byt

Pretty interesting stuff. I'm amazed that there are only 14 nodes in the cluster with 1TB SSD each; message size must be fairly small on disk (less than 295 bytes, unreplicated). I'd also recommend looking at maybe tuning the shard_request_cache; there are some possible improvements to be made there, if you're running that many indices. Finally, are all of the indices of approximately the same size? Are they time-boxed? edit: Put together a logging cluster consisting of 14 nodes, 12 data + 2 indexer/search API, with ~40TB of consumer-grade SSDs. Ran ~14k indices based on log type & timestamp, with a whole raft of custom field configuration to handle aggregations and different tokenizations. In sum, Elasticsearch looks easy to configure and tune but is amazingly hard to do well - but incredibly rewarding.

searchfaster 10y ago
Cool.. how many records per shard do you use for optimal performance ?
- packetized 10y ago
  In the cluster I previously built/worked on, somewhere between 4mil & 9mil records/shard - this also assumed 3 shards per index, with only one replica set (total of 6 shards).
notyourwork 10y ago
> but is amazingly hard to do well - but incredibly rewarding. Isn't this true of everything. The greater the reward the more effort required in most scenarios? I do agree, Elasticsearch can take some trial and error along with experimentation to get right!
- packetized 10y ago
  Yeah, that's nearly a tautology. I guess in this sense what I mean is that, there's a much deeper Trough of Disillusionment when it comes to learning how to properly configure & tune ES to your needs, as opposed to a lot of other tools I've used over the years. It's very easy to set up, but also very hard to get right, which I'd imagine causes some people to settle or give up.
stickperson 10y ago
The small message results from only storing message, channel, and server ids as opposed to the entire message. How efficient is ES at storing the rest of the message data in the inverted index? No clue, but apparently pretty good.
j_s 10y ago
Sounds like you need to start a freelance Elastic Search optimization business!

searchfaster 10y ago

Cool.. how many records per shard do you use for optimal performance ?

packetized 10y ago
In the cluster I previously built/worked on, somewhere between 4mil & 9mil records/shard - this also assumed 3 shards per index, with only one replica set (total of 6 shards).

notyourwork 10y ago

> but is amazingly hard to do well - but incredibly rewarding. Isn't this true of everything. The greater the reward the more effort required in most scenarios? I do agree, Elasticsearch can take some trial and error along with experimentation to get right!

packetized 10y ago
Yeah, that's nearly a tautology. I guess in this sense what I mean is that, there's a much deeper Trough of Disillusionment when it comes to learning how to properly configure & tune ES to your needs, as opposed to a lot of other tools I've used over the years. It's very easy to set up, but also very hard to get right, which I'd imagine causes some people to settle or give up.

stickperson 10y ago

The small message results from only storing message, channel, and server ids as opposed to the entire message. How efficient is ES at storing the rest of the message data in the inverted index? No clue, but apparently pretty good.

j_s 10y ago

Sounds like you need to start a freelance Elastic Search optimization business!