karlney

6 ms·

1.
by karlney 4y ago
Shameless plug, but one of the other blog posts in the above series explain how we addressed high and unpredictable memory load in relation to wildcard searches. https://underthehood.meltwater.com/blog/2022/11&#x2F
2.
by karlney 4y ago
This is a good resource for the up to date resilience status in Elasticsearch https://www.elastic.co/guide/en/elasticsearch/resiliency/cur... They have gotten A LOT better since the 1.x days
3.
by karlney 4y ago
We feel that ~300 nodes strikes a good balance in the cattle vs pets philosophy. Going up to i3en.12xlarge (or equivalent) would probably have worked as well. But after that the impact of loosing just one node would be too big.
4.
by karlney 4y ago
One time, a few years ago a particularly nasty query was executed over and over again and it took a few hours to find it and then block it. And during that time so many nodes had became slow and unresponsive that another (for us) previously
5.
by karlney 4y ago
Thank you for those kind words. And yes we have had our fair share of pain with the old cluster for sure. the new version (7.17) is still behaving a lot better so far and feels a lot more predictable.
6.
by karlney 4y ago
Thank you. Yes it was a massive project. I don't want to spoil the other blog posts but we managed to solve almost all of our custom use cases without modifying elasticsearch itself. We still have one custom plugin but only to enhance
7.
by karlney 7y ago
Thanks for this info! Doing a quick googling on elasticsearch and ZGC I just found this https://github.com/elastic/elasticsearch/issues/44321 where the official response from elastic is that only G1GC and CMS
8.
by karlney 7y ago
Hi, do you have real use experience running elasticsearch with 64g+ heap? Is there any articles/benchmark/notes or anything that you would be willing to share? We have considered trying out 64g+ heaps for our cluster but we are co
9.
by karlney 8y ago
The social data are fairly small documents, but we index lots of other things as well that is larger in size. Another factor is our large replica count, especially for the hot data. We also have quite a lot of metadata.
10.
by karlney 8y ago
We use datadog for monitoring our cluster. Shardonnay also produces lots of custom metrics that you don't get from the vanilla elasticsearch datadog integration.
11.
by karlney 8y ago
We tried the hot/cold architecture as well, and used it before in our data centre architecture. But currently we have concluded that it is not worth the added complexity. But that might change again if/when we learn more or get ne
12.
by karlney 8y ago
Hi. We are actually using an even older version, 1.7.6 for our cluster. This is off not an ideal set-up and we will upgrade as soon as we can. That is not an easy task in a cluster of this size and given our uptime requirements. We know tha