3 ms·
Congrats on finishing that monster migration! > In order to control how queries are executed, we have built a plugin which exposes a set of custom query types.
by andrelaszlo 4y ago
Congrats on finishing that monster migration!
> In order to control how queries are executed, we have built a plugin which exposes a set of custom query types. We use these query types to provide functionality and performance optimisations not available in stock Elasticsearch. For example, we have implemented wildcards within phrases, with support for executing within SpanNear queries. We optimise “*” to a match-all-query. And a whole lot of other things.
Did you port your the in-house plugins? Seems like a big blocker.
- karlney 4y agoThank you. Yes it was a massive project. I don't want to spoil the other blog posts but we managed to solve almost all of our custom use cases without modifying elasticsearch itself. We still have one custom plugin but only to enhance functionality, not for performance and stability reasons.
- andrelaszlo 4y agoCool! Will stay tuned for the next post :)
- semi-extrinsic 4y agoWhile I fully understand why you run this thing with 300+ nodes as you do, I have to wonder, just for fun - could you actually fit this whole thing on a single large server? Looks like something with 16 TiB RAM and 2 PiB SSD storage is actually a server you could theoretically buy today?
- karlney 4y agoWe feel that ~300 nodes strikes a good balance in the cattle vs pets philosophy. Going up to i3en.12xlarge (or equivalent) would probably have worked as well. But after that the impact of loosing just one node would be too big.