2 ms·
Elasticsearch has had support for hot/warm, and now with 6.7+, for hot/warm/cold via Index Life Management [0], where you'd typically run hot nodes with NVMe or
by ignoramous 7y ago
Elasticsearch has had support for hot/warm, and now with 6.7+, for hot/warm/cold via Index Life Management [0], where you'd typically run hot nodes with NVMe or SSDs, warm nodes with HDDs, and cold nodes with network attached storage.
AWS' key innovation seems to be that hot nodes are the ones you provision to keep both primary and secondary indices for hot data (writable and recent), and ultra-warm nodes, of different ec2-instance-type than the hot nodes, prefetch speculatively / predictively caches of index-data downloaded on-demand from S3 required to execute read-only requests transparently to the rest of the cluster.
Impressive: It is almost like running ultrawarm nodes with Amazon FSx for Lustre instead of EBS.
I wonder if the Lucene / Elasticsearch changes required to do this will make it to OpenDistro. If not, that'd be tremendously disappointing [1].
---
[0] https://www.elastic.co/blog/implementing-hot-warm-cold-in-elasticsearch-with-index-lifecycle-management https://www.elastic.co/blog/implementing-hot-warm-cold-in-el...
[1] In addition, the innovation focus has shifted from furthering the open source distribution to making the proprietary distribution popular. This means that the majority of new Elasticsearch users are now, in fact, running proprietary software. https://news.ycombinator.com/item?id=19359602 https://news.ycombinator.com/item?id=19359602