5 ms·

So they will be forcing my few dozen terabyte ES cluster that today runs fully on dedicated physical hardware to use extremely expensive cloud storage services

by jgb1984 4y ago

So they will be forcing my few dozen terabyte ES cluster that today runs fully on dedicated physical hardware to use extremely expensive cloud storage services instead? What an awful idea! I hope the option to fully self-host on dedicated hardware remains viable, as I enjoy the high performance, low price and full control of my own systems.

danw1979 4y ago
Fingers crossed they support any S3-compatible local object storage system (e.g. Ceph) then all you need to do is completely change your storage system and probably duplicate it’s capacity during migration. If this is the future of ES I don’t see a happy path for non-cloud customers…
- _jezell_ 4y ago
  minio
  - More-nitors 4y ago
    ...which means another layer that eats up performance?
- robertlagrant 4y ago
  > Fingers crossed they support any S3-compatible local object storage system Do you mean they "only support"?
ignoramous 4y ago
I believe it would be cheaper to run stateless (given the reduced overhead of devops)? It is an open secret that the most expensive commodity in a datacenter (servers) is also the most under utilised [0]. Hence, folks like me prefer to pay per-query cost. Elastic had to do this in face of competition from the likes of quickwit.io and snowflake search. AWS might have something similar up their sleeve, because I don't believe Athena/Trino can do super fast searches, yet. [0] https://youtube.com/watch?v=dInADzgCI-s&t=538 https://youtube.com/watch?v=dInADzgCI-s&t=538
jillesvangurp 4y ago
You can easily emulate object storage with e.g. s3 compatible APIs. And probably it will just support whatever file system, including nfs (currently not supported). So, that should not be a show stopper. And technically this could actually reduce your cost by a lot because you will be able to use some central network storage instead of having to have a lot of high end SSDs. You might still want to use those for caching of course. But getting rid of some of the data replication probably might actually help lower your cost. As Elasticsearch is closed source, you can also choose to switch to Opensearch which will of course not get any of these changes. Though I would not be surprised to see this move mirrored on their side as it makes a lot of sense to do this. But it would end up being an independent implementation of the same concept. As a long time Elasticsearch user, this stateless architecture makes a lot of sense to me. Especially for very large clusters. Basically, it vastly simplifies scaling and cluster operations. You can literally auto scale nodes both on the indexing and querying tiers. That's a big deal. You want it faster? Add more nodes. Likewise upgrades are a lot easier. Simply bring new nodes online and re-index to some new objects. Once it is done, take down the old ones. Also it simplifies testing. You can simply bring up a few test nodes and query your production data without having to worry about affecting production loads. Everything gets easier. It will be interesting to see how they will bring this to market. This reads like they are starting the work on this, not like they are ready to deploy this. I guess this would be part of a future major release and they just had their previous one fairly recently.
- petabite 4y ago
  How is Elasticsearch "closed source"? The code is literally right here: https://github.com/elastic/elasticsearch https://github.com/elastic/elasticsearch
  - Hamcha 4y ago
    Probably meant "Source available" since neither Elasticsearch code licenses are "Open source" (going by OSI approval).
    - petabite 4y ago
deleted 4y ago
[deleted]
cavisne 4y ago
What they are really doing is offloading the job of persisting data from ES. So you wouldn't need to use S3 or some s3 emulator like other comments have suggested. Give the ES cluster a disk it can write to that is guaranteed to be replicated (ie a RAID cluster) and thats it. The limits of object storage (no appending, no in place editing) tend to make those clusters cheaper so you could also run something like Ceph locally. It's probably going to do a better job of persisting data than ES also.

danw1979 4y ago

Fingers crossed they support any S3-compatible local object storage system (e.g. Ceph) then all you need to do is completely change your storage system and probably duplicate it’s capacity during migration. If this is the future of ES I don’t see a happy path for non-cloud customers…

_jezell_ 4y ago
minio
- More-nitors 4y ago
  ...which means another layer that eats up performance?
robertlagrant 4y ago
> Fingers crossed they support any S3-compatible local object storage system Do you mean they "only support"?

ignoramous 4y ago

I believe it would be cheaper to run stateless (given the reduced overhead of devops)? It is an open secret that the most expensive commodity in a datacenter (servers) is also the most under utilised [0]. Hence, folks like me prefer to pay per-query cost. Elastic had to do this in face of competition from the likes of quickwit.io and snowflake search. AWS might have something similar up their sleeve, because I don't believe Athena/Trino can do super fast searches, yet. [0] https://youtube.com/watch?v=dInADzgCI-s&t=538 https://youtube.com/watch?v=dInADzgCI-s&t=538

jillesvangurp 4y ago

You can easily emulate object storage with e.g. s3 compatible APIs. And probably it will just support whatever file system, including nfs (currently not supported). So, that should not be a show stopper. And technically this could actually reduce your cost by a lot because you will be able to use some central network storage instead of having to have a lot of high end SSDs. You might still want to use those for caching of course. But getting rid of some of the data replication probably might actually help lower your cost. As Elasticsearch is closed source, you can also choose to switch to Opensearch which will of course not get any of these changes. Though I would not be surprised to see this move mirrored on their side as it makes a lot of sense to do this. But it would end up being an independent implementation of the same concept. As a long time Elasticsearch user, this stateless architecture makes a lot of sense to me. Especially for very large clusters. Basically, it vastly simplifies scaling and cluster operations. You can literally auto scale nodes both on the indexing and querying tiers. That's a big deal. You want it faster? Add more nodes. Likewise upgrades are a lot easier. Simply bring new nodes online and re-index to some new objects. Once it is done, take down the old ones. Also it simplifies testing. You can simply bring up a few test nodes and query your production data without having to worry about affecting production loads. Everything gets easier. It will be interesting to see how they will bring this to market. This reads like they are starting the work on this, not like they are ready to deploy this. I guess this would be part of a future major release and they just had their previous one fairly recently.

petabite 4y ago
How is Elasticsearch "closed source"? The code is literally right here: https://github.com/elastic/elasticsearch https://github.com/elastic/elasticsearch
- Hamcha 4y ago
  Probably meant "Source available" since neither Elasticsearch code licenses are "Open source" (going by OSI approval).
  - petabite 4y ago

deleted 4y ago

[deleted]

cavisne 4y ago

What they are really doing is offloading the job of persisting data from ES. So you wouldn't need to use S3 or some s3 emulator like other comments have suggested. Give the ES cluster a disk it can write to that is guaranteed to be replicated (ie a RAID cluster) and thats it. The limits of object storage (no appending, no in place editing) tend to make those clusters cheaper so you could also run something like Ceph locally. It's probably going to do a better job of persisting data than ES also.