4 ms·

Many of the people commenting don't realize that for the majority of ES users, it is a pain in the butt to manage and scale their cluster storage in lock step w

Many of the people commenting don't realize that for the majority of ES users, it is a pain in the butt to manage and scale their cluster storage in lock step with their compute. Most companies use ES for relatively small scale search with people who are not experts in managing an ES cluster, or really infra at all. A cloud offering that decouples storage from compute makes this a lot easier and becomes more of a no brainer for 90% of the use cases. If you're an outlier with TBs or PBs of search data this probably you can keep using on prem if you want. Though I don't really immediately grant that it's worse for your usecase, especially when object storage is insanely cheap and supposing they provide a compat layer for S3 you can get away with everything from R2, S3, Ceph, Minio, Backblaze, etc. This is very much a case you should benchmark/analyze as a proper engineer. I've managed many TB ES clusters. This would have reduced my cost and time to manage it so I could focus on other features that would have benefitted my users.

jrochkind1 4y ago
I'm not familiar with this space, but I would have thoguht existing managed cloud offerings already decoupled storage from compute as far as the customer is concerned. But not true, or not as much true as it could be with new architecture?
- ianbutler 4y ago
  Not as much true as it could be, with the offerings I'm aware of (AWS OpenSearch, Elasti, Co's own cloud), and having used OpenSearch more extensively, you're still basically fully responsible for managing the cluster topology and the "data" node type is what handles storage and the important compute for search. If you need more storage you'll have to bump the EBS volume sizes on a new cluster and then do replication, or add a new node to the existing cluster and rebalance your data which are both very expensive operations. If you need more compute you're stuck with adding more storage regardless, but you can at least limit how much. In the case of both services, they make the processes easier but you still have time and expense considerations to them, if only just waiting time and not active work on your part. But it also means you have a lot more monitoring concerns you have to implement and then account for. Plus your choice in storage IOPS greatly effects query time so you also have to consider that and high IOPS storage is expensive. It's a lot of things for someone who just wants a workable search for their product to have to know. The new architecture they're proposing seems to allow for much more seamless scaling as your data grows without a bunch of manual intervention, monitoring and infra knowledge to make sure things don't fall over on you.
  - darkwater 4y ago
    > Not as much true as it could be, with the offerings I'm aware of (AWS OpenSearch, Elasti, Co's own cloud), and having used OpenSearch more extensively, you're still basically fully responsible for managing the cluster topology and the "data" node type is what handles storage and the important compute for search. If you need more storage you'll have to bump the EBS volume sizes on a new cluster and then do replication, or add a new node to the existing cluster and rebalance your data which are both very expensive operations. If you need more compute you're stuck with adding more storage regardless, but you can at least limit how much. This, absolutely. I'll add that I once tried to look at the Elasti.co offer for an ELK solution that could handle the logs we are currently managing internally (on EC2 managed by us with some automation), and you basically had to specify all the details of the topology to get a quote. It was basically a layer on top of AWS Cost Calculator with their margin baked in. No way you can tell them "I send on average X bytes/day, I want a quick answer for Y days and a slower answer for Z days and a global retention of XY days" and get a "you will pay aprox NNN dollars/month in the HA config, MMM in the non-HA config"

jrochkind1 4y ago

I'm not familiar with this space, but I would have thoguht existing managed cloud offerings already decoupled storage from compute as far as the customer is concerned. But not true, or not as much true as it could be with new architecture?

ianbutler 4y ago
Not as much true as it could be, with the offerings I'm aware of (AWS OpenSearch, Elasti, Co's own cloud), and having used OpenSearch more extensively, you're still basically fully responsible for managing the cluster topology and the "data" node type is what handles storage and the important compute for search. If you need more storage you'll have to bump the EBS volume sizes on a new cluster and then do replication, or add a new node to the existing cluster and rebalance your data which are both very expensive operations. If you need more compute you're stuck with adding more storage regardless, but you can at least limit how much. In the case of both services, they make the processes easier but you still have time and expense considerations to them, if only just waiting time and not active work on your part. But it also means you have a lot more monitoring concerns you have to implement and then account for. Plus your choice in storage IOPS greatly effects query time so you also have to consider that and high IOPS storage is expensive. It's a lot of things for someone who just wants a workable search for their product to have to know. The new architecture they're proposing seems to allow for much more seamless scaling as your data grows without a bunch of manual intervention, monitoring and infra knowledge to make sure things don't fall over on you.
- darkwater 4y ago
  > Not as much true as it could be, with the offerings I'm aware of (AWS OpenSearch, Elasti, Co's own cloud), and having used OpenSearch more extensively, you're still basically fully responsible for managing the cluster topology and the "data" node type is what handles storage and the important compute for search. If you need more storage you'll have to bump the EBS volume sizes on a new cluster and then do replication, or add a new node to the existing cluster and rebalance your data which are both very expensive operations. If you need more compute you're stuck with adding more storage regardless, but you can at least limit how much. This, absolutely. I'll add that I once tried to look at the Elasti.co offer for an ELK solution that could handle the logs we are currently managing internally (on EC2 managed by us with some automation), and you basically had to specify all the details of the topology to get a quote. It was basically a layer on top of AWS Cost Calculator with their margin baked in. No way you can tell them "I send on average X bytes/day, I want a quick answer for Y days and a slower answer for Z days and a global retention of XY days" and get a "you will pay aprox NNN dollars/month in the HA config, MMM in the non-HA config"