2 ms·

Not as much true as it could be, with the offerings I'm aware of (AWS OpenSearch, Elasti, Co's own cloud), and having used OpenSearch more extensively, you're s

Not as much true as it could be, with the offerings I'm aware of (AWS OpenSearch, Elasti, Co's own cloud), and having used OpenSearch more extensively, you're still basically fully responsible for managing the cluster topology and the "data" node type is what handles storage and the important compute for search. If you need more storage you'll have to bump the EBS volume sizes on a new cluster and then do replication, or add a new node to the existing cluster and rebalance your data which are both very expensive operations. If you need more compute you're stuck with adding more storage regardless, but you can at least limit how much. In the case of both services, they make the processes easier but you still have time and expense considerations to them, if only just waiting time and not active work on your part. But it also means you have a lot more monitoring concerns you have to implement and then account for. Plus your choice in storage IOPS greatly effects query time so you also have to consider that and high IOPS storage is expensive. It's a lot of things for someone who just wants a workable search for their product to have to know. The new architecture they're proposing seems to allow for much more seamless scaling as your data grows without a bunch of manual intervention, monitoring and infra knowledge to make sure things don't fall over on you.

darkwater 4y ago
> Not as much true as it could be, with the offerings I'm aware of (AWS OpenSearch, Elasti, Co's own cloud), and having used OpenSearch more extensively, you're still basically fully responsible for managing the cluster topology and the "data" node type is what handles storage and the important compute for search. If you need more storage you'll have to bump the EBS volume sizes on a new cluster and then do replication, or add a new node to the existing cluster and rebalance your data which are both very expensive operations. If you need more compute you're stuck with adding more storage regardless, but you can at least limit how much. This, absolutely. I'll add that I once tried to look at the Elasti.co offer for an ELK solution that could handle the logs we are currently managing internally (on EC2 managed by us with some automation), and you basically had to specify all the details of the topology to get a quote. It was basically a layer on top of AWS Cost Calculator with their margin baked in. No way you can tell them "I send on average X bytes/day, I want a quick answer for Y days and a slower answer for Z days and a global retention of XY days" and get a "you will pay aprox NNN dollars/month in the HA config, MMM in the non-HA config"

darkwater 4y ago

> Not as much true as it could be, with the offerings I'm aware of (AWS OpenSearch, Elasti, Co's own cloud), and having used OpenSearch more extensively, you're still basically fully responsible for managing the cluster topology and the "data" node type is what handles storage and the important compute for search. If you need more storage you'll have to bump the EBS volume sizes on a new cluster and then do replication, or add a new node to the existing cluster and rebalance your data which are both very expensive operations. If you need more compute you're stuck with adding more storage regardless, but you can at least limit how much. This, absolutely. I'll add that I once tried to look at the Elasti.co offer for an ELK solution that could handle the logs we are currently managing internally (on EC2 managed by us with some automation), and you basically had to specify all the details of the topology to get a quote. It was basically a layer on top of AWS Cost Calculator with their margin baked in. No way you can tell them "I send on average X bytes/day, I want a quick answer for Y days and a slower answer for Z days and a global retention of XY days" and get a "you will pay aprox NNN dollars/month in the HA config, MMM in the non-HA config"