3 ms·

> Elasticsearch, definitely. I always recommend using it in hosted form and not running your own cluster. That allows you to focus on getting data in and out of

> Elasticsearch, definitely. I always recommend using it in hosted form and not running your own cluster. That allows you to focus on getting data in and out of your cluster instead of sinking time into doing devops. I think everyone doing Elasticsearch well has to bring it in-house eventually. AWS's hosted solution is poor, Logz.io and ElasticCloud are expensive. There's a 7-figure/yr Elastic Cloud customer I work with who is so tired of Elastic just randomly killing their clusters out of nowhere and having to spend basically triple to deal with it that they're bringing it all in house.

showerst 7y ago
I think it depends massively on how much data you have. A great many companies and websites only need a few hundred megs of text data indexed, which is easier to outsource. Once you grow larger than that though, the hosted service prices get astronomical compared to standing up a cluster, assuming you have someone who can admin it.
- devonkim 7y ago
  Things are not quite cut and dry operationally. Some places may have only a few hundred MB of data but they need really high availability and performance (probably picked the wrong solution IMO but still...) which is better guaranteed hosting it yourself and perhaps even outside the cloud. Most places the availability of whatever a hosted solution provides is a better value than spending engineering hours to maintain these things. Devops / SRE folks are expensive compared to most other engineers that deliver features primarily. My point is that a ton of companies (more for cultural reasons than business requirements IME) are so freaked out by any downtime in any way that they’re going to pay for an engineer to just maintain these things themselves.
  - busterarm 7y ago
    The only issue I have with this suggestion is that all of the situational awareness needed to _use_ Elasticsearch effectively is the same as is needed to _operate_ Elasticsearch effectively. If you're just spinning up something's defaults and throwing data in it, that bill is going to due eventually and it's probably going to be ugly.
amelius 7y ago
Also, some clients may prohibit uploading of data to third parties. In those cases you simply have no other choice than to run your own cluster.
ragle 7y ago
> AWS's hosted solution is poor What has been poor about it in your experience?
- boyter 7y ago
  Not OP but the list I had from a year ago, didn’t support all multi az (only 2), lack of admin features meant it broke and you could work out what happened, limited choice of instance type, the backup happened once a day regardless if you needed it and it had issues under load. The backup issue has been resolved I believe but the others... On AWS I’d suggest elastic on ECS and you can use the leftover compute on the cluster to run other applications effectively for free.
- __blockcipher__ 7y ago
  Not the GP, but you aren't allowed to touch settings like the shard recovery rate. If a configuration change (changing # of nodes, instance type, etc) goes wrong, your cluster indefinitely gets stuck in Processing due to a race condition. The only way to get unstuck is to file a ticket. The company I'm at doesn't pay for AWS support, so at one point we ended up completely tearing down our cluster and rebuilding a new one (via Terraform) after getting tired of waiting for the reps. (They advised us to cut off log flow to let the system get out of processing, which we did, but it didn't work because once it gets stuck in processing like that it's just completely stuck). It's difficult to troubleshoot issues - you can get some logs via Cloudwatch, but they're hard to search through and I'm not entirely positive everything shows up there. Amazon is always several releases behind Elasticsearch versions. -- Elastic.co's offering looks much better, just by reading their excellent comparison article: https://www.elastic.co/blog/hosted-elasticsearch-services-roundup-elastic-cloud-and-amazon-elasticsearch-service https://www.elastic.co/blog/hosted-elasticsearch-services-ro... (We haven't used Elastic.co but what they say makes sense and I imagine their service is much better) -- Once you hit a big enough scale - for us, we're pushing about 2TB a day of logs (that's before accounting for replication of course), it doesn't make sense to stay on Amazon's hosted service. I'm in the process of advocating for in-housing our Elasticsearch setup and just building on top of ec2. Elasticsearh seems like the perfect candidate for Kubernetes since rebalancing is automatic and the affinity rules are simple (every Elasticsearch instance needs its own node). Cluster autoscaling (i.e. node-level, not _horizontal_) just makes too much sense. Unfortunately I haven't gotten the go-ahead to take it inhouse, but I've been gunning for the project for some time now, so I'm hopeful I'll get the opportunity. -- BTW, totally unrelated but for anyone managing Elasticsearch, make sure you have your shard count tuned properly. When I came to this company, they had their data way oversharded with primary shards varying between hundreds of kb to a few gb; i.e. orders of magnitude difference. Switching to ~50 GB shards (done via simplifying the way we were indexing) massively improved performance. Also i3 instances > anything with EBS. [/ramble]
otisg 7y ago
> so tired of Elastic just randomly killing their clusters out of nowhere and having to spend basically triple to deal with it that they're bringing it all in house. That is because Elastic Cloud is not a fully managed Elasticsearch. People often don't get that with Elastic Cloud you are still responsible for your ES cluster. That's one of the differentiators that e.g. Sematext has (disclaimer: founder). > AWS's hosted solution is poor, Logz.io and ElasticCloud are expensive. Right. Have a look at https://sematext.com/logsene https://sematext.com/logsene pricing. People say Sematext compares favorably to Logz.io and Elastic Cloud.
- sandGorgon 7y ago
  Why are all ELK stack based logging solutions so much more expensive than custom rolled solutions like logdna, datadog, papertrail,etc. The per gb cost on the other ones start at 1.20 $/gb and goes till 2$/gb. While almost all hosted ELK solutions start off at 3$/gb. Im asking because i would very much like to adopt an ELK based hosted solution..but I'm not able to justify paying double. Is it that running+resource costs for ELK are so high that the extra charge needs to happen ?
  - busterarm 7y ago
    Datadog, etc are not cheap when you're paying per agent/per month to ship your logs in the first place. Which they want you to do, of course.
    - sandGorgon 7y ago
      The pricing is not per agent per month. The pricing for everyone is always either per million events or per gb. You do not need an agent. You can ship directly from syslog ( https://docs.datadoghq.com/integrations/rsyslog/?tab=datadogussite https://docs.datadoghq.com/integrations/rsyslog/?tab=datadog...)
  - manigandham 7y ago
    Yes it's mostly about resource costs. ES is a generic search that can handle logs but isn't focused on it, and indexing everything can be costly. There are major efficiencies gained in creating your own log-focused storage and access methods with object storage, columnar formats, and zone mapping, etc.
jillesvangurp 7y ago
Elastic Cloud is pretty reasonable IMHO. We have a logging cluster that costs us around 250 euro per month. Yes, running it ourselves would be cheaper but doing that in Amazon would eliminate most of the cost benefits. If you then consider devops cost needed to that, there is no difference. Most companies getting started with Elasticsearch should probably not do that on day 1. You can always start doing this later if there's some reason to. I've never had clusters being killed randomly but I did have a few self inflicted issues with cluster instability due to flooding the cluster with too much data and not having suitable data retention policies in place. With the recently added index life cycle management (an x-pack feature), this is easier to manage these days. If you are spending seven figures a year on elastic search, you clearly are not a beginning user and there's some cost savings that you might be able to realize by taking ownership of the problem of hosting it somewhere cheaper. For that, their recently open sourced kubernetes helm charts are worth a look. Those scripts take care of a lot of things and get you a self hosted version of Elastic Cloud. Amazon hosted clusters are indeed a bit bare-bones (as is their support for these clusters) and I would also not recommend them; you get more value for money by using Elastic Cloud.

showerst 7y ago

I think it depends massively on how much data you have. A great many companies and websites only need a few hundred megs of text data indexed, which is easier to outsource. Once you grow larger than that though, the hosted service prices get astronomical compared to standing up a cluster, assuming you have someone who can admin it.

devonkim 7y ago
Things are not quite cut and dry operationally. Some places may have only a few hundred MB of data but they need really high availability and performance (probably picked the wrong solution IMO but still...) which is better guaranteed hosting it yourself and perhaps even outside the cloud. Most places the availability of whatever a hosted solution provides is a better value than spending engineering hours to maintain these things. Devops / SRE folks are expensive compared to most other engineers that deliver features primarily. My point is that a ton of companies (more for cultural reasons than business requirements IME) are so freaked out by any downtime in any way that they’re going to pay for an engineer to just maintain these things themselves.
- busterarm 7y ago
  The only issue I have with this suggestion is that all of the situational awareness needed to _use_ Elasticsearch effectively is the same as is needed to _operate_ Elasticsearch effectively. If you're just spinning up something's defaults and throwing data in it, that bill is going to due eventually and it's probably going to be ugly.

amelius 7y ago

Also, some clients may prohibit uploading of data to third parties. In those cases you simply have no other choice than to run your own cluster.

ragle 7y ago

> AWS's hosted solution is poor What has been poor about it in your experience?

boyter 7y ago
Not OP but the list I had from a year ago, didn’t support all multi az (only 2), lack of admin features meant it broke and you could work out what happened, limited choice of instance type, the backup happened once a day regardless if you needed it and it had issues under load. The backup issue has been resolved I believe but the others... On AWS I’d suggest elastic on ECS and you can use the leftover compute on the cluster to run other applications effectively for free.
__blockcipher__ 7y ago
Not the GP, but you aren't allowed to touch settings like the shard recovery rate. If a configuration change (changing # of nodes, instance type, etc) goes wrong, your cluster indefinitely gets stuck in Processing due to a race condition. The only way to get unstuck is to file a ticket. The company I'm at doesn't pay for AWS support, so at one point we ended up completely tearing down our cluster and rebuilding a new one (via Terraform) after getting tired of waiting for the reps. (They advised us to cut off log flow to let the system get out of processing, which we did, but it didn't work because once it gets stuck in processing like that it's just completely stuck). It's difficult to troubleshoot issues - you can get some logs via Cloudwatch, but they're hard to search through and I'm not entirely positive everything shows up there. Amazon is always several releases behind Elasticsearch versions. -- Elastic.co's offering looks much better, just by reading their excellent comparison article: https://www.elastic.co/blog/hosted-elasticsearch-services-roundup-elastic-cloud-and-amazon-elasticsearch-service https://www.elastic.co/blog/hosted-elasticsearch-services-ro... (We haven't used Elastic.co but what they say makes sense and I imagine their service is much better) -- Once you hit a big enough scale - for us, we're pushing about 2TB a day of logs (that's before accounting for replication of course), it doesn't make sense to stay on Amazon's hosted service. I'm in the process of advocating for in-housing our Elasticsearch setup and just building on top of ec2. Elasticsearh seems like the perfect candidate for Kubernetes since rebalancing is automatic and the affinity rules are simple (every Elasticsearch instance needs its own node). Cluster autoscaling (i.e. node-level, not _horizontal_) just makes too much sense. Unfortunately I haven't gotten the go-ahead to take it inhouse, but I've been gunning for the project for some time now, so I'm hopeful I'll get the opportunity. -- BTW, totally unrelated but for anyone managing Elasticsearch, make sure you have your shard count tuned properly. When I came to this company, they had their data way oversharded with primary shards varying between hundreds of kb to a few gb; i.e. orders of magnitude difference. Switching to ~50 GB shards (done via simplifying the way we were indexing) massively improved performance. Also i3 instances > anything with EBS. [/ramble]

otisg 7y ago

> so tired of Elastic just randomly killing their clusters out of nowhere and having to spend basically triple to deal with it that they're bringing it all in house. That is because Elastic Cloud is not a fully managed Elasticsearch. People often don't get that with Elastic Cloud you are still responsible for your ES cluster. That's one of the differentiators that e.g. Sematext has (disclaimer: founder). > AWS's hosted solution is poor, Logz.io and ElasticCloud are expensive. Right. Have a look at https://sematext.com/logsene https://sematext.com/logsene pricing. People say Sematext compares favorably to Logz.io and Elastic Cloud.

sandGorgon 7y ago
Why are all ELK stack based logging solutions so much more expensive than custom rolled solutions like logdna, datadog, papertrail,etc. The per gb cost on the other ones start at 1.20 $/gb and goes till 2$/gb. While almost all hosted ELK solutions start off at 3$/gb. Im asking because i would very much like to adopt an ELK based hosted solution..but I'm not able to justify paying double. Is it that running+resource costs for ELK are so high that the extra charge needs to happen ?
- busterarm 7y ago
  Datadog, etc are not cheap when you're paying per agent/per month to ship your logs in the first place. Which they want you to do, of course.
  - sandGorgon 7y ago
    The pricing is not per agent per month. The pricing for everyone is always either per million events or per gb. You do not need an agent. You can ship directly from syslog ( https://docs.datadoghq.com/integrations/rsyslog/?tab=datadogussite https://docs.datadoghq.com/integrations/rsyslog/?tab=datadog...)
- manigandham 7y ago
  Yes it's mostly about resource costs. ES is a generic search that can handle logs but isn't focused on it, and indexing everything can be costly. There are major efficiencies gained in creating your own log-focused storage and access methods with object storage, columnar formats, and zone mapping, etc.

jillesvangurp 7y ago

Elastic Cloud is pretty reasonable IMHO. We have a logging cluster that costs us around 250 euro per month. Yes, running it ourselves would be cheaper but doing that in Amazon would eliminate most of the cost benefits. If you then consider devops cost needed to that, there is no difference. Most companies getting started with Elasticsearch should probably not do that on day 1. You can always start doing this later if there's some reason to. I've never had clusters being killed randomly but I did have a few self inflicted issues with cluster instability due to flooding the cluster with too much data and not having suitable data retention policies in place. With the recently added index life cycle management (an x-pack feature), this is easier to manage these days. If you are spending seven figures a year on elastic search, you clearly are not a beginning user and there's some cost savings that you might be able to realize by taking ownership of the problem of hosting it somewhere cheaper. For that, their recently open sourced kubernetes helm charts are worth a look. Those scripts take care of a lot of things and get you a self hosted version of Elastic Cloud. Amazon hosted clusters are indeed a bit bare-bones (as is their support for these clusters) and I would also not recommend them; you get more value for money by using Elastic Cloud.