9 ms·
Things to know before using AWS’s Elasticsearch Service
- whatnotests 9y agoWow WTH Amazon? Take the training wheels off already or fix the defaults for this service.
- moonka 9y agoPreferably both.
- justonepost 9y agoGood warning. Yeah, beta software released to production.
- StreamBright 9y agoThere is that and also from fragility from ES. I was wondering what alternatives are out there to ES. I know of Solr only.
- justonepost 9y agoSolr has the added complexity of zookeeper. ES isn't bad, but in an MT context you really have to layer a lot on top for security and configurability. It's possible it's not MT and they just didn't write the facade APIs. That'd be pretty crazy. My biggest complaint would be lack of plugin support.
- bpicolo 9y agoTo be fair, ZK is great at it's job, and that responsibility is something ES has had a lot of trouble replicating.
- StreamBright 9y agoIf you want to have consistency in a distributed system you need something like ZK. If ES does not have ZK it surely has something else, probably with different trade offs.
- rpedela 9y agoThey have fixed most of the stability issues in 5.x. I suspect some of the problems people have had with AWS ES is actually using a pre-5.x version.
- coredog64 9y agoOur 2.x cluster is much more stable than our 5.1.2 cluster despite the fact that our 5.x cluster is significantly smaller (in node count) than it's older brother. Also of note: Amazon's documentation on HTTP limits is wrong. There are some instance types listed as having a 100mb max payload that are only 10mb. We found that out when Logstash recorded a crapload of errors with the 10mb limit on what was allegedly a 100mb instance type.
- StreamBright 9y agoWell after talking to some of their tech guys we determined that we have different definitions for the work "fixed".
- rpedela 9y agoSuch as?
- StreamBright 9y agoSuch as disallowing configuration they do not like, having memory/resource leaks that they cannot find the root cause for and few other things I forgot already. G1GC is disallowed because they had a data corruption bug with it. This is few months back, they might changed some it already. The question for me is what do I get using ES over Solr? If Solr's features are enough for our use cases should I even try ES?
- rpedela 9y agoHere is their resiliency status: https://www.elastic.co/guide/en/elasticsearch/resiliency/current/index.html https://www.elastic.co/guide/en/elasticsearch/resiliency/cur... It depends on your use case. If you are already familiar with Solr and it is good enough for your use case, then use it. Solr and ES are about the same feature-wise. Scaling is easier for ES because it is built-in. Here is a good comparison of their APIs. part 1: http://opensourceconnections.com/blog/2015/12/15/solr-vs-elasticsearch-relevance-part-one/ http://opensourceconnections.com/blog/2015/12/15/solr-vs-ela... part 2: http://opensourceconnections.com/blog/2016/01/22/solr-vs-elasticsearch-relevance-part-two/ http://opensourceconnections.com/blog/2016/01/22/solr-vs-ela...
- kiernanmcgowan 9y agoAnother point of frustration is that because these endpoints are locked down[0] you cannot fully use management tools like curator - https://github.com/elastic/curator https://github.com/elastic/curator [0]https://www.elastic.co/guide/en/elasticsearch/client/curator/current/faq_aws_iam.html https://www.elastic.co/guide/en/elasticsearch/client/curator...
- jbyers 9y agoCurator support got better in the AWS ES 5.3 release a few weeks ago: https://github.com/elastic/curator/releases/tag/v5.1.0 https://github.com/elastic/curator/releases/tag/v5.1.0 "AWS ES 5.3 officially supports Curator now. Documentation has been updated to reflect this."
- yissachar 9y agoYou still can't use Curator to take snapshots, because AWS ES doesn't expose the `/_snapshot/_status` endpoint. https://github.com/elastic/curator/issues/639 https://github.com/elastic/curator/issues/639
- coredog64 9y agoIf you are stuck on 5.1.2, the Talend fork of curator works. The change is trivial, so I get the sense that Elastic is just fucking with Amazon.
- jakozaur 9y agoIt even get worse if you use Aws ElastSearch for logs. Logs are usually high volume and it can quickly beczme nightmare.
- coredog64 9y agoIt used to be worse. The max EBS volume size was 512GB (with 15% reserved for Amazon) and a max cluster size of 20. We hit that limit and had to ruthlessly prune live data. You can now add 1.5TB per node (with very large and expensive instance types) as well as scale past 20. Requesting the limit increase was a lot more difficult than most other limit increases.
- jonaf 9y agoSomething else people should know: AWS ES is on the Internet. You can't deploy it to a vpc yet, and you can only lock it down using IAM, which may or may not be good enough for your use-case. For those that prefer the ease provided by AWS ES Service, consider Elastic Cloud, which affords most of the same capabilities but is run by Elastic themselves (it was previously known as Found, which Elastic purchased a few years ago). There's also an Enterprise offering. If you're looking for a hosted Elasticsearch solution, it's probably better than what AWS is offering. Side note: they update about as often as elastic releases, whereas AWS ES is consistently behind.
- justonepost 9y agoThis sounds good. Any feedback on cost? How is plugin support / security? Integration with IAM?
- brasetvik 9y ago(Disclaimer: I work on Elastic's Cloud team) While AWS ES can be cheaper in some configurations, Elastic Cloud is actually quite competitive in pricing for larger clusters when compared to AWS' ES-service. This post compares the two services, and there's an example price comparison at the end of the post: https://www.elastic.co/blog/hosted-elasticsearch-services-roundup-elastic-cloud-and-amazon-elasticsearch-service https://www.elastic.co/blog/hosted-elasticsearch-services-ro... We support most official plugins, and if you get a gold or platinum subscription you can upload your own plugins. Elastic's X-Pack is included in every cluster, which includes security features like role based access control. It's not possible for external service providers to integrate with IAM at this point.
- bpicolo 9y agoOne issue I've found with Elastic Cloud is that there doesn't seem to be a horizontal scale-out option other than multi-DC or getting bigger boxes. Is horizontal scaling in the works? Easy horizontal scaling seems like one of the better benefits of ES. Or, alternatively, am I mistaken about how configuration works?
- rpedela 9y agoThis appears to be somewhat out of date. If you use version 5.x then pending_tasks is available. http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/aes-supported-es-operations.html http://docs.aws.amazon.com/elasticsearch-service/latest/deve...
- good_regex 9y agoGood to know, I didn't realize 5.x has that API available. Why it's only available in 5.x makes no since ES has had the API since at least 1.x
- trengrj 9y agoI looked at using AWS Elasticsearch Service for a project but had to back out due to the lack of plugin support. Running elasticsearch yourself, even in a HA setup is actually fairly easy.
- jknoepfler 9y agoI can confirm the author's frustrations with AWS ES. Having set up clusters on my own (on EC2 hosts) and using the service... The latter is expensive, inefficient, behind on features, hard to integrate with, and generally just a really crappy piece of work (like almost every peripheral AWS service, ie anything but EC2, S3, and DynamoDb). Elastic search is honestly pretty simple to set up, save yourself money and trouble and just do it.
- theparanoid 9y agoI had the same experience with their code hosting products (CodeCommit). It was better to just setup an EC2 instance and manage it myself.
- cheald 9y agoSeconded. ES is honestly one of the lowest-maintenance products I've ever deployed. It has a few quirks, but for the most part, it Just Works.
- hendzen 9y agoNot all AWS services are created equal... Some are rock solid and others (cough Data Pipeline cough Kinesis Firehose cough) must have been written by interns.
- aianus 9y agoWhat issues did you have with kinesis firehose? I just deployed a couple of those and would like to know what to watch out for.
- bpicolo 9y agoOne issue we've hit is that logrotate with copytruncate enabled breaks firehose, but afaik it's mostly been good.
- otterley 9y agoUsing copytruncate breaks a lot of software, not just Firehose. I generally discourage its use in favor of addressing whatever root cause is making you want to use it.
- vacri 9y agoIn my experience, any 'improve logging' ticket goes straight to the back of the dev's backlog. Followed the next week by complaints about the logging system not being all that good... :)
- hendzen 9y agoLots of downtime. Probably 3-4 serious incidents within a 6 month period. We also had a high number of transient errors that AWS Support considered 'expected'.
- zodiac 9y agoWhy the hate for interns :'(
- Andys 9y agoWorth mentioning that elastic.co themselves run a hosted service on AWS that is of a high quality and has none of these flaws.
- Roritharr 9y agoHaving a DevOps Engineer that wants me to go the AWS Dedicated Everything Route, I need articles like this to explain to him my fear that our problems will just change, not go away, by going that route. + Adding a fat layer of dependency.
- pg_is_a_butt 9y agoyeah, i mean, why research yourself.... it's not like it's your job. just decide to follow your fears, and search out the fake news that will feed your desires. you're completely pathetic. you're all idiots.
- subway 9y agoAlso fun is being an engineer who constantly has to explain this fear to VPs with a "Nobody ever got fired for using Amazon" mindset.
- meddlepal 9y agoComplexity never goes away... it just shifts. I dunno if that is a common saying or not but a former coworker of mine once said it and it's very true IMO. I do infrastructure engineering for a small startup and really I think with any of these managed systems you need to step back and evaluate them within the context of TCO, lock-in, security, reliability, performance and flexibility/customizability. I've heard ES isn't that much of a PITA to manage on its own, but on the flip-side I'd never sign up a small team to run PgSQL at scale.
- vacri 9y ago> I've heard ES isn't that much of a PITA to manage on its own I just run ES for my logstash setup, and ES is lovely and rock-solid... except when it isn't. For example ES deciding to just silently refuse input when its disk is 90% full - that was a bit hard to find when it happened. ES looked alive, but hunting down the reason why it stopped wasn't trivial. I've had a couple of similar but lesser gotchas as well. I guess you could say of my experience that it's not that much of a PITA (as you say), but it is still a bit of a PITA. Disclaimer: if these things weren't a bit of a PITA, there'd be no need for us sysadmins, so I should be grateful...
- xchaotic 9y agoBrutal plug, but MarkLogic is a good alternative if you want a good search solution that runs and scales on AWS (and you can migrate to another cloud or on prem)
- petethepig 9y agoHad a very similar experience with Redis on ElastiCache. When things go south, it's really hard to debug. You don't get access to logs, you don't get to change a lot of config parameters. Had to provision our own EC2 instances. It was 2 years ago though, things might be different now.
- ianamartin 9y agoBookmarked to look at the next time my boss wants to lock us into yet another AWS service. Thank you.
- cavisne 9y agoArticle is a bit naive about what it takes to run a shared service. Any API that AWS ES exposes has to be there forever, clearly pending_tasks had some risk of leaking internal implementation details that either couldn't be exposed, or that they didn't want customers building a dependency on. Likewise with the doubling of nodes, this is obviously a blue-green style deployment. In place updates would be quicker but ES can get into all sorts of weird states that require manual debugging to fix, with blue green for most of the deployment you can simply flip back. I've been pretty impressed with AWS ES compared to running it myself (other than the poor fit of IAM auth)
- dmix 9y ago> Any API that AWS ES exposes has to be there forever, clearly pending_tasks had some risk of leaking internal implementation details that either couldn't be exposed, or that they didn't want customers building a dependency on. If this is a reality of using cloud based ES then clearly it's something to seriously consider before using it - which is all the author is saying. The article is titled 'things to consider' not 'things AWS needs to fix'. ES is big complex beast of a Java app. This is good advice regardless from someone who has used both approaches (self hosted vs AWS) in production. I did not get the impress that he's saying that AWS can resolve this easily.
- jack9 9y ago> Article is a bit naive about what it takes to run a shared service. This is a bad assumption. Loggly is a shared service. > Any API that AWS ES exposes has to be there forever This a bad assumption. No API is forever. Maybe you meant a different timescale. AWS has removed and made breaking changes to APIs over the years (e.g. random breaking change: https://forums.aws.amazon.com/message.jspa?messageID=513640 https://forums.aws.amazon.com/message.jspa?messageID=513640).
- MightySCollins 9y agoI just want to get rid of the stupid proxy I have just so I can make it work... Amazon just let me put it in a VPC
- CD1212 9y agoDoes anyone have any experience of Compose.io's hosted Elasticsearch offering? I have been using it on a new project the last couple of weeks and it seems to be working well.
- skywhopper 9y agoLike almost everything else you can build on AWS managed services (RDS, Elasticache, API Gateway/Lambda, Kinesis, etc), if it's truly critical to your application's uptime, you should be managing it yourself. But if your need for ES is to support a backend system that would make your life inconvenient for a while if there are problems, is relatively small and won't grow too fast, but isn't business-critical, then the AWS managed service is fine.
- jdc0589 9y agothere's also the bit about how adding a whitelisted IP for access takes like 20 FREAKING MINUTES to take effect.
- manigandham 9y agoSide note: reading anything on medium.com is frustrating, such a slow and janky site for a glorified blog.