8 ms·
AWS Elasticsearch Service Woes
- freeman478 10y agoThe main issue of this service is just that it is very outdated (it only supports ES 1.5.2) when the latest is 2.3.3 and ES 5 is just around the corner. The recent versions are far more stable and includes lots of efficiency in terms of memory usage.
- rdrock 10y agoTrue. As much as a service entices us, old version and having no timeframe of when we can expect an update to a newer version made us keep our self-managed cluster on EC2.
- vacri 10y ago1.5.2 is only 15 months old, which isn't all that long in 'enterprise time'.
- happyslobro 10y agoForget enterprise time, it's about to be two major versions old in major version time, once v5 is official. Not sure what happened to v3 and v4, I think Elastic skipped them to get all of their products aligned to the same version. Elastic says they don't backport fixes either, so in this case, it might be a good idea to stay near the leading edge of the release cycle.
- Shizka 10y agoIs it a better alternative to set it up on EC2 instances?
- derptacos 10y agoWould defeat the purpose, managing it yourself could cost substantially more in $ + time.
- wahnfrieden 10y agoIt's worth running the math on that one.
- elssar 10y agoIn terms of AWS costs, running your own cluster will cost significantly less, specially if you use reserved instances. The AWS ES instances cost ~1.6x more than their regular counterparts. Also, you can't reserve ES instances.
- njovin 10y agoThat's if you have a qualified sysadmin on-staff to manage it. We tried managing our own ES setup for a little over a year before giving up. Despite our simple needs and low traffic volume we regularly had difficult-to-diagnose issues with memory usage, node interconnectivity, and performance. With 3 EC2 nodes in a cluster we would occasionally have the cluster health go to yellow and remain in a 'recovering' sate indefinitely, bringing performance to a halt, and we'd essentially have to rebuild and swap the cluster. Granted, none of us were proper ES admins, but we had a lot of experience working with system administration and specifically database performance and clustering. Despite that we were definitely in over our heads with ES.
- toomuchtodo 10y agoQualified sysadmin/devops here: You can run a small Elasticsearch cluster (<9 nodes) without a dedicated ops person if necessary. Run with more than 3 nodes, ensure you have a proper number of index shards/replicas, and ALWAYS use an odd number of nodes. Some overprovisioning will be required, but with the extra infra spend you're delaying the need for a dedicated role to manage it.
- techdragon 10y agoSoftware I have to 3x over provision to keep alive is crappy software. ElasticSearch is by far one of the most obnoxious software programs I ever have the misfortune to administer. I now avoid self hosting it at all costs.
- balls187 10y agoYes. Amazon's typical approach for AWS products is to MVP them, and iterate functionality. This often means that early services work very well for a handful of use cases. If your case deviates slightly in requirements, you will be better off in the short term running it yourself--assuming you can't change requirements. In my experience, the AWS teams are very responsive to feedback and work pretty hard to add functionality to their mvp services.
- andrewvc 10y agoAs the author mentions AWS Elasticsearch does lag behind in versions. They still only support 1.5.2[1] which is over a year old at this point and is missing a lot of performance optimizations and operational improvements. Elastic (the company behind Elasticsearch) provides an AWS based SaaS Elasticsearch product https://www.elastic.co/cloud/as-a-service https://www.elastic.co/cloud/as-a-service with a very sane version policy[2] that keeps pace with the latest Elasticsearch releases. Disclaimer, I work for Elastic 1. https://aws.amazon.com/elasticsearch-service/faqs/ https://aws.amazon.com/elasticsearch-service/faqs/ 2. https://www.elastic.co/guide/en/cloud/current/version-policy.html https://www.elastic.co/guide/en/cloud/current/version-policy...
- Tomdarkness 10y agoWe've looked into using Elastic hosted Elasticsearch ourselves but what really put us off is that even if your whole cluster fails the only support method you get is a forum. You have to purchase a support package for the whole of Elasticsearch at some unspecified price just to get proper support for the hosted service itself. We don't want support for using Elasticsearch just for the hosted service we're paying for.
- jdc0589 10y agothe "unspecified" price is a lot. I think they reworked their pricing structure sometime around ElastiCon this year, but prior to that a subscription (which includes support, and all their products; only way they sold stuff then) was ~$5k /server/year. I think it may be cheaper now though.
- namidark 10y agoLast I checked a month or so ago it was still around 5-7k per node so that sounds accurate
- padelt 10y agoGot a price slightly above that in EUR. A cluster of 1 to 5 nodes should go for ~24.000€ (customer would've been SMB non-IT enduser company). My gut reaction was: Insane. They do include infrastructure consulting, support and the premium plugins as part of that. So at least if you can grab lots of consulting that way I can see where they are going. Then again, needing lots of consulting is a problem in itself. And I guess operating Elasticsearch is not for the faint of heart anyway...
- aaronkrolik 10y agoAnother annoying point: the documentation only lists the api methods you can call, but not the ones you cannot. I was a bit irritated to find out that the /_update method is not supported on AWS ESS
- skywhopper 10y agoI hear 2.3 support is coming in the next few weeks. Being stuck on an old version is painful but if your needs are simple then it's very useful. Ultimately if your needs are not simple then you probably should be running it yourself, just like anything else. Managed hosting for any database is never going to be as customizeable as you need if you operate at a huge scale.
- donretag 10y agoMy sources tell me it is version 2.2 at the end of June (now) or July.
- dcosson 10y agoThis intel is like gold, where'd you hear it from? The lack of transparency is my least favorite thing about AWS. I see the value in keeping brand new services or huge feature additions secret until they're released. But for small issues like when they'll update elasticsearch version or when they'll fix an acknowledged bug, I don't get why they can't share that information with customers (e.g. the way many open source projects track next version milestones on github). Is it any different at the Enterprise support level?
- notyourwork 10y ago> The lack of transparency is my least favorite thing about AWS. What do you expect from AWS? They are part of a publicly traded company, they cannot make announcements willy nilly.
- skywhopper 10y agoIf you pay for top-level support and spend a lot of money on the services you can get some information ahead of time, but it's all strictly under NDA. The more money you spend the more open the teams are willing to be, but they are understandably cagey about committing to timelines until features are imminent.
- jdubs 10y agoIt's a crappy service. When the cluster autoscales to deal with load the cluster becomes unavailable for a minute or two while it restarts. In my previous role we had a very large cloud search cluster that was always under high load and very frequently threw 500s when the environment scaled up.
- AznHisoka 10y agoHave you ever managed your own non-trivial Elasticsearch cluster personally? Elasticsearch is filled with so many subtle gotchas that aren't present in other non-distributed databases. I personally feel once you've managed your own ES cluster for awhile, you tell yourself there's no freaking way any cloud system can abstract all these ugly (but important) configuration settings and corner cases into a neat, nice cloud product. No freaking way. I'm gonna manage this myself.
- willejs 10y agoOne of the main issues i had with it was that getting the IAM policies correct on the cluster was an extremely slow process. Each change requires a rolling cluster update, replacing the underlying EC2 instances with the correct roles. Secondly access control sucks. You can only whitelist by external IP, and you cant access elasticsearch from your VPCs directly, as they sit outside your VPC. I abandoned it, wrote some chef and terraform and had a much more stable and flexible setup. However at an increased management overhead. If it works for you thats cool, but there are caveats, so beware.
- simonvdv 10y agoWe ran into the access control limitations as well. They are caused by the fact that for some reason AWS ES only supports resource based policies which is imho the wrong way around to manage your policies. We did get it to work in a useable manner by having the ES policy apply to a role (i.e. the principal is a role). If you than apply that role to your instances it will work for with instance profile based auth.
- willejs 10y agoThats exactly what i was doing with the elasticsearch plugin for logstash, but i still couldent figure out how to auth vanilla http requests to it from kibana or the like... Then i decided id wasted way too much time on this, and would just build it myself. Other services such as bonzai support basic auth which i would have almost preferred :/
- steeef 10y agoI was able to use the code mentioned in this AWS forum post to configure a proxy using node.js: https://forums.aws.amazon.com/thread.jspa?threadID=218214 https://forums.aws.amazon.com/thread.jspa?threadID=218214 Code: https://gist.github.com/nakedible-p/ad95dfb1c16e75af1ad5 https://gist.github.com/nakedible-p/ad95dfb1c16e75af1ad5 Looks like it's been turned into an NPM-installable module too: https://github.com/santthosh/aws-es-kibana https://github.com/santthosh/aws-es-kibana
- 10y ago
- cheald 10y agoElasticsearch is one of the easiest clustered databases to administrate, and it's a lot cheaper to just run your own EC2 instances with ES than it is to use the managed ES service. A roll-your own deployment is basically "install Elasticsearch, install cloud-aws, configure cloud-aws with a security group ID to identify other machines in the cluster, add that security group to all machines you want to cluster together, start the service". Totally braindead-easy.
- Xylakant 10y agoI do among other things elasticsearch consulting and a significant part of the urgent jobs I do is fixing braindead-easy ES setups. It's easy to install and configure, it will probably even run fine with a small dataset or few queries, but it will start to fall apart if you're putting load on it and haven't thought at least a little about your setup. That's not a surprising thing in itself, practically no piece of software will just eat any garbage you throw at it, so ES is at least on par with a lot of other options out there. It's often just a small thing, tweaking shard counts and index sizes, using the appropriate instance type, something like that. But it's still easy to get wrong and a lot of people do.
- fizx 10y ago> Elasticsearch is one of the easiest clustered databases to administrate You seem to have installation confused with administration. Off the top of my head you forgot security, monitoring, logging config, backups, handling common production issues such as splitbrains, write multiplication, garbage collection snafus, upgrades between versions with questionably compatible internal apis. Running any database in the cloud is non-trivial, and, running elasticsearch effectively in the cloud is harder than running Zookeeper or Cassandra, for example. Source: I founded and sold an ElasticSearch hosting company.
- cheald 10y agoMuch of that isn't specific to Elasticsearch; security, backups, logging config, etc is all going to be extremely common across just about every similar product. I didn't say that Elasticsearch was "trivial", I said that it's one of the easiest clustered DBs to administrate. Anything distributed has a whole set of caveats that you have to be aware of and know how to handle, but using managed ES vs hosting your own doesn't really change any of that. My point is that the value-add from AWS' managed ES product is much, much smaller than it might be for other similar products because of the relative ease of administrating ES.
- trungonnews 10y agoWhy are they still stuck on 1.5?
- notyourwork 10y agoWhy are you implying they are stuck and cannot move forward? The question should say "Why have they not upgraded to a newer version?". Stuck implies something prohibiting them from doing so when in reality it could be a voluntary choice they have decided.