11 ms·
AWS Elasticsearch: a fundamentally-flawed offering
- bifrost 7y agoI've been following this issue for quite some time and I totally agree with the author. I've had a lot of odd issues with AWS's productized Elasticsearch, enough that I gave up on it entirely.
- dlojudice 7y agomy experience shows that this would be even more expensive with not that good support (very enterprise sales / customer success contact)
- bifrost 7y agoI'm not sure I follow. Running Elasticsearch on your own isn't that hard but yeah, its more than a couple of clicks. Now that you can use Elastic.co's cloud solution its probably even easier.
- outworlder 7y ago> I'm not sure I follow. Running Elasticsearch on your own isn't that hard Hah! Please do try, at any non-trivial scale. Once you have got your battle scars, report back.
- bifrost 7y agoI had a 32 node on-prem 512 core 8TB of RAM cluster in 2014, all local SSDs. It was fine.
- greatjack613 7y agoTBH sounds like growing pains, I would imagine with the might of amazon engineering behind the hard-fork it will eventually surpass the original. Question is when??
- __blockcipher__ 7y agoDon’t bet on it. AWS Elasticsearch is really as bad as this article indicates.
- lflux 7y agoAgreed, this article captures my experience accurately as well.
- tedivm 7y agoYeah, we ran a minor upgrade to our cluster earlier this week and it knocked out the entire cluster for over two hours, and we were getting AWS specific errors regarding hostname headers that we've never gotten before. I managed to get a developer advocate on twitter to lend us a hand, but if I had actually waited on support things would likely still be down. Unfortunately that seems more like a trend with AWS- there are a lot of new services that feel like they're 80% ready for production, but which are being sold as complete solutoins.
- journalctl 7y agoFinishing things isn’t cool or sexy, don’tchaknow! Gotta get that MVP out the door and move onto the next project. It’s fine, you can iterate on it after re:Invent!
- bifrost 7y agoIts been this bad for a while, I think its just a stalled project inside of AWS or the people who work on it think it functions as designed...
- nwsm 7y agoThat's a weird assumption to make. AWS Elasticsearch is focused on providing minimum necessary functionality to sell instances. Opensource Elasticsearch is focused on improving Elasticsearch.
- greatjack613 7y ago
- peterwwillis 7y agoSome would say Elasticsearch is fundamentally flawed.
- bifrost 7y agoEven if it is, people still use it. Flaws don't make something unusable, you just have to know your limitations and work within them. MySQL has come a very long way for something so initially flawed.
- tlynchpin 7y agoIf someone would say that on HackerNews then they would expect to hear "citation needed". Anecdotally what I hear is a bunch of bitching and moaning about ES yet it clearly does work and has generally all of the difficulties of any CAP problem. This indicates to me that ES is addressing a Hard Problem and to the extent that it is long lived and quite popular, it's likely not substantially worse than any reasonable alternative. Please tell us what you view as ElasticSearch fundamental flaws and give some proposed alternatives either as revisions to ES or entire other solution components.
- jng 7y agoIt is not necessary to have a valid alternative to validly declare something as fundamentally flawed. ElasticSearch is as brittle as you can get. If you don't dimension Java heap sizes properly, nodes crash all the time and uncontrollable ultra-expensive shard relocation happens. Their open source available monitoring tools have the nice side effect of overloading the cluster and bringing it down (!). The result of it being a whole hodgepodge of Java-based repurposed Lucene does show in poor performance and very poor stability. I've spent many a weekend trying to bring up a fallen ElasticSearch cluster, in some cases brought down just from monitoring. We had a use case that wasn't that easy, but not massive (100ks concurrent users, but not concurrent millions), and a properly developed C++ or even Python distributed solution would be more than able to handle it quite easily (source: ended up having to write it myself, didn't require massive anything to handle properly). Frankly I admire Elastic because I have no idea how you can turn such a piece of software into ~$90MM yearly revenues, and, mainly, how you can turn that ~$90MM yearly revenue into a publicly traded company with a nearly $7bn market cap. So much to learn from them!
- time0ut 7y agoIt is also wildly more expensive than just running your own on EC2.
- bifrost 7y agoMost people using the cloud aren't really that cost concious. EC2 is about 8x the cost of bare metal... If you factor in operational cost AWS can be cheaper but often isn't really.
- nwsm 7y agoLots of (most?) companies will eat some cost to simplify their company/organization.
- bifrost 7y ago#accurate I've seen this happen a lot. Basically it continues til it starts to eat the company and they struggle to reign it in. I did some work at $company, they were basically profitable if not for their 7 figure AWS bill. I handed them a plan that would have cut their bill in half with a one time $75k spend. They also had static load so moving to dedicated instances would've cut a huge amount off their bill as well with basically zero effort.
- erik_seaberg 7y agoAWS must make a ton of money on small deploys of dead code because it would cost more to have an eng confirm that it's safe to decom.
- time0ut 7y agoThe managed ES service makes sense for small things where the operational costs of self managed dwarf the overhead. But at scale, it just doesn't make any sense. EC2 costs are so variable too due to reserve, spot, and elasticity (depending on workload). It can be hard to compare.
- deleted 7y ago
- lflux 7y agoHeaven forbid you make a configuration change that triggers a blue-green deployment and during the deploy one of the AZs runs out of that instance SKU. Your deployment has halted until you can get ahold of AWS support to get them to unstick it (this takes a couple days even with enterprise support). There's no way to know if the AZ has capacity or not, it's something you find out in the deploy. The workaround AWS support proposed was to reserve 2x capacity so we wouldn't run into this issue on subsequent deploys.
- bifrost 7y ago> The workaround AWS support proposed was to reserve 2x capacity so we wouldn't run into this issue on subsequent deploys. Thats pretty lame of them to make you pay them more money instead of functioning properly.
- lflux 7y agoThat's exactly what we told our TAM, SA and the AWS ES PM cc'd on that thread.
- bifrost 7y agoDid they ever give you an explanation as to why it happened?
- lflux 7y ago1. We changed a configuration parameter, which we thought would be a no-op but caused a deploy 2. We ran a fairly large cluster with a large amount of i3.xlarge 3. One AZ in us-west-1 didn't have enough capacity of that SKU . The other AZ did, but this doesn't help if you have a multi-AZ deployment. 4. We switched to EBS-based instances after this
- nostrebored 7y agoReach out to your TAM again to talk about CR and RI combinations which can help to mitigate this problem.
- gshakir 7y agoGiven the lawsuit that is going between Elastic and AWS in the background, can someone confirm if shared rebalancing is a flaw in AWS offering?
- DominoTree 7y agoIt exists in mainline Elasticsearch as well as the open fork and appears to exist in AWS's offering as well - AWS appears to have forcibly disabled it across the board for unknown reasons. I think it's probably an issue with the overall back-end architecture/implementation of their managed service.
- alienreborn 7y agoTheir "managed" kafka service is even bigger clusterfuck - Doesn't expose metrics via JMX port - Doesn't support version upgrade - Doesn't support schema versioning afaik - Doesn't support adding a node or two to an existing cluster I could go on...
- pram 7y agoYes it's pretty basic. Last I looked it didn't have interbroker SASL, no TLS, can't expand node storage, and the API didn't have configurable broker parameters ...? Also Kafka and Zookeeper are just about the last things I'd want sealed away in a black box.
- minitoar 7y agoDo you have experience with Confluent? How does that compare?
- dullgiulio 7y agoConsidering that Confluent actually develops the Open Source Kafka project, I expect these problems to be AWS only. Go for the real thing and don't give Bezos money for a crippled product.
- alienreborn 7y agoConfluent Cloud is pay by usage (per GB in, per GB out, per GB stored), so its much pricier based on your org's usage. However, it is definitely feature rich. - Doesn't expose metrics via JMX but does provide a nice tool called confluent control center for monitoring and managing kafka cluster - Built in and Managed Schema Registry but - AFAIK they won't reveal cluster size or cluster version (except client compability). So, they do scaling and upgrades automatically. It's a double edged-sword but should work for lot of orgs. - Overall, definitely a better product than aws msk.
- andriosr 7y agoA similar article for Kafka would be great :)
- jalopy 7y agoThis is great intel - sorry it had to come with such painful experience. Does anyone know if Open Distro for ElasticSearch (https://opendistro.github.io/for-elasticsearch/ https://opendistro.github.io/for-elasticsearch/) has these problems? Or is it related to how AWS configures/maintains ES on their platform?
- ctvo 7y agoOpen Distro is the Amazon hard fork mentioned [1]. The missing features will most likely be a 1:1 issue. The author does a disservice to their audience: > As has happened before, Amazon took the open-source side of Elasticsearch, did a hard fork, and has been selling it as a hosted service, slowly implementing their own versions of features that have been available in one form or fashion in mainline Elasticsearch for years. When what happened was Elasticsearch changed its licensing model after benefiting off of the open source community for years to be more restrictive, forcing the fork. 1 - Amazon blog post announcing fork https://aws.amazon.com/blogs/opensource/keeping-open-source-open-open-distro-for-elasticsearch/ https://aws.amazon.com/blogs/opensource/keeping-open-source-...
- phd514 7y agoThere's been no license change. For years, Elastic has had a set of commercially-licensed features on top of Apache2-licensed Elasticsearch. Within the last year or two, they made those commercially-licensed features source-available. Ironically enough, making the source available seems to have prompted a bunch of claims that they changed their licensing model.
- arpinum 7y agoElastic mixed open source code with source available code, possibly as a landmine to sue large hosting providers, like Amazon. Amazon's fork includes removing these landmines. Elastic's hands aren't clean.
- CameronNemo 7y ago
- forty 7y agoI like elasticsearch, but it's not really fun to manage. We are still running latest 5.x because upgrading means we would lose access to our oldest snapshots (unless we reindex them all). I would love it to be managed for me but at the same time I don't really want to give access to my data to another third party, so managed by AWS is the only option (our data is there anyway). I was also under the impression that their ES offering is not great so for now I'm still handling things myself.
- bifrost 7y agoLooks like someone from Elasticsearch picked this up and tweeted about it too. I suspect one way or another this could get Amazon to pay attention... https://twitter.com/kimchy/status/1182745675617824768 https://twitter.com/kimchy/status/1182745675617824768
- jmlucjav 7y agonot just 'someone', the creator of Elasticsearch
- bifrost 7y agoColor me embarassed heh. Pretty awesome though.
- nodesocket 7y agoCompletely agree, use Elasticsearch the companies offering. After being bitten by previously no VPC support for AWS Elasticsearch (though supported now), and blue-green deploys even with a small configuration changes (used to be security policy updates even) that can take 12-24 hours I generally don't recommend AWS Elasticsearch.
- goldenkey 7y agoSince I worked for Amazon and saw how the sausage was made, I am reluctant to ever use AWS again. Random grab bags of teams using all different tech with different coding standards and methodology. Jerry-rigged micro-services turned into APIs through total object instantiation rather than efficient mutable updates.
- raghava 7y agoSomeone should point him to https://news.ycombinator.com/item?id=21228693 https://news.ycombinator.com/item?id=21228693 too. Their ESCloud is yet another unit. :D Also, based on the previous in incident regarding search guard copying xpack code, I am guessing that there is maybe some intent to get AWS to just buy out ES , thus carving a wonderful exit. Just thinking!
- reilly3000 7y agoIts worth noting that AWS's lack of security features for Elasticsearch Service have been the root cause of some gigantic breaches: https://www.infosecurity-magazine.com/infosec/why-do-elasticsearch-databases-1-1/ https://www.infosecurity-magazine.com/infosec/why-do-elastic...
- jit_hacker 7y agoLack of security? AWS offers very granular, per index, authorizations that are tied into IAM in the same way you would configure S3 or DynamoDB. If user's are failing to implement good policies, AWS is not to blame.
- mixedCase 7y agoWhen your authorization system is a usability shitfest, you're partly responsible for bad usage.
- bifrost 7y agoPreach on! The only way I've really been able to understand it well is by writing code that uses boto and seeing what errors out lol. I've found some really interesting bugs/inconsistencies too. Nothing horrible but its def unintuitive sometimes.
- ajb 7y agoThat's the right way of doing it IMO. I've got a PoC script which finds the minimum subset of permissions to allow some action: https://github.com/KanoComputing/aws-tools/blob/master/bin/aws-policy-minimize https://github.com/KanoComputing/aws-tools/blob/master/bin/a... Haven't had time to productise it yet. I think doing this makes you quite a bit safer, because it means you don't end up giving up and allowing more than you need. However, you still need to understand which actions shouldn't be allowed, so it's not the whole solution.
- 7y ago
- scarface74 7y agoThe better question, if you need advanced ES functionality, you don't want to manage it yourself, and AWS's offering isn't up to par, why not just use ElasticCo's managed version that also runs on AWS?
- hkchad 7y agoFor us it was roughly 2x the cost for the same instance sizes. Granted you get a lot more functionality but in our case we didn't need it so it didn't make sense for us to stay on elastic.co. I've been running 15ish ES domains on AWS (multiple regions) for the past 3 years. Our API is microservices based so we split up our elasticsearch clusters so we could scale each microservice independently. Yes it's a lot to manage but blue-green's are quicker with the smaller domains and once you automate snapshots and index cleanup it's not all that bad.
- mrkurt 7y agoThat's really the crux of why the AWS offering is "meh". It just needs to be good enough because they don't need to make any extra margin on it. elastic.co has to make money in addition to channeling money Amazon's way.
- bifrost 7y agoThat probably will eventually be what most people do if they keep running into this.
- deleted 7y ago[deleted]
- jit_hacker 7y agoThis article amounts to someone whining because they didn't take necessary action to prevent a dumpster fire. AWS's Managed Elasticsearch has tradeoffs and you should understand them before choosing it, but AWS is not to blame if you've under provisioned your cluster and imbalanced your shards.
- tus88 7y agoThe very, very poor tradesman blames the tool. > However, so many fundamental features are either disabled or missing in AWS Elasticsearch that it’s exacerbated almost every other issue we face. No, your choice to use AWS Elastic Search is compounding all the other poor decisions you have already made (and admitted to). This is just another one of them. Petabyte scale ElastcSearch clusters are approaching edge case usage scenarios, and one most people would expect an average SAAS solution may not be optimized for. You could have spun up - and managed - your own cluster on EC2 but instead decided to make bad decisions probably without any research or beta testing, got badly burned as a result, and are now trying to unload your sour grapes on to the interwebs.
- fiddlerwoaroof 7y agoThis sort of experience report is extremely valuable for people evaluating AWS services or trying to make a case to switch to a self-managed service.
- outworlder 7y ago> The very, very poor tradesman blames the tool. But a good tradesman will not use inferior tools.
- tus88 7y agoA good tradesman uses the right tool for the task at hand.
- deleted 7y ago[deleted]
- pearjuice 7y agoCould be entirely translated at "This very generic offering doesn't suit my very specific demand".
- romski 7y agoGranted the rebalancing is a real missing feature but what if you had used EBS volumes for data storage, to grow the disk if one node approaches its limit? I'm guessing the author is using local SSD/NVMe to get the most query performance but that does come at the cost of flexibility. Of course EBS is not as fast, and I'm sure others have run into availability issues, but one has to look at their requirements; in a fast changing environment when data needs are not certain having the ability to expand with a push-button is powerful.
- DominoTree 7y agoOur experience was that the more data you store, the more IOPS you need to search through it at a reasonable pace, and with a large dataset things can really start to crawl. That said, it's worth pointing out that EBS is the default data store in AWS Elasticsearch, and for people without a ton of data it might actually end up working "as intended"
- ignoramous 7y agoSome valid points and some relevant real-world aws support nightmare scenarios, though I think there is a chance the author might be wrong abt a few things, or may be I misunderstood them. My 2 cents: > Amazon’s implementation is missing a lot of things like RBAC and auditing. Open-distro (which AWS uses for elasticsearch deployments) supports this: https://opendistro.github.io/for-elasticsearch/features/security.html https://opendistro.github.io/for-elasticsearch/features/secu... > Shard rebalancing, a central concept to Elasticsearch working as well as it does, does not work on AWS’s implementation. Not sure why the author says AWS doesn't support it, but I have seen that it does rebalance shards just like vanilla elasticsearch would. In fact, it wouldn't rebalance only when the shard-allocator is unable to find suitable home for the unassigned shards (and that's vanilla behaviour, iirc): https://aws.amazon.com/blogs/opensource/open-distro-elasticsearch-shard-allocation/ https://aws.amazon.com/blogs/opensource/open-distro-elastics... > ...if a single node in your Elasticsearch cluster runs out of space, the entire cluster stops ingesting data, full stop. Amazon’s solution to this is to have users go through a nightmare process of periodically changing the shard counts in their index templates and then reindexing their existing data into new indices, deleting the previous indices, and then reindexing the data again to the previous index name if necessary. I think the author should employ alerts for cluster-health https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/cloudwatch-alarms.html https://docs.aws.amazon.com/elasticsearch-service/latest/dev... or write them https://github.com/opendistro-for-elasticsearch/alerting https://github.com/opendistro-for-elasticsearch/alerting and def read abt best practices for offloading petabyte-scale clusters to aws (I am sure they've read abt it already, given they're in touch with SMEs and TAMs): https://aws.amazon.com/blogs/database/run-a-petabyte-scale-cluster-in-amazon-elasticsearch-service/ https://aws.amazon.com/blogs/database/run-a-petabyte-scale-c... > Hope you had a backup of what you needed to dump. Amazingly, AWS Elasticsearch does automated hourly backups and retains them for 14 days, for free: https://aws.amazon.com/about-aws/whats-new/2019/07/amazon-elasticsearch-service-increases-data-protection-with-automated-hourly-snapshots-at-no-extra-charge/ https://aws.amazon.com/about-aws/whats-new/2019/07/amazon-el... > The second option is to add more nodes to the cluster or resize the existing ones to larger instance types. AWS Elasticsearch doesn't yet scale-out (change in instance-count) without resorting to blue-green deployments. They should have impl that by now, like they did for policy-updates: https://aws.amazon.com/about-aws/whats-new/2018/03/amazon-elasticsearch-service-now-supports-instant-access-policy-updates/ https://aws.amazon.com/about-aws/whats-new/2018/03/amazon-el... I hope fixing this is in their roadmap. ---- Also, I believe the real problem with managed elasticsearch offering is that end-users still have to worry abt the servers as it isn't truly hands-off, in a way that Lambda or DynamoDB are. This is complicated by the fact that elasticsearch exposes innumerable ways to configure cluster and index setups (read: shoot yourself in the foot). I guess, AWS Elasticsearch needs something like an Aurora Serverless Data API as the current offering takes away too much control away from power-users (can't ssh into the nodes to fix anything at all and the constant reliance on the oft-incompetent aws support to do the firefighting whilst having to frustratingly wait on the fringes with little to no transparency is a big red-flag): https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/data-api.html https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide... With perf-analyzer and automatic-index-management included in open-distro, they might already be half-way there: https://news.ycombinator.com/item?id=19361847 https://news.ycombinator.com/item?id=19361847
- bratao 7y agoShameless promo ( no affiliation, just a happy user who migrated from Elastic Search). You should check https://vespa.ai https://vespa.ai - App containers and first class tensor support is a blessing.
- atombender 7y agoI've looked into Vespa a bit lately. It looks pretty good! I'm a little disappointed in its data type support, though. With ES you can throw deeply nested data structures (maps, arrays, arrays of maps, maps of arrays, etc. ad nauseum) at it and have them be fully indexed and searchable. But Vespa doesn't really do indexing of nested structures. This means that if your application's schema is already dependent on such nested data structures, you need a mapping layer that flattens your structures. For example, if you have: address: { streetAddress: "1 Bone Way", city: "Boneville", state: "WA" } then you have to flatten it to something like: { address_streetAddress: "1 Bone Way", address_city: "Boneville", address_state: "WA" } And then, of course, you have to unflatten when you get the results (unless you only use Vespa for the IDs and look up the original data in your main data store). Same thing with arrays. Vespa doesn't really support arrays, whereas in ES, all attributes are technically arrays. (I.e., a "term" query/filter doesn't distinguish between the two: {term: {foo: "bar"}} will match both documents {foo: "bar"} and {foo: ["bar"]}.) Another oddity is the system for updating your schema, which includes not just data model definitions, but a whole bunch of files which you upload as a batch. The programmatic API for updating the schema is a little impractical, much less practical than with ES where you can just do "curl -d @mappings.json" and you're done. Also not at all a fan of their use of XML. Overall, Vespa feels more than a little antiquated. It's an old project, after all. That said, I'm probably willing to deal with the warts if it's more solid. I like that the core server is written in C++, not Java. What has your experience been in terms of clustering? With ES you can just boot up a bunch of nodes and, on a good day, it will self-organize into a pretty nice and scalable setup. (On a bad day, your cluster will become "red" for unpredictable reasons.) Is Vespa as seamless here?
- mdaniel 7y agoThat's my big complaint with Solr, too, which I want to like because on paper it seems a ton more sane, but realistically the ability to throw a random JSON document at ES -- without having to sit down and pre-define the schema -- is invaluable. I stopped by to check on Solr before posting this, and even their "schemaless" document is like 8 pages long and filled with XML settings: https://lucene.apache.org/solr/guide/8_1/schemaless-mode.html https://lucene.apache.org/solr/guide/8_1/schemaless-mode.htm...
- jillesvangurp 7y agoA couple of misconceptions about open distro that Amazon seems to be advertising to overstate what they are doing: 1) It's not a fork. There's no such thing as an Amazon specific fork of the Elasticsearch git repo in the opendistro github account. There are no Amazon specific patches to Elasticsearch. 2) Instead, Amazon redistributes a vanilla OSS build of Elastic. As is and unmodified. Producing these builds is and always has been a feature of the elasticsearch build scripts. With every release they produce OSS binaries and OSS docker containers (both without the closed source plugins) in addition to the ones that include their x-pack components. All Amazon does is take those builds and bundle their own OSS plugins. 3) What is and is not open source is clearly documented in the Elastic repository. There is zero ambiguity here (legal or otherwise) unlike what Amazon implies in their marketing. If it's in the x-pack directory, it may be closed sourced (some plugins are OSS). If that bothers you, use the before mentioned OSS builds. Everything outside the x-pack directory is OSS. OSS here qualifies as Apache 2.0 licensed or compatible. It's that simple. The OSS plugins that amazon provides are of course nice if you need them. Less nice is that they seem to be perpetually several releases behind Elastic with both the plugins and opendistro. So if you use this, you are running with known & fixed bugs that may or may not affect you. You could argue that Amazon maybe does a lot of testing. If so, those tests don't appear to be part of their OSS repos. The other explanation is of course that they only update their cloud service a couple of times a year and simply ignore bug-fixes or even patch releases to what they shipped, other fixes and improvements that happen upstream, etc. Or even any documentation for that (refer to the Elastic official release notes and documentation for what was actually fixed). If these bugs happen to affect you, you are on your own. The release notes are "whatever Elastic said a few months ago". Refer to the Amazon release notes here if you don't believe me: https://opendistro.github.io/for-elasticsearch-docs/version-history/ https://opendistro.github.io/for-elasticsearch-docs/version-.... The last few releases were basically "bump the version number" and absolutely nothing else that Amazon considered worth reporting. So, if you are comfortable running that in production use it at your own peril. I'd argue it's probably better to take the latest Elastic oss build, fork the amazon plugins you actually need (if any) and simply bump the version numbers to match the current elastic version. Amazon seems to do little more than that between releases; so you are not really missing out on any meaningful QA, support, or other stuff Amazon implies they are doing that they are clearly not doing.
- manigandham 7y agoWhy is anyone using AWS Elasticsearch anyway? AWS is a first-mover and is great when you need managed services not available elsewhere, but it's usually not the best product. Since most vendors have finally caught on to the demand for managed offerings, I don't see much reason to go with AWS for services like Elasticsearch. The cloud hosting plans directly from Elastic are much better. Easier to manage, same cost, better performance, and more reliable. And support is included.
- philliphaydon 7y agoPeople like myself don’t want to manage 50 services from 50 vendors. We want to manage it all in 1 place.
- manigandham 7y agoIs it actually 50 services? Or a few that come with a much better experience? The trade-off seems clearly worth it when not exaggerated.
- icheishvili 7y agoAWS managed services are usually subpar. We run our own ES clusters for this very reason.
- cloakandswagger 7y agoThe whole shard rebalancing problem is solved if you have an acceptable disaster recovery solution in place, which I'm assuming the author doesn't if they're complaining about have to keep around the raw copy of their data to be indexed. My team has an automated workflow that runs once a week. It creates a new cluster, re-indexes from source, starts taking customer traffic and then deletes the old cluster. The shards stay balanced, and we can recover from a total cluster failure within about 6 hours.
- DominoTree 7y agoCan I ask how large your clusters are and how many records you’re reindexing? I haven’t been able to scale this to a cluster taking in several TB a day
- deleted 7y ago[deleted]
- Esthrowaway123 7y agoYes, you should have a DR scenario. No, you should not need to use it because an aws product is a piece of shit.
- taf2 7y agoYeah unless your database is not a critical component of your application, like a experiment or doesn’t have to be available feature you should almost always self operate. It pays to have control of your data.
- rawoke083600 7y agoJa never liked the offering(ok fine not much reason given) I also just spun up an instance and installed Solr. Lol until my new team decided we need 3 solr in 2 docker containers (master / slave)... Lol wasted 2 weeks.
- Esthrowaway123 7y agoUsed to work in AWS. ES was always a bad org with crazy attrition, none of this surprises me.
- hardwaresofton 7y agoUntil people disentangle providing bare metal and providing the managed services on top of the bare metal you received, it just won't matter that AWS is worse, because customers are already in the garden, and they'll pivot their way to being better while developers suffer in silence (but pad their resumes).
- truth_seeker 7y agoPostgreSQL Full-text search is good enough. Also recently in PG12 they did many perf optimizations (Table Partitioning, Indexing, Vacuum) which will help in this regard. http://rachbelaid.com/postgres-full-text-search-is-good-enough/ http://rachbelaid.com/postgres-full-text-search-is-good-enou... Text search functions - https://www.postgresql.org/docs/12/functions-textsearch.html https://www.postgresql.org/docs/12/functions-textsearch.html
- deleted 7y ago[deleted]
- bpicolo 7y agoIf you need basic text filtering and ranking, ES is overkill. When you need more powerful ranking, highlighting, etc, postgres isn't as sufficient
- truth_seeker 7y agoNope. PG12 is just fine for any kind of ranking and highlighting. Of course you may have to use extensions like Citus DB to scale horizontally