3 ms·
Some years ago, I chose Elasticsearch for the main search engine at one of my previous jobs. Hoping to support open source, I avoided Amazon's offering and pai
by rpdillon 6y ago
Some years ago, I chose Elasticsearch for the main search engine at one of my previous jobs. Hoping to support open source, I avoided Amazon's offering and paid for hosting from Elastic, telling management that not only would it support the company that created the software, but also would put us in contact with engineers that knew the product extremely well. I even went to Elasticon!
It worked really well, but as our query rate increased, our queries' geographic component, which we would search using Elastic's geohashing, created high CPU usage on our cluster. We soon learned the CPU demands for each query were significantly higher compared to more traditional full-text search queries.
Elastic simply had no way to allow us to optimize for CPU heavy workloads in their hosted product. We talked with their product team and engineers, and they were good, and promised they were working on it, but ultimately my team needed to move to a hosted solution that offered better instance selection so we could optimize for our particular workload while keeping costs under control. So, we moved to AWS. This required navigating some weirdness with request signing, but overall worked great.
While I'm sure the hosting issues have since been addressed, this lesson taught me that the licensing of the software impacts my ability as an engineer to find alternative hosts with a variety of hosting and pricing models, and this move to a more restrictive license will likely add friction for those providers that will reduce the number of companies that will offer hosted Elasticsearch. I think this is a material practical concern.
It does leave an open question about how to fund open source, which is also a practical concern. It's probably another discussion, but I have a hunch that rather than forcing third-parties to open source their hosting stack, it makes more sense for the open-source business to make their hosting stack proprietary (Elastic's already is, IIRC), and provide an awesome developer experience with the hosted product, so hosting providers can compete on the SaaS experience and share the underlying open-source product. Just a hunch, though.
- rognjen 6y agoIt's quite simple to set up on EC2. The ansible playbook works well.