5 ms·

> Given how easy ES is to operate, I don't know why anybody would pay for a less flexible, more expensive hosted version. Easy. Amazon's service is fully manag

> Given how easy ES is to operate, I don't know why anybody would pay for a less flexible, more expensive hosted version. Easy. Amazon's service is fully managed. Getting ES running on EC2 is easy enough. Making it fault tolerant, automated backups to S3 and multi-az capable, takes more work. Even more to make sure it works properly. Add the time and complexity required to test your recovery/tolerance/redundancy plans, and you pretty much lose out on the savings. Given the choice of spending 2-3 days getting everything configured right, vs letting Amazon deal with it, I'd pick Amazon. Using platitudes: the time I save not dorking with infrastructure is time that I can spend solving my customers problems.

dangrossman 11y ago
+1. This is the reason I pay for Amazon RDS/Aurora, where the price difference versus the same EC2 instances is even greater. Sure I could set up a database... and replication to another database... and script scheduled snapshots... and build monitoring and fault tolerance against all the ways things can break... but by the time I've set all that up, I've wasted a week or more building servers instead of building a business.
- matt2000 11y ago
  Are you using aurora? How's your experience been so far? It sounds interesting but I haven't heard much real world experience yet.
  - dangrossman 11y ago
    I'm running W3Counter on it now. That's ~100GB of data and a couple hundred queries per second. It's about equal read/write load with one insert and one select for every page view I log. Nothing you can't do on a single server, but it's all I can talk about. I was renting two ~$300/mo servers at Softlayer to run MySQL (Percona), with one as a backup + failover + to run my monthly browser/OS market share reports against without killing the website by overloading the main server. I regularly spiked high loads that killed latency anyway this spring. I was going to have to upgrade soon anyway, so I decided to try out Aurora during the free preview period, and migrated everything over there one weekend. Turns out a single R3.Large instance (the smallest) was more than sufficient. It's sitting at under 25% CPU and 50% memory usage right now, which means there's plenty of room to grow too. I'd say performance is as good or better than native MySQL based on that. I'm paying less now than I did for the two servers, and get more for it, with free multi-AZ replication, nightly snapshots, and the ability to spin up a live replica in just a few minutes to run backups/large reports against, then shut down and only pay for the hour.
    - bdcravens 11y ago
      That's awesome; thanks for sharing. To me, the managed aspect where I don't have to worry about monitoring the hard drive space is a big win. How long does it typically take for the live replica to come online?
ihsw 11y ago
> Given the choice of spending 2-3 days getting everything configured right It's never 2-3 days, especially when the next dude managing it needs to be on-boarded. He might not be familiar with how you do things. This externalizes all that management overhead. One less thing that you (and your co-workers, and the next dude after you leave) can break. Also all the configuration settings are exposed via a neat UI rather than hidden in various files scattered out and about. Yes source control, Puppet, Ansible, Docker, etc makes this vastly simpler but then you need to hire and train people that are familiar with these things.
typpo 11y ago
The last time I set up an Elasticsearch instance it got owned a year later due to a remote code execution vulnerability [1]. I found out because Amazon asked me why I was DOSing people. For people who don't have time or resources to actively manage an ES cluster, security is another reason to go with Amazon's service. [1] http://bouk.co/blog/elasticsearch-rce/ http://bouk.co/blog/elasticsearch-rce/
curun1r 11y ago
Beyond that, it seems that they've spent the time to integrate ES with IAM, so having the ability to grant fine-grained access to ES using IAM policies will be a big plus for some people. We're currently using Found to host our ES index and we had to come up with some pretty creative solutions to avoid embedding the auth credentials in our applications that need access to ES. With the new service, we'd simply setup the correct IAM policies for ES and assign the proper IAM roles to the instances running our application and be done.

dangrossman 11y ago

+1. This is the reason I pay for Amazon RDS/Aurora, where the price difference versus the same EC2 instances is even greater. Sure I could set up a database... and replication to another database... and script scheduled snapshots... and build monitoring and fault tolerance against all the ways things can break... but by the time I've set all that up, I've wasted a week or more building servers instead of building a business.

matt2000 11y ago
Are you using aurora? How's your experience been so far? It sounds interesting but I haven't heard much real world experience yet.
- dangrossman 11y ago
  I'm running W3Counter on it now. That's ~100GB of data and a couple hundred queries per second. It's about equal read/write load with one insert and one select for every page view I log. Nothing you can't do on a single server, but it's all I can talk about. I was renting two ~$300/mo servers at Softlayer to run MySQL (Percona), with one as a backup + failover + to run my monthly browser/OS market share reports against without killing the website by overloading the main server. I regularly spiked high loads that killed latency anyway this spring. I was going to have to upgrade soon anyway, so I decided to try out Aurora during the free preview period, and migrated everything over there one weekend. Turns out a single R3.Large instance (the smallest) was more than sufficient. It's sitting at under 25% CPU and 50% memory usage right now, which means there's plenty of room to grow too. I'd say performance is as good or better than native MySQL based on that. I'm paying less now than I did for the two servers, and get more for it, with free multi-AZ replication, nightly snapshots, and the ability to spin up a live replica in just a few minutes to run backups/large reports against, then shut down and only pay for the hour.
  - bdcravens 11y ago
    That's awesome; thanks for sharing. To me, the managed aspect where I don't have to worry about monitoring the hard drive space is a big win. How long does it typically take for the live replica to come online?

ihsw 11y ago

> Given the choice of spending 2-3 days getting everything configured right It's never 2-3 days, especially when the next dude managing it needs to be on-boarded. He might not be familiar with how you do things. This externalizes all that management overhead. One less thing that you (and your co-workers, and the next dude after you leave) can break. Also all the configuration settings are exposed via a neat UI rather than hidden in various files scattered out and about. Yes source control, Puppet, Ansible, Docker, etc makes this vastly simpler but then you need to hire and train people that are familiar with these things.

typpo 11y ago

The last time I set up an Elasticsearch instance it got owned a year later due to a remote code execution vulnerability [1]. I found out because Amazon asked me why I was DOSing people. For people who don't have time or resources to actively manage an ES cluster, security is another reason to go with Amazon's service. [1] http://bouk.co/blog/elasticsearch-rce/ http://bouk.co/blog/elasticsearch-rce/

curun1r 11y ago

Beyond that, it seems that they've spent the time to integrate ES with IAM, so having the ability to grant fine-grained access to ES using IAM policies will be a big plus for some people. We're currently using Found to host our ES index and we had to come up with some pretty creative solutions to avoid embedding the auth credentials in our applications that need access to ES. With the new service, we'd simply setup the correct IAM policies for ES and assign the proper IAM roles to the instances running our application and be done.