7 ms·

Seems very expensive compared to just running ES on instances yourself. Using i2.xlarges (as we do for our ES cluster) it's $1.194/hour for managed ES and just

by necubi 11y ago

Seems very expensive compared to just running ES on instances yourself. Using i2.xlarges (as we do for our ES cluster) it's $1.194/hour for managed ES and just $0.853/hour for spot i2 instances. If you're will to reserve for a year, it's only $0.3555/hour. Given how easy ES is to operate, I don't know why anybody would pay for a less flexible, more expensive hosted version.

beachstartup 11y ago
> Given how easy ES is to operate you overestimate the average web dude's ability to run a service that actually works, and stays up. you also underestimate the budget that lazy devops people have when their employer is used to (i.e. captive to) to a $100k amazon bill that really should be $25k somewhere else.
balls187 11y ago
> Given how easy ES is to operate, I don't know why anybody would pay for a less flexible, more expensive hosted version. Easy. Amazon's service is fully managed. Getting ES running on EC2 is easy enough. Making it fault tolerant, automated backups to S3 and multi-az capable, takes more work. Even more to make sure it works properly. Add the time and complexity required to test your recovery/tolerance/redundancy plans, and you pretty much lose out on the savings. Given the choice of spending 2-3 days getting everything configured right, vs letting Amazon deal with it, I'd pick Amazon. Using platitudes: the time I save not dorking with infrastructure is time that I can spend solving my customers problems.
- dangrossman 11y ago
  +1. This is the reason I pay for Amazon RDS/Aurora, where the price difference versus the same EC2 instances is even greater. Sure I could set up a database... and replication to another database... and script scheduled snapshots... and build monitoring and fault tolerance against all the ways things can break... but by the time I've set all that up, I've wasted a week or more building servers instead of building a business.
  - matt2000 11y ago
    Are you using aurora? How's your experience been so far? It sounds interesting but I haven't heard much real world experience yet.
    - dangrossman 11y ago
      I'm running W3Counter on it now. That's ~100GB of data and a couple hundred queries per second. It's about equal read/write load with one insert and one select for every page view I log. Nothing you can't do on a single server, but it's all I can talk about. I was renting two ~$300/mo servers at Softlayer to run MySQL (Percona), with one as a backup + failover + to run my monthly browser/OS market share reports against without killing the website by overloading the main server. I regularly spiked high loads that killed latency anyway this spring. I was going to have to upgrade soon anyway, so I decided to try out Aurora during the free preview period, and migrated everything over there one weekend. Turns out a single R3.Large instance (the smallest) was more than sufficient. It's sitting at under 25% CPU and 50% memory usage right now, which means there's plenty of room to grow too. I'd say performance is as good or better than native MySQL based on that. I'm paying less now than I did for the two servers, and get more for it, with free multi-AZ replication, nightly snapshots, and the ability to spin up a live replica in just a few minutes to run backups/large reports against, then shut down and only pay for the hour.
      - bdcravens 11y ago
        That's awesome; thanks for sharing. To me, the managed aspect where I don't have to worry about monitoring the hard drive space is a big win. How long does it typically take for the live replica to come online?
- ihsw 11y ago
  > Given the choice of spending 2-3 days getting everything configured right It's never 2-3 days, especially when the next dude managing it needs to be on-boarded. He might not be familiar with how you do things. This externalizes all that management overhead. One less thing that you (and your co-workers, and the next dude after you leave) can break. Also all the configuration settings are exposed via a neat UI rather than hidden in various files scattered out and about. Yes source control, Puppet, Ansible, Docker, etc makes this vastly simpler but then you need to hire and train people that are familiar with these things.
- typpo 11y ago
  The last time I set up an Elasticsearch instance it got owned a year later due to a remote code execution vulnerability [1]. I found out because Amazon asked me why I was DOSing people. For people who don't have time or resources to actively manage an ES cluster, security is another reason to go with Amazon's service. [1] http://bouk.co/blog/elasticsearch-rce/ http://bouk.co/blog/elasticsearch-rce/
- curun1r 11y ago
  Beyond that, it seems that they've spent the time to integrate ES with IAM, so having the ability to grant fine-grained access to ES using IAM policies will be a big plus for some people. We're currently using Found to host our ES index and we had to come up with some pretty creative solutions to avoid embedding the auth credentials in our applications that need access to ES. With the new service, we'd simply setup the correct IAM policies for ES and assign the proper IAM roles to the instances running our application and be done.
jowiar 11y ago
That price difference comes in at under a half day of developer time per month. That's rounding error in the cost of shipping something.
- necubi 11y ago
  That's the per-instance price, so multiply it by the number of nodes you need. We use about a dozen, so the difference for us would be around $7000/month.
adrr 11y ago
When it breaks, you can call/email someone to look at it. Factor in the cost of having someone on retainer or hiring a person that has experience with hosting ES.
- 23david 11y ago
  Call and/or email Amazon support to fix ES issues?
- necubi 11y ago
  Hahaha... that's not how AWS works (unless you pay them hundreds of thousands to millions of dollars a month). Amazon is terrifically unresponsive even for medium-to-large clients.
  - joesmo 11y ago
    If you have business support (10% of total a month / $100 minimum I think) or better that is how it works, except you have to wait hours sometimes and it's very slow to work with the Amazon reps. Also, always make requests urgent, otherwise you'll never get a rep.
  - kvz 11y ago
    You just need to pay premium support, which is < 100$ a mo I believe. They reply within a day, but to satisfaction in our experience. Might not suffice for all use case but thought I'd share it's not totally bad for small companies.
  - yeukhon 11y ago
    If you are not getting the kind of support with premium or enterprise support, you better talk to your TAM. We run into problems with Redshift all the time and they are quite responsive (well they do violate their SLA often and we are too nice to deal with that right now). But of course you can't get a hold of the developers until support can't do anything else.

by necubi 11y ago

beachstartup 11y ago

> Given how easy ES is to operate you overestimate the average web dude's ability to run a service that actually works, and stays up. you also underestimate the budget that lazy devops people have when their employer is used to (i.e. captive to) to a $100k amazon bill that really should be $25k somewhere else.

balls187 11y ago

> Given how easy ES is to operate, I don't know why anybody would pay for a less flexible, more expensive hosted version. Easy. Amazon's service is fully managed. Getting ES running on EC2 is easy enough. Making it fault tolerant, automated backups to S3 and multi-az capable, takes more work. Even more to make sure it works properly. Add the time and complexity required to test your recovery/tolerance/redundancy plans, and you pretty much lose out on the savings. Given the choice of spending 2-3 days getting everything configured right, vs letting Amazon deal with it, I'd pick Amazon. Using platitudes: the time I save not dorking with infrastructure is time that I can spend solving my customers problems.

dangrossman 11y ago
+1. This is the reason I pay for Amazon RDS/Aurora, where the price difference versus the same EC2 instances is even greater. Sure I could set up a database... and replication to another database... and script scheduled snapshots... and build monitoring and fault tolerance against all the ways things can break... but by the time I've set all that up, I've wasted a week or more building servers instead of building a business.
- matt2000 11y ago
  Are you using aurora? How's your experience been so far? It sounds interesting but I haven't heard much real world experience yet.
  - dangrossman 11y ago
    I'm running W3Counter on it now. That's ~100GB of data and a couple hundred queries per second. It's about equal read/write load with one insert and one select for every page view I log. Nothing you can't do on a single server, but it's all I can talk about. I was renting two ~$300/mo servers at Softlayer to run MySQL (Percona), with one as a backup + failover + to run my monthly browser/OS market share reports against without killing the website by overloading the main server. I regularly spiked high loads that killed latency anyway this spring. I was going to have to upgrade soon anyway, so I decided to try out Aurora during the free preview period, and migrated everything over there one weekend. Turns out a single R3.Large instance (the smallest) was more than sufficient. It's sitting at under 25% CPU and 50% memory usage right now, which means there's plenty of room to grow too. I'd say performance is as good or better than native MySQL based on that. I'm paying less now than I did for the two servers, and get more for it, with free multi-AZ replication, nightly snapshots, and the ability to spin up a live replica in just a few minutes to run backups/large reports against, then shut down and only pay for the hour.
    - bdcravens 11y ago
      That's awesome; thanks for sharing. To me, the managed aspect where I don't have to worry about monitoring the hard drive space is a big win. How long does it typically take for the live replica to come online?
ihsw 11y ago
> Given the choice of spending 2-3 days getting everything configured right It's never 2-3 days, especially when the next dude managing it needs to be on-boarded. He might not be familiar with how you do things. This externalizes all that management overhead. One less thing that you (and your co-workers, and the next dude after you leave) can break. Also all the configuration settings are exposed via a neat UI rather than hidden in various files scattered out and about. Yes source control, Puppet, Ansible, Docker, etc makes this vastly simpler but then you need to hire and train people that are familiar with these things.
typpo 11y ago
The last time I set up an Elasticsearch instance it got owned a year later due to a remote code execution vulnerability [1]. I found out because Amazon asked me why I was DOSing people. For people who don't have time or resources to actively manage an ES cluster, security is another reason to go with Amazon's service. [1] http://bouk.co/blog/elasticsearch-rce/ http://bouk.co/blog/elasticsearch-rce/
curun1r 11y ago
Beyond that, it seems that they've spent the time to integrate ES with IAM, so having the ability to grant fine-grained access to ES using IAM policies will be a big plus for some people. We're currently using Found to host our ES index and we had to come up with some pretty creative solutions to avoid embedding the auth credentials in our applications that need access to ES. With the new service, we'd simply setup the correct IAM policies for ES and assign the proper IAM roles to the instances running our application and be done.

jowiar 11y ago

That price difference comes in at under a half day of developer time per month. That's rounding error in the cost of shipping something.

necubi 11y ago
That's the per-instance price, so multiply it by the number of nodes you need. We use about a dozen, so the difference for us would be around $7000/month.

adrr 11y ago

When it breaks, you can call/email someone to look at it. Factor in the cost of having someone on retainer or hiring a person that has experience with hosting ES.

23david 11y ago
Call and/or email Amazon support to fix ES issues?
necubi 11y ago
Hahaha... that's not how AWS works (unless you pay them hundreds of thousands to millions of dollars a month). Amazon is terrifically unresponsive even for medium-to-large clients.
- joesmo 11y ago
  If you have business support (10% of total a month / $100 minimum I think) or better that is how it works, except you have to wait hours sometimes and it's very slow to work with the Amazon reps. Also, always make requests urgent, otherwise you'll never get a rep.
- kvz 11y ago
  You just need to pay premium support, which is < 100$ a mo I believe. They reply within a day, but to satisfaction in our experience. Might not suffice for all use case but thought I'd share it's not totally bad for small companies.
- yeukhon 11y ago
  If you are not getting the kind of support with premium or enterprise support, you better talk to your TAM. We run into problems with Redshift all the time and they are quite responsive (well they do violate their SLA often and we are too nice to deal with that right now). But of course you can't get a hold of the developers until support can't do anything else.