5 ms·

Amazon Elasticsearch Service

147 points by johns 11y ago

bdcravens 11y ago
I wonder what this says about Amazon CloudSearch?
- room271 11y ago
  It says it is not very good. There are a lot of problems with Cloudsearch. By contrast, Elasticsearch is great and not too difficult to run yourself in AWS, or indeed elsewhere. Aside from product differences, Cloudsearch was phenomenally expensive, particularly if you had a larger dataset but low read/write volume, as Cloudsearch did not use EBS (at least not in a way you could control), unlike RDS for example.
- dubcanada 11y ago
  Nothing? Amazon has lots of Amazon + non Amazon services. They know people use Elastic Search, so people will give them money. Therefore make it.
Jgrubb 11y ago
Well that's a tiny bit annoying, since I just set up our own cluster 3 days ago to centralize all our logging...
SEJeff 11y ago
For reference: https://aphyr.com/posts/317-call-me-maybe-elasticsearch https://aphyr.com/posts/317-call-me-maybe-elasticsearch And the updated: https://aphyr.com/posts/323-call-me-maybe-elasticsearch-1-5-0 https://aphyr.com/posts/323-call-me-maybe-elasticsearch-1-5-...
aaronkrolik 11y ago
Will amazon support the head plugin? Can we enable our own plugins?
- sscarduzio 11y ago
  They support just ICU and Kibana. Shield is not there too.
icpmacdo 11y ago
I thought there was already an AWS Elasticsearch product? Am I remembering something different or is this coming out of beta or something to that effect?
- imaginenore 11y ago
  You're probably thinking of CloudSearch https://aws.amazon.com/cloudsearch/ https://aws.amazon.com/cloudsearch/
- ihsw 11y ago
  There is CloudSearch however it doesn't support schemaless stuff. Looks like this is AWS' answer to that. There's also the issue of CloudSearch taking in document batches rather than individual documents, which means you have to do a fair bit of more leg work to get your stuff in there. ElasticSearch can take in Logstash inputs which is handy and quite performant. Plus a lot of data analysts are used to Kibana so there is some appeal there.
necubi 11y ago
Seems very expensive compared to just running ES on instances yourself. Using i2.xlarges (as we do for our ES cluster) it's $1.194/hour for managed ES and just $0.853/hour for spot i2 instances. If you're will to reserve for a year, it's only $0.3555/hour. Given how easy ES is to operate, I don't know why anybody would pay for a less flexible, more expensive hosted version.
- beachstartup 11y ago
  > Given how easy ES is to operate you overestimate the average web dude's ability to run a service that actually works, and stays up. you also underestimate the budget that lazy devops people have when their employer is used to (i.e. captive to) to a $100k amazon bill that really should be $25k somewhere else.
- balls187 11y ago
  > Given how easy ES is to operate, I don't know why anybody would pay for a less flexible, more expensive hosted version. Easy. Amazon's service is fully managed. Getting ES running on EC2 is easy enough. Making it fault tolerant, automated backups to S3 and multi-az capable, takes more work. Even more to make sure it works properly. Add the time and complexity required to test your recovery/tolerance/redundancy plans, and you pretty much lose out on the savings. Given the choice of spending 2-3 days getting everything configured right, vs letting Amazon deal with it, I'd pick Amazon. Using platitudes: the time I save not dorking with infrastructure is time that I can spend solving my customers problems.
  - dangrossman 11y ago
    +1. This is the reason I pay for Amazon RDS/Aurora, where the price difference versus the same EC2 instances is even greater. Sure I could set up a database... and replication to another database... and script scheduled snapshots... and build monitoring and fault tolerance against all the ways things can break... but by the time I've set all that up, I've wasted a week or more building servers instead of building a business.
    - 11y ago
ssalat 11y ago
This new aws service offer is really boring, I don't see a real advantage migrating to it anytime soon. What will be the expectation here: - Autoscaling functionalities out of the box - Better speed than on regular EC2 instances (I/O, Network) - More visiblity, better monitoring out of the box (e.g. Marvel) - Lower pricing than running on plain EC2 Still a long way to go here...
- dangrossman 11y ago
  RDS seems to be doing just fine with a large markup over EC2. Why would the managed version of a service be lower priced than the same infrastructure being managed?
  - ssalat 11y ago
    Amazon has more possiblities to run a really huge cluster and balance the different load peaks of the search nodes. The whole business model of AWS is based on this... but you're right as well, price isn't everything and comfort worth money. And yes, I love RDS as well, especially Aurora... it's a great deal!
    - p0rkbelly 11y ago
      Running one big cluster and using application level separation would make lots of customers apprehensive...
- davidzyc 11y ago
  That's soo true.
fideloper 11y ago
Opinions on the price being high seem to miss the point of AWS. AWS is very expensive relative to other clouds, with exceptions so some very nice services (S3, SQS, SES, Route53). "Core" (Very Important™) services such as EC2/RDS are (appropriately) expensive due to the tooling and high availability options available. I'd expect the tradeoff for their hosted ES to be price, since I'm buying availability and less headache, not just the search service. I know for sure that I don't know enough to monitor JVM applications. I'd love to pay for their ES service if/when it made sense for a business.
spankalee 11y ago
Will Elasticsearch ban queries for AppleTV and Chromecast?
- phpnode 11y ago
  Solr and Sphinx
bitroliest 11y ago
We will definitely be checking this out. Our experience with another vendor and ES has not been great. From having the whole cluster come down when we're doing batch inserts for initial loads to having 10s timeouts on random queries during low activity windows with the vendor saying "dunno, everything looks good here" (when we measured that this 10s were spent waiting for ES).
- evboyle 11y ago
  If you're looking for a search backend without having to worry about managing infrastructure, you should check out Azure Search. I'm a dev on the product. If you're interested in trying it out let me know. You can ping me at my username @microsoft.com if you have questions or need pointers.
  - nullrouted 11y ago
    Side question: Do you know if Azure will every allow traceroute through their network?
    - evboyle 11y ago
      no idea, sorry
      - nullrouted 11y ago
        ipv6 support?
Thaxll 11y ago
It seems weird to handle the number of instances on top of AWS service, I mean it's AWS they should provide a solution where you don't have to think about that... It looks like ES installed by Puppet on top of EC2.
ShaneOG 11y ago
sa-east-1 gets the short straw again. > The service does not support any i2 and r3 instance types in the sa-east-1 region.
odiroot 11y ago
Nice, they even support Kibana. Just few days after I managed to finally set up ELK stack correctly by hand.
andrewvc 11y ago
Interesting, This would be Amazon's version of Found ( https://www.elastic.co/found https://www.elastic.co/found ), which is Elastic's (makers of elasticsearch) aws based cloud solution. I wonder what this means for cloud search? I should note that I work for Elastic on the Logstash team and used to work on Found.
johnhkg 11y ago
How would this compare to Algolia ? Does anyone have experience with elasticsearch and algolia to shed some light on this ?

147 points by johns 11y ago

bdcravens 11y ago

I wonder what this says about Amazon CloudSearch?

room271 11y ago
It says it is not very good. There are a lot of problems with Cloudsearch. By contrast, Elasticsearch is great and not too difficult to run yourself in AWS, or indeed elsewhere. Aside from product differences, Cloudsearch was phenomenally expensive, particularly if you had a larger dataset but low read/write volume, as Cloudsearch did not use EBS (at least not in a way you could control), unlike RDS for example.
dubcanada 11y ago
Nothing? Amazon has lots of Amazon + non Amazon services. They know people use Elastic Search, so people will give them money. Therefore make it.

Jgrubb 11y ago

Well that's a tiny bit annoying, since I just set up our own cluster 3 days ago to centralize all our logging...

SEJeff 11y ago

For reference: https://aphyr.com/posts/317-call-me-maybe-elasticsearch https://aphyr.com/posts/317-call-me-maybe-elasticsearch And the updated: https://aphyr.com/posts/323-call-me-maybe-elasticsearch-1-5-0 https://aphyr.com/posts/323-call-me-maybe-elasticsearch-1-5-...

aaronkrolik 11y ago

Will amazon support the head plugin? Can we enable our own plugins?

sscarduzio 11y ago
They support just ICU and Kibana. Shield is not there too.

icpmacdo 11y ago

I thought there was already an AWS Elasticsearch product? Am I remembering something different or is this coming out of beta or something to that effect?

imaginenore 11y ago
You're probably thinking of CloudSearch https://aws.amazon.com/cloudsearch/ https://aws.amazon.com/cloudsearch/
ihsw 11y ago
There is CloudSearch however it doesn't support schemaless stuff. Looks like this is AWS' answer to that. There's also the issue of CloudSearch taking in document batches rather than individual documents, which means you have to do a fair bit of more leg work to get your stuff in there. ElasticSearch can take in Logstash inputs which is handy and quite performant. Plus a lot of data analysts are used to Kibana so there is some appeal there.

necubi 11y ago

Seems very expensive compared to just running ES on instances yourself. Using i2.xlarges (as we do for our ES cluster) it's $1.194/hour for managed ES and just $0.853/hour for spot i2 instances. If you're will to reserve for a year, it's only $0.3555/hour. Given how easy ES is to operate, I don't know why anybody would pay for a less flexible, more expensive hosted version.

beachstartup 11y ago
> Given how easy ES is to operate you overestimate the average web dude's ability to run a service that actually works, and stays up. you also underestimate the budget that lazy devops people have when their employer is used to (i.e. captive to) to a $100k amazon bill that really should be $25k somewhere else.
balls187 11y ago
> Given how easy ES is to operate, I don't know why anybody would pay for a less flexible, more expensive hosted version. Easy. Amazon's service is fully managed. Getting ES running on EC2 is easy enough. Making it fault tolerant, automated backups to S3 and multi-az capable, takes more work. Even more to make sure it works properly. Add the time and complexity required to test your recovery/tolerance/redundancy plans, and you pretty much lose out on the savings. Given the choice of spending 2-3 days getting everything configured right, vs letting Amazon deal with it, I'd pick Amazon. Using platitudes: the time I save not dorking with infrastructure is time that I can spend solving my customers problems.
- dangrossman 11y ago
  +1. This is the reason I pay for Amazon RDS/Aurora, where the price difference versus the same EC2 instances is even greater. Sure I could set up a database... and replication to another database... and script scheduled snapshots... and build monitoring and fault tolerance against all the ways things can break... but by the time I've set all that up, I've wasted a week or more building servers instead of building a business.
  - 11y ago

ssalat 11y ago

This new aws service offer is really boring, I don't see a real advantage migrating to it anytime soon. What will be the expectation here: - Autoscaling functionalities out of the box - Better speed than on regular EC2 instances (I/O, Network) - More visiblity, better monitoring out of the box (e.g. Marvel) - Lower pricing than running on plain EC2 Still a long way to go here...

dangrossman 11y ago
RDS seems to be doing just fine with a large markup over EC2. Why would the managed version of a service be lower priced than the same infrastructure being managed?
- ssalat 11y ago
  Amazon has more possiblities to run a really huge cluster and balance the different load peaks of the search nodes. The whole business model of AWS is based on this... but you're right as well, price isn't everything and comfort worth money. And yes, I love RDS as well, especially Aurora... it's a great deal!
  - p0rkbelly 11y ago
    Running one big cluster and using application level separation would make lots of customers apprehensive...
davidzyc 11y ago
That's soo true.

fideloper 11y ago

Opinions on the price being high seem to miss the point of AWS. AWS is very expensive relative to other clouds, with exceptions so some very nice services (S3, SQS, SES, Route53). "Core" (Very Important™) services such as EC2/RDS are (appropriately) expensive due to the tooling and high availability options available. I'd expect the tradeoff for their hosted ES to be price, since I'm buying availability and less headache, not just the search service. I know for sure that I don't know enough to monitor JVM applications. I'd love to pay for their ES service if/when it made sense for a business.

spankalee 11y ago

Will Elasticsearch ban queries for AppleTV and Chromecast?

phpnode 11y ago
Solr and Sphinx

bitroliest 11y ago

We will definitely be checking this out. Our experience with another vendor and ES has not been great. From having the whole cluster come down when we're doing batch inserts for initial loads to having 10s timeouts on random queries during low activity windows with the vendor saying "dunno, everything looks good here" (when we measured that this 10s were spent waiting for ES).

evboyle 11y ago
If you're looking for a search backend without having to worry about managing infrastructure, you should check out Azure Search. I'm a dev on the product. If you're interested in trying it out let me know. You can ping me at my username @microsoft.com if you have questions or need pointers.
- nullrouted 11y ago
  Side question: Do you know if Azure will every allow traceroute through their network?
  - evboyle 11y ago
    no idea, sorry
    - nullrouted 11y ago
      ipv6 support?

Thaxll 11y ago

It seems weird to handle the number of instances on top of AWS service, I mean it's AWS they should provide a solution where you don't have to think about that... It looks like ES installed by Puppet on top of EC2.

ShaneOG 11y ago

sa-east-1 gets the short straw again. > The service does not support any i2 and r3 instance types in the sa-east-1 region.

odiroot 11y ago

Nice, they even support Kibana. Just few days after I managed to finally set up ELK stack correctly by hand.

andrewvc 11y ago

Interesting, This would be Amazon's version of Found ( https://www.elastic.co/found https://www.elastic.co/found ), which is Elastic's (makers of elasticsearch) aws based cloud solution. I wonder what this means for cloud search? I should note that I work for Elastic on the Logstash team and used to work on Found.

johnhkg 11y ago

How would this compare to Algolia ? Does anyone have experience with elasticsearch and algolia to shed some light on this ?