5 ms·

Something else people should know: AWS ES is on the Internet. You can't deploy it to a vpc yet, and you can only lock it down using IAM, which may or may not be

by jonaf 9y ago

Something else people should know: AWS ES is on the Internet. You can't deploy it to a vpc yet, and you can only lock it down using IAM, which may or may not be good enough for your use-case. For those that prefer the ease provided by AWS ES Service, consider Elastic Cloud, which affords most of the same capabilities but is run by Elastic themselves (it was previously known as Found, which Elastic purchased a few years ago). There's also an Enterprise offering. If you're looking for a hosted Elasticsearch solution, it's probably better than what AWS is offering. Side note: they update about as often as elastic releases, whereas AWS ES is consistently behind.

justonepost 9y ago
This sounds good. Any feedback on cost? How is plugin support / security? Integration with IAM?
- brasetvik 9y ago
  (Disclaimer: I work on Elastic's Cloud team) While AWS ES can be cheaper in some configurations, Elastic Cloud is actually quite competitive in pricing for larger clusters when compared to AWS' ES-service. This post compares the two services, and there's an example price comparison at the end of the post: https://www.elastic.co/blog/hosted-elasticsearch-services-roundup-elastic-cloud-and-amazon-elasticsearch-service https://www.elastic.co/blog/hosted-elasticsearch-services-ro... We support most official plugins, and if you get a gold or platinum subscription you can upload your own plugins. Elastic's X-Pack is included in every cluster, which includes security features like role based access control. It's not possible for external service providers to integrate with IAM at this point.
  - bpicolo 9y ago
    One issue I've found with Elastic Cloud is that there doesn't seem to be a horizontal scale-out option other than multi-DC or getting bigger boxes. Is horizontal scaling in the works? Easy horizontal scaling seems like one of the better benefits of ES. Or, alternatively, am I mistaken about how configuration works?
    - brasetvik 9y ago
      Per availability zone, Elastic Cloud currently scales vertically (in power-of-two increments) until a cluster hits 64GiB memory, at which point multiple 64GiB-nodes are added. While you can run Elasticsearch with e.g. two 8GiB nodes per zone, we prefer a single 16GiB node as there's fewer things that can go wrong. (If you want the second 8GiB node for redundancy, then that's exactly what our multi-zone HA configurations are for, and we encourage HA setups by making them less than twice (or thrice) the price, throwing in additional master-only nodes for free) (A bit of history: When Found (the company Elastic acquired and which is now Elastic Cloud) was in private beta in early 2012, we actually did allow custom cluster topologies. We ultimately disabled that as it was overwhelmingly used to make sub-optimal cluster configurations, such as 5 x 1GiB memory nodes)
  - randerson 9y ago
    Any idea if/when Elastic.co will support multiple user accounts and 2FA in the management portal? Not having those was a deal-breaker for us when we evaluated it awhile back, and was the sole reason we went with the less stable AWS service.
    - brasetvik 9y ago
      This is (understandably!) a common request, and both are on our roadmap.
  - deleted 9y ago
    [deleted]
closeparen 9y ago
An attacker is only ever RCE on one server away from being on your VPC subnet. You're going to have to set up authentication for internal applications anyway, although I suppose vulnerabilities in the login process are harder to exploit if you can't even get to it.
jchw 9y ago
I'm curious about how the pricing compares. I'm not very satisfied with AWS ES, but managing ES manually doesn't seem like the most fun either. (In fairness it looks like there's not too many knobs to turn, but it's still another concern to have to personally deal with.)
- andrewvc 9y ago
  Elastic cloud doesn't require any more management than AWS. You just need to click some buttons to add capacity https://www.elastic.co/cloud https://www.elastic.co/cloud
edaemon 9y ago
As Daniel Parker (acloud.guru) mentioned in the article comments, they went with Algolia [1] despite being AWS experts. He cited the uncertainty and complexity around AWS ES as the problem. [1] https://www.algolia.com/ https://www.algolia.com/
luhn 9y ago
The IAM authentication is really annoying. It's not supported by many client libraries, nor have I found an easy way to make arbitrary HTTP calls with signature v4. The only other options are completely public or IP-based whitelist, the latter which is untenable in most cloud environments.
- ecnahc515 9y ago
  You can also use a signing proxy.
  - luhn 9y ago
    I wasn't aware of that option. I'll look into it.
    - Leon 9y ago
      A simple solution in this vein is to white list your the EIP addresses of your NAT. This would give access to all resources in a private subnet (this is useful for Lambda's running in subnets).
- Sir_Substance 9y ago
  >nor have I found an easy way to make arbitrary HTTP calls with signature v4. https://github.com/okigan/awscurl https://github.com/okigan/awscurl
  - okigan 9y ago
    Yep, that's precisely why I made awscurl "easy way to make calls to AWS". I can be easily tested with AWS Elasticsearch.
    - Sir_Substance 9y ago
      It's a great tool man, I use it tonnes, thanks for making it!
boyter 9y ago
My understanding is that this is going to change and soon. Amazon has been poaching key elastic search employees presumably with the idea to improve the service to on par or better.
- canhazelastic 9y ago
  Elastic employee here, and this is the first I'm hearing of that. I don't know a single person that has left for Amazon, from any team. Certainly no "key elastic search employees". Got a source?

by jonaf 9y ago

justonepost 9y ago

This sounds good. Any feedback on cost? How is plugin support / security? Integration with IAM?

brasetvik 9y ago
(Disclaimer: I work on Elastic's Cloud team) While AWS ES can be cheaper in some configurations, Elastic Cloud is actually quite competitive in pricing for larger clusters when compared to AWS' ES-service. This post compares the two services, and there's an example price comparison at the end of the post: https://www.elastic.co/blog/hosted-elasticsearch-services-roundup-elastic-cloud-and-amazon-elasticsearch-service https://www.elastic.co/blog/hosted-elasticsearch-services-ro... We support most official plugins, and if you get a gold or platinum subscription you can upload your own plugins. Elastic's X-Pack is included in every cluster, which includes security features like role based access control. It's not possible for external service providers to integrate with IAM at this point.
- bpicolo 9y ago
  One issue I've found with Elastic Cloud is that there doesn't seem to be a horizontal scale-out option other than multi-DC or getting bigger boxes. Is horizontal scaling in the works? Easy horizontal scaling seems like one of the better benefits of ES. Or, alternatively, am I mistaken about how configuration works?
  - brasetvik 9y ago
    Per availability zone, Elastic Cloud currently scales vertically (in power-of-two increments) until a cluster hits 64GiB memory, at which point multiple 64GiB-nodes are added. While you can run Elasticsearch with e.g. two 8GiB nodes per zone, we prefer a single 16GiB node as there's fewer things that can go wrong. (If you want the second 8GiB node for redundancy, then that's exactly what our multi-zone HA configurations are for, and we encourage HA setups by making them less than twice (or thrice) the price, throwing in additional master-only nodes for free) (A bit of history: When Found (the company Elastic acquired and which is now Elastic Cloud) was in private beta in early 2012, we actually did allow custom cluster topologies. We ultimately disabled that as it was overwhelmingly used to make sub-optimal cluster configurations, such as 5 x 1GiB memory nodes)
- randerson 9y ago
  Any idea if/when Elastic.co will support multiple user accounts and 2FA in the management portal? Not having those was a deal-breaker for us when we evaluated it awhile back, and was the sole reason we went with the less stable AWS service.
  - brasetvik 9y ago
    This is (understandably!) a common request, and both are on our roadmap.
- deleted 9y ago
  [deleted]

closeparen 9y ago

An attacker is only ever RCE on one server away from being on your VPC subnet. You're going to have to set up authentication for internal applications anyway, although I suppose vulnerabilities in the login process are harder to exploit if you can't even get to it.

jchw 9y ago

I'm curious about how the pricing compares. I'm not very satisfied with AWS ES, but managing ES manually doesn't seem like the most fun either. (In fairness it looks like there's not too many knobs to turn, but it's still another concern to have to personally deal with.)

andrewvc 9y ago
Elastic cloud doesn't require any more management than AWS. You just need to click some buttons to add capacity https://www.elastic.co/cloud https://www.elastic.co/cloud

edaemon 9y ago

As Daniel Parker (acloud.guru) mentioned in the article comments, they went with Algolia [1] despite being AWS experts. He cited the uncertainty and complexity around AWS ES as the problem. [1] https://www.algolia.com/ https://www.algolia.com/

luhn 9y ago

The IAM authentication is really annoying. It's not supported by many client libraries, nor have I found an easy way to make arbitrary HTTP calls with signature v4. The only other options are completely public or IP-based whitelist, the latter which is untenable in most cloud environments.

ecnahc515 9y ago
You can also use a signing proxy.
- luhn 9y ago
  I wasn't aware of that option. I'll look into it.
  - Leon 9y ago
    A simple solution in this vein is to white list your the EIP addresses of your NAT. This would give access to all resources in a private subnet (this is useful for Lambda's running in subnets).
Sir_Substance 9y ago
>nor have I found an easy way to make arbitrary HTTP calls with signature v4. https://github.com/okigan/awscurl https://github.com/okigan/awscurl
- okigan 9y ago
  Yep, that's precisely why I made awscurl "easy way to make calls to AWS". I can be easily tested with AWS Elasticsearch.
  - Sir_Substance 9y ago
    It's a great tool man, I use it tonnes, thanks for making it!

boyter 9y ago

My understanding is that this is going to change and soon. Amazon has been poaching key elastic search employees presumably with the idea to improve the service to on par or better.

canhazelastic 9y ago
Elastic employee here, and this is the first I'm hearing of that. I don't know a single person that has left for Amazon, from any team. Certainly no "key elastic search employees". Got a source?