4 ms·

Curious he does not mention the hosted elasticsearch+kibana on Amazon. Even integrates with cloudwatch and lambda to pull in the data.

• BoorishBears 10y ago
  We just migrated off AWS ES to elastic.co this week even though we pay for AWS support(and this try to centralize on it). For basic use cases it might be fine, but AWS ES doesn't support X Pack, lags behind official releases, and I even recall it being more expensive once you setup production level ES domains since master nodes are paid for separately. XPack is huge as far as solving headaches with ES, even for the authentication management alone.
  • FLGMwt 10y ago
    Whoops. We just decided to try out AES for a new project we're in the early stages of spinning up. I'd be curious to hear any more details about the two options should you have the time to spare. We use Terraform to provision our stack though and it seems that the Elastic Cloud doesn't have any programmatic configuration options. How does your team orchestrate your elasticsearch cluster(s)? EDIT: I should clarify, our use case is not ELK, it's search over a modest data set.
    • BoorishBears 10y ago
      We were using Cloudformation templates via Serverless since the ES domain was tightly integrated with the Lambdas we were deploying. Today we'd probably use SAM with the same CF templates (https://github.com/awslabs/serverless-application-model/blob/master/README.md https://github.com/awslabs/serverless-application-model/blob...) to keep things simple. We're pretty write heavy/low read with a comparatively small data set (~50GB expected over 4 months). I know ES can be tuned for that use case to improve index throughput at the cost of slower searching but I never tried setting it up on AES (I'm guessing it'd go in the CF-Template). One challenge with elastic.co is there's no equivalent to Amazons ElasticsearchDestinationFirehose. We were putting data into a Firehose which batched, backed up to S3 and accounted for failures via a Lambda but elastic.co is giving you "raw" ES instances, so you need to handle those usecases. You could probably write a Kinesis stream that replicated that functionality(and from what I understand Firehose came to be because that's exactly what people were doing), but we control the devices uploading to the Lambda, so we had the devices act as the buffer (increased send thresholds, and propagated failures to send to ES to our devices which retry automatically) and sent a duplicate record to an S3DestinationFirehose for backup and replaying data
      • cryptarch 10y ago
        Might I ask how you bootstrap your CloudFormation setup? Do you set up all IAM accounts via CloudFormation on the root account, or do you use something else for that? What do you run your CloudFormation templates on, dev machines or something on AWS? I'm currently tasked with getting a SaaS application onto Amazon and picking it up as I go along, mostly just from the docs. It's been very hard to find high-level overviews of ways to use AWS's ops tools like CloudFormation, most articles I've found are about one specific API and not about architecturing AWS apps.
        • BoorishBears 10y ago
          Serverless gets its own user that has the permissions needed to apply our CF templates, then the templates define the IAM roles. But I guess depending on the structure of your team and deploy that can be an issue, since having the serverless credentials = being able to create and assign IAMs. For example, if those credentials are in your CI, now giving someone access to your CI also gives them access to an AWS user who's practically a "superuser" (although tbh, I'm not sure how much of a problem that is since access to CI is already a pretty big deal). We run serverless (which in turn applies the CF templates) on Jenkins instead of locally (which also helps automate passing the correct stage and parameters). I guess the equivalent without serverless would be just be using aws-cli, since serverless doesn't do anything special to the CF templates There was a great webinar on production deployments with SAM (which you could replace with CF) via CodePipeline, but I can't seem to find the slides for it anywhere. If we "did it again" we'd probably use CodePipline/CodeBuild(We just used Jenkins because we didn't really have a chance to look at other options)
          • cryptarch 10y ago
            Thanks for the advice, I think I'm going the CodePipeline/CodeBuild route :)