20 ms·
AWS services to avoid
- swyx 7y agoanyone have a non paywalled link? hate Medium
- mdaniel 7y agohttps://archive.is/hz1RG https://archive.is/hz1RG
- EdwardDiego 7y ago> Furthermore, in order to have multiple workers, Kinesis Streams require you to use multiple shards. Each worker will make claim to a shard. It helps if you think of Kinesis as AWS Kafkaesque rather than a message queue, because then shards = partitions and how you work with a Kinesis stream makes a lot of sense. Multiple concurrent consumers? You're going to need shards/partitions. Now, question is, as a replacement for homerolled Kafka in EC2, or AWS's Managed Kafka, is it cheaper? So far, magic 8-ball says "benchmarking for your given use case needed". So far my experiments for our workload and patterns say - maybe, but more investigation is needed. I plan to role out a side-by-side Kafka and Kinesis experiment on a given topic and ascertain the costs. Although ultimately, like any distributed messaging system, you end up engineering to the foibles of the system - in Kinesis' case, it's the fact that it rounds each sent message (which could be a batch of records, or a single record) up to the nearest 5KB for billing that you have to engineer for.
- jpgvm 7y agoKinesis is just a very very poor Kafka. They wanted to have a Kafka but they also had this silly idea that everything needed to be HTTP so they built a poor clone with a HTTP interface. Unless forced into it by some other system there is never any reason to consider Kinesis over Kafka or Pulsar.
- EdwardDiego 7y agoOnly reason I'm looking into it is cost engineering tbh. As to the Kafka vs. Pulsar, I've been running a Kafka cluster for several years now, and was interested in Pulsar as a Kafka++, and have been evaluating it for a client who wants to choose between one of the two, and at the moment, I think Pulsar needs about another year before I'd recommend it. It's exciting tech, but as is normal of recently open sourced projects, there's a lot of bugs being surfaced, and the documentation has a lot of unanswered questions, like, if I'm consuming a multi-partitioned topic in an exclusive subscription, what does that mean for ordering? I think it has some great ideas (especially decoupling brokers from storage), but yeah, it feels a bit like Kafka pre 0.8, interesting, but you're taking on a lot of work adopting it at the moment.
- jpgvm 7y agoI think saying it's analogous to Kafka 0.8 is reasonably apt. It's very stable and performant in the verticals it has been used in at Yahoo but not as widely integrated, understood and optimised as Kafka across a broader set of use cases. That said if what you are trying to do fits squarely in the box of things that do work well it's considerably better than Kafka in a few keys ways. The first is the obvious separation of storage and compute/broker responsibilities, the benefits of which also led to the same design being used in LogDevice - a similar system designed and built completely independently and at roughly the same time at Facebook. The second is selective acknowledgement, i.e the ability to acknowledge messages as processed by a consumer out of order rather than merely the offset of the latest message processed. This allows Pulsar to be more easily used as a workqueue without the multitude of hacks and layered infrastructure required to get the same out of Kafka. Shared subscriptions/partitioning model. Compared to Kafka its more flexible, less punishing and less beholden to the architecture of consumers. Finally I would say tiering, for some it means nothing but depending on your use case it can be defining. Pulsar can offload historical segements to long term stable storage but still present a unified offset like API to consume historical data. So I think for some Pulsar provides enough benefit to make up for any shortcomings in integration as long as you have sufficient engineers able to debug/patch issues.
- helsinkiandrew 7y ago> With the Lambda server-less paradigm, you end up with 1 lambda function per route I’m not sure why this is the case. You could host all modules in the same lambda endpoint /api/v1/* Using the same technology you would use with any other backend
- blntechie 7y agoI was going to say the same. For smaller apps, just build a monolith as a function and proxy all routes to it from API Gateway. If microservices based app, each resource or route mapped to a function like what he mentioned is common though.
- damagednoob 7y agoYeah, we use https://github.com/awslabs/aws-serverless-express/ https://github.com/awslabs/aws-serverless-express/ which solves the problem of multiple routes and running an express app locally.
- philliphaydon 7y agoYeah I use ASP.NET Core Lambda. Article's comments don't seem well researched. Just used tech. Ran into road block. Blogged. Complained.
- popotamonga 7y agoI use Api gateway to redirect to a regular Flask lambda with all the endpoints
- zegl 7y agoAgreed, and even if do want to use a different lambda per route, it's not as troublesome to setup as the author claims.
- dlhavema 7y agoYeah a swagger file broken up with multi-file-swagger npm module works great
- mattacular 7y agoThe CloudFormation one is pretty accurate. Relying on nested stacks to "share" templating pieces: never again.
- gonzo41 7y agoAnd that's exactly the feeling I had, but hey we're in prod now, so we'll circle back in 5 years to see what we do next.
- mattacular 7y agoDon't fret! You still have to look forward to all the weird service limits you'll hit both directly and indirectly using CFN to maintain and extend a prod infrastructure YoY
- mvanbaak 7y agoOne can use AWS::Include. See https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/create-reusable-transform-function-snippets-and-add-to-your-template-with-aws-include-transform.html https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGui...
- synthc 7y agoNested stacks are a pain: use imports and exports and keep your stacks small to lessen the pain.
- WilliamEdward 7y agoI know it still makes sense but 'amazon web services services' is funny
- 2ion 7y agoWhile CloudFormation is annoying to use without extra tooling (like Sceptre -- makes management of dependencies between outputs and inputs go away easily, and it's extensible with plain Python), Terraform is horrible in its own ways. The AWS provider has many consistency issues, even glaring ones like not detecting that IAM policy documents have not changed when using lists in actions=[] and resources=[], and after having tried it, it makes me uncomfortable managing large landscapes with it. Terraform makes mostly sense -- because there is good tooling around CloudFormation, and CFN more often than not does the well enough -- when you find out that CloudFormation is like Terraform just a driver for AWS APIs, likely developed by a dedicated team, and as such features of the underlying APIs are often not exposed at release time in CloudFormation. I noticed this especially when experimenting with EKS: There's eksctl, which integrates with CloudFormation (generates some stacks) but is utterly useless for integration because you can't import outputs or exports, so you have to hardcode all your SG IDs, VPC IDs, Subnet IDs if you want to integrate with existing infrastructure (https://github.com/weaveworks/eksctl/issues/1141 https://github.com/weaveworks/eksctl/issues/1141). Dealbreaker, waste of time, disappointing for an "official" CLI. Next, there's pure CloudFormation -- but no luck for you, to this day AWS doesn't support EKS endpoint access control settings through CloudFormation (https://github.com/aws/containers-roadmap/issues/242 https://github.com/aws/containers-roadmap/issues/242 - dealbraker, if you need that and are allergic against public control endpoints in your infrastructure. Looking at Terraform -- it supports integration with existing CloudFormation infrastructure and can access CFN exports, it supports all the EKS settings you'd want, it offers a consistent interface to these features, so you got something to use, short of rolling your own. Mind you, CloudFormation is extensible using Custom resources, but hacking around CloudFormation is likely not worth your time and something Amazon should do. Anyway, CloudFormation is likely one of the most tested and well-working parts of AWS, so I'd prefer it over third-party state management unless I find a very good reason to do so.
- scarface74 7y agoso you have to hardcode all your SG IDs, VPC IDs, Subnet IDs if you want to integrate with existing infrastructure (https://github.com/weaveworks/eksctl/issues/1141 https://github.com/weaveworks/eksctl/issues/1141). Dealbreaker, waste of time, disappointing for an "official" CLI You can reference Parameter Store and Secret Manager entries in CF so you don’t have to hardcore values.
- ciguy 7y agoCloudformation is spot on. There is simply no excuse in 2020 to use it when cloud agnostic Terraform exists. Elasticache is expensive and the analysis is generally correct on that, however if you do actually need a high capacity rock solid Redis cluster and you can afford it, it does the job. Paying someone to setup and scale a cluster of that magnitude would probably cost a year's worth of usage. Of course if you don't actually need to scale it yet then just use the suggestions in the article to run it yourself.
- scarface74 7y agoTerraform is not “cloud agnostic”. Every provisioner is specific to the platform.
- pepemon 7y agoRight, but the HCL syntax and overall Terraform paradigm is not.
- ciguy 7y agoPerhaps bad terminology on my part but I would argue that Terraform IS cloud agnostic, it is NOT cloud portable. You can use Terraform with any cloud. You cannot seamlessly move your AWS configuration to Google Cloud but you can still write Google Cloud configuration with the same paradigm as your AWS configuration.
- eraad 7y ago
- KaiserPro 7y agoCognito: yes, its a complex poorly documented pile of shite. BUT. It does work as a reasonably secure OAUTH2 thingamebob. However I was told by my AWS account manager that auth0 was the way forward, and I agree. Cloudformation: Meh, I have about 35k lines of active CF at the moment. Its much of a muchness. Unless you are using parameters with selectors, you are going to have a bad time. Hard linking templates together (I assume thats what nested stacks are) is terrible. I've only briefly used terraform, so I have no idea if its much better. CF _could be_ a lot better. Like compile time validation, not just in time. that would stop a lot of anger when you realise you've spelt a CF parameter wrong(or the value fails validation) but only after you've spent ten minutes for it to spin up. Thats frankly unforgivable. Elasticache: Yes. Its expensive. KINESIS: What a disappointment. Stupid naming conventions, Terrible throttling and throughput. Its just horrific. Whats worse is that they looked at SQS and thought: "this compares favouribly" NATS.io is a great fit for certain usecases (no, kafka is never the answer) LAmbda: I don't actually get this myself. I made a REST api exclusively in lambda. It meant that I could build a working prototype really quickly. Once proven we ported it to fastapi in an autoscale group. The API gateway was heavily integrated into the lambda spinup (controlled in CF) so I really don't see what the issue it. Also it understands swagger, so I struggle to understand the criticism
- PaywallBuster 7y agoAuthor is making a new lambda for each route. But you can point each route in API Gateway to a different function in the same lambda. Additionally, using serverless framework facilitate local testing too. That solves all the complaints from the author.
- kdidbsbsbd 7y agoThat only works if your functions require same resources. If you need different memory/cpu power you're unevenly splitting functions where ever
- zhdc1 7y agoI've used Cognito for personal/pet web app projects before. It was good for my use case - quick and cheap user authentication.
- blickentwapft 7y agoWhere’s dynamodb?
- goostavos 7y agoDynamoDB is fine as long as it fits your use case. The trouble is that without a time machine, it's very tough to realize that your use case isn't the one DynamoDB is good at. The docs and Best Practices certainly suggest that anything is possible. Do you need God Tier Scale (and understand enough about DDB to achieve it (and have the $$$))? DynamoDB is awesome! Otherwise... maybe consider a few other options...
- cddotdotslash 7y ago> With the Lambda server-less paradigm, you end up with 1 lambda function per route. Only if you use a framework that splits it up that way (which I agree is horrible). But nothing about Lambda _requires_ this architecture. In fact, API Gateway has a proxy mode that allows you to serve all requests from a single Lambda.
- serguzest 7y agoMedium doesn't let me read this unless I pay. You should avoid medium blog writing too.
- mdaniel 7y agoWhile I agree that medium should die in a fire, it has been my experience that someone has usually submitted the URL to archive.is and that is true in this case, too: https://archive.is/hz1RG https://archive.is/hz1RG You may also enjoy trying an incognito window in the future
- antirez 7y ago"Remember redis is meant to hold ephemeral data in memory, so please don’t treat it as a DB. It’s best to design your system assuming that redis may or may not lose whatever is inside" Of course I disagree with that statement. Many serious Redis use cases at big companies assume Redis is there and will hold your data. What they usually assume is that under certain cases during failovers or other serious failures you many lose some acknowledged writes that were sent immediately before. And this most of the times does not happen too. Note that many high performance applications using SQL DBs with a relaxed fsync configuration will have the same assumption. The suggestion should be more: if Redis for you is just a volatile aid, use a given setup, if it's were you store data, use another setup. Anyway if you want to see a version of Redis where you can store also bank accounts, transactions or any other of the most critical stuff you can store in a DB make sure to check the news at Redis Conf 2020 (the conf is in streaming and free). EDIT: But even after that announcement the biggest value for Redis is that it can be fast and provide a "best effort" consistency level that is adequate for a lot of use cases in practice. Many systems were sacrified in the temple of wanting to provide linearizability for all the use cases.
- lidHanteyk 7y agoRegardless of whether Redis is meant to hold ephemeral data in memory, it is well-known to anybody who has had to maintain a Redis instance as part of their deployment that Redis is only effective when used for ephemeral in-memory work. I understand your defensiveness, but please understand that because of your massive experience with Redis, either you have never lost data with Redis, which means that nobody besides you understands how to run your software in production, or you have lost data with Redis, which means that it's somewhat hypocritical of you to insist that Redis is durable.
- tomnipotent 7y ago> that Redis is only effective when used for ephemeral in-memory work Hogwash and there are plenty of examples out there that prove otherwise. > somewhat hypocritical of you to insist that Redis is durable If Jepsen has demonstrated anything, it's that no database is as durable as it claims to be.
- gerbilly 7y agoI can't say I liked one of his proposed alternatives: Terraform I tried it and found the experience to be far inferior to Ansible, especially if you provision actual Linux instances with it. Terraform was always getting stuck in an inconsistent state, and trying to roll back, which doesn't work well for any type of stateful resource. I guess Terraform may be ok for provisioning resources which are stateless and interchangeable, say security groups and other simple services. I can't say I enjoyed using it much, though the rest of Hashicorp's offerings, Nomad, Consul etc are really great.
- allanmacgregor 7y agoWell now you are comparing apples to oranges, terraform main focus is managing and provisioning infrastructure while ansible is more commonly used to do the actual instance provisioning and configuration. In fact, Terraform + Ansible is a fairly common and powerful stack.
- HatchedLake721 7y agoAnsible and Terraform is like Apples and Oranges. They’re completely different tools completing each other.
- mvanbaak 7y agoWhile it's possible to provision actual instances with terraform, it's not what you should be doing. Let terraform handle the infrastructure, and have ansible or puppet or saltstack or cfengine or whatever provisioning tool to do the instance setup. It's all about the right tool for the job, and managing your linux install with terraform is not that.
- diroussel 7y agoYes terraform and ansible have different sweet spots. Terraform is best when managing stateless resources. It can handle stateful resources, but as always managing state gets tricky fast. If what you are CRUD-ing doesn't map well to the terraform concept of a resource, then don't use it. SO i don't like to use to create anaything inside a VM. Using terraform to create an auto-scaling group is good, though, as it can be represented as a stateless resouce. Or rather any state in it is transient.
- eraad 7y agoI believe the title of the article should be "AWS Services You Should Avoid if you don't need them or if they are not the right fit for your specific application, for example, if you need social login you are better off with other alternatives more fit to the social login use-case; or for example, if Lambda is not the best fit for containing and deploying my code given my architectural decisions at the moment of this writing".
- kdidbsbsbd 7y agoThat is a very long sentence
- CSMastermind 7y agoI wholeheartedly agree with the Cloudformation take. Terraform is better in almost every way. If you use Cloudformation you'll end up writing a bunch of bash script wrappers or similar around it to make it actually do what you want. Everyone who's tried both at scale has said the same thing.
- boredgamer2 7y agoTerraform is better in almost every way. I disagree. I spent time in Terraform a few years ago working with a client and Terraform had the ability to create but not tear down resources for some services. I was shocked -- check out the Github issues history. I ended up writing a "bunch of bash script wrappers or similar around it".
- noirbot 7y agoYea, this is exactly what I was talking about. It's not to say that Cloudformtation can't run into the same thing - deleting S3 buckets is difficult in any situation - but there's a lot of things Terraform doesn't/won't do, and often you're left with just making your own second layer of automation to work around.
- noirbot 7y agoTo be fair, that also happens a lot in Terraform. I'm taking over work at my org where people have wrapped Terraform in 2-3 different tools and templating engines. And then you have to work out how to robustly store and back up your TF states because piecing it back together if you lose your state, or someone else corrupts it, can be just as bad as the worst CF issues. Overall, I think Terraform is better if you're deploying a lot of inter-connected resources, but Cloudformation makes a lot of things "just work" in ways that Terraform doesn't. I think of it like ECS vs. EKS/Your Own Kube Whatever. ECS is full of gotchas and limitations, but if you play along with it you get a lot of things "for free".
- oneplane 7y agoIf you store your TF state on S3 and use versioning you'll never have that lost state problem.
- WatchDog 7y agoRegarding using lambda for a REST API, just set up API gateway endpoint in proxy mode, and run all your endpoints from one lambda, will be much simpler, and if you use a decent framework, it will be really simple to convert your app into a regular daemon process if you ever decide to move off lambda. Regarding cloudformation, I view manually writing cloudformation json/yaml as an anti-pattern. I would recommend looking at using CDK, which is a framework for writing your cloudformation stacks as actual real code, not markup. It's a great tool, and I find it enormously more productive and expressive than terraform. CDK allows you to write CDK apps in a variety of languages, but I would recommend just using typescript, since that's what its written in. I tried the java bindings, but it was pretty clunky. For a non-AWS specific CDK like experience, you could also look at using pulumi, but I haven't really used it much myself. Trying to define infrastructure purely by declarative markup based languages is such a waste of time.
- mvanbaak 7y agoEven if you dont want to use the proxy route, you can point multiple routes to the same lambda function, and every ok-ish routing module/framework will know what code path to follow.
- impowski 7y agoOne time a hosted an ElasticCache for testing purposes for Bull library for a couple of days and then I forgot about it for a month but I wasn't even using it. Got the bill for 1500$ :)
- sputknick 7y agoI work in a security office managing AWS infrastructure resources across a large org and three of these services work great for us: CloudFormation: this is an excellent resource when you need to tell teams across our org how you want them to set up resources. Rather than have numerous lengthy meetings where we tell them what to do, we just give them the CF template, super simple, and we guarantee everyone has the same setup. Kinesis: works great for ingesting the data we had them set up resources for with the above mentioned CF script. I can't speak to the Java dependency the author mentioned. Not an issue for us. YMMV. Lambda: Also works great with the CF template setup mentioned above. Super cheap to use. Maybe the difference between our implementation and the authors is the frequency and trigger used. Our lambda functions are all time based and run once a day, or maybe a few times a day. Super reliable, super easy, super cheap. I think the overall thesis here is that the usefulness of AWS services depends on what you are using them for.
- dlhavema 7y agoThat's a great use case for lambda. Small self contained scripts/jobs that don't charge you all day to be available once.
- languagehacker 7y agoThis is definitely a post based in experience, but maybe not based on broad expertise in using these tools. I don't want to pick apart the entire post, but I will say that the Lambda + API Gateway example is maybe the best example of jumping into the cloud-native world with blinders on. Just about every modern programming language has a toolset right now that will do the work of generating a CF template for you that creates an API gateway that forwards all HTTP routes to a single Lambda, and then that Lambda is responsible for handling the actual routing of that request. Examples include Zappa, ClaudiaJS, and Ruby on Jets just to name a few. I can't imagine providing a feature-rich web application in Lambda without this kind of abstraction. Not knowing that such tooling exists, or explicitly choosing not to use such tooling, and experiencing pain as a result seems to be common theme in this article. If you dive into architecting a system using AWS's product offerings without understanding the tradeoffs you're making, you will experience greater cost and greater friction -- hands down.
- dayjah 7y agoYeah, missing the proxy pass option will definitely burn a prospective Lambda user. I also felt a similar tinge when the post talks about KCL. Yes, you use it, and yes there are rules about output streams - but they all make sense when deployed in fairly complex environments. I can’t remember when I last used stdout for logging outside of greenfield development. Once it’s on prod everything goes to stderr and is highly structured. Similar sentiments about Cognito. At some point of complexity you must have a server in between using the Admin* range of commands. If you want their “get going quickly” then yes, there are trade offs like WebViews. That said; I’ve never used Auth0 - so insert a Luddite warning here. Finally, one hundred percent agree on CF.
- jacobra2 7y agoCould you elaborate on the proxy pass option piece?
- davidgh 7y agohttps://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-lambda-proxy-integrations.html https://docs.aws.amazon.com/apigateway/latest/developerguide... From the article: In Lambda proxy integration, when a client submits an API request, API Gateway passes to the integrated Lambda function the raw request as-is. ...and: You can set up a Lambda proxy integration for any API method. But a Lambda proxy integration is more potent when it is configured for an API method involving a generic proxy resource. The generic proxy resource can be denoted by a special templated path variable of {proxy+}, the catch-all ANY method placeholder, or both.
- watermelon0 7y ago> With lack of drift detection comes great uncertainty. But CloudFormation does have drift detection: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/detect-drift-stack.html https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGui... I'd pick Terraform before CloudFormation, but there are things that are better with CF, such as rollbacks (they really do work as expected in almost all cases), and AutoScalingGroup's UpdatePolicy (rolling/replacing deploy of new launch configuration with healthchecks), which is not available outside of CF.
- mike503 7y ago> AutoScalingGroup's UpdatePolicy (rolling/replacing deploy of new launch configuration with healthchecks), which is not available outside of CF. This drives me crazy. I never want to touch CF, but if I want rolling ASG, I have to. Why?!?
- dastx 7y agoThere are a lot of issues with both cfn and tf. As always,it comes down use-case. In my opinion, tf is terrible choice unless you're going to be using it for more than AWS
- acdha 7y agoOr if you want to use AWS features on a deterministic timescale: CloudFormation frequently lags on support for new features - I’ve worked on multiple projects which used Terraform for months before the equivalent feature could be used in CloudFormation. For example, when ECS launched secrets support it took most of a year before CF was updated. This is a big deal with classic CloudFormation because there was no escape hatch. Terraform can run arbitrary code if needed but you had to split CF stacks apart so you could have something else run in between. That’s probably better with CDK, of course.
- synthc 7y agoCloudFormation has an escape hatch via custom resources, which can call Lambda functions do create/update/destroy resources. It's ugly, but it works.
- smitty1e 7y ago> With the Lambda server-less paradigm, you end up with 1 lambda function per route. You do? Why can't a function be written to handle multiple routes?
- noirbot 7y agoI'm confused that the author says about Cloudformation: > Lack of Drift detection or reconciliation. With lack of drift detection comes great uncertainty. Cloudformation has definitely had Drift Detection since 2018: https://aws.amazon.com/blogs/aws/new-cloudformation-drift-detection/ https://aws.amazon.com/blogs/aws/new-cloudformation-drift-de... It's not everything you might want it to be, but it's not like Terraform will reconcile your drift automatically either, that I know of.
- ReidZB 7y agoDrift detection is only implemented for a very small subset of resources: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-stack-drift-resource-list.html https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGui... Terraform does indeed reconcile drift 'automatically' across all resources, by which I mean the plan will include changing everything that's drifted back to the specified configuration. That may not always be desirable, which is why building a good plan/apply process with approval is important. (Same goes for CloudFormation, though.)
- jtdev 7y agoFor Python based Lambda REST API development, check out Chalice: https://github.com/aws/chalice/ https://github.com/aws/chalice/ Very Flask like and includes CLI components to automate deployment. Even includes simple decorator based event mapping for S3, SQS, scheduled tasks, etc.
- buzzy_hacker 7y agoAWS CDK can generate CloudFormation templates, it’s pretty cool: https://aws.amazon.com/cdk/ https://aws.amazon.com/cdk/
- thanksforfish 7y ago> Cloudformation The approach CFN used made sense when there was no alternative, but Terraform's approach is so much better. I wonder how AWS approaches this space in the future. CFN has great vendor lock-in, and needs to remain supported for any customers who are locked in, but its harmful for any new development to use CFN. I get that AWS likes having infrastructure framework level lock-in, but they need to do better. Does anyone have a feel for how hard AWS support, account reps, or training push CFN versus Terraform? I've been out of the AWS cloud for a while.
- noirbot 7y agoHaving been in some calls with reps/solutions architect folks recently, the push hasn't been super hard. I don't think the CFN vs. Terraform discussion has ever come up.
- trill123 7y agoSupport doesn't officially support third-party products, and support engineers may or may not know Terraform. Engineers definitely aren't trained on Terraform, and your mileage will vary a lot depending on the specific engineer and their background. It really depends on whether they've run across it in other jobs, personal projects, etc. Customer obsession is a big deal for support engineers. So if they can, they'll try to help solve a Terraform issue. But it'll be on a personal, best-effort basis. For such a mud-slinging article, I'm surprised the author didn't complain about the most pressing issue (IMO) with CloudFormation: There have been a few instances where thing in AWS changed and Terraform was updated more quickly than CloudFormation was. Happens less and less now though.
- exidy 7y ago> Support doesn't officially support third-party products This is not entirely correct. "AWS Support Business and Enterprise levels include limited support for common operating systems and common application stack components" as documented here: https://aws.amazon.com/premiumsupport/faqs/ https://aws.amazon.com/premiumsupport/faqs/ under "Third-party software"
- Twirrim 7y agoI'm honestly not familiar with most of the services mentioned from a user perspective, the AWS service I worked on didn't consume them at the time, and I work for another cloud platform now. That said, this point gave me particular pause, because it brought in to question the other assertions: > The application collected small json records, and stuffed them into Kinesis with the python boto3 api . On the other side, worker process running inside EC2/ECS were pulling these records with boto3 and processing them. We then discovered that retrieving records out of Kinesis Streams when you have multiple worker is non-trivial. Yeah.. because that's really not what Kinesis is designed for. They even, rightly, point out that SQS is a better fit for that purpose. That hardly makes it a service to avoid.
- senderista 7y agoThat was the most annoying part of the article. A streaming data platform is not a task queue. Why do you think they built Kinesis in the first place when SQS had been around for years?
- karterk 7y agoCan't believe Cloudwatch wasn't mentioned here. Could be such a useful product that is so messed up that there is probably multiple billion dollar companies in this space.
- RandomTisk 7y agoThey need to do away with CloudWatch/Logs API Call limits. They actually push users into using Kinesis if they need frequent access to their logs, which adds an unbelievable layer of complexity just for the privilege of avoiding even more latency from using S3 and writing their own scripts/code to collect and pull what they want.
- zwily 7y agoI’ve heard rumblings from inside AWS that there is internal pressure not to make CloudWatch too powerful, lest it harm their partners (like Datadog, etc). I’ve heard this a couple times after complaining to AWS engineers about CloudWatch shortcomings. That said, CloudWatch has gotten tons better than a few years ago.
- pentlander 7y agoI don't think that's true, internally there was a long term plan to bring cloudwatch up to par with internal metrics and alarming systems. At the time when I left Amazon, there were some shims to use internal dashboards and alarms with cloudwatch data. The eventual goal was for the whole company to move to cloudwatch instead of maintaining two systems.
- somebodythere 7y agoWhat I don't get was the broad internal push to building on AWS directly, which was great most of the time, but came with big downgrades in metrics and alarms if you were to use the AWS equivalent.
- dirtydroog 7y agoReally. I found it extremely useful when applying custom scaling policies. It's miles better then GCP's Monitoring.
- vagab0nd 7y agoI have mixed feelings about Cognito. I love the simplicity. I love the 3rd party integration that comes with it. I love the fact that I don't have to manage user credentials and everything else by myself. However, I hate that the UI is not polished, and there's very little customization allowed. As an example, from the sign in/up page, you can't add a link to go back to the home page. I hate that if user signs up via Google, and later on tries to reset password via the Google email, Cognito silently drops the request. I hate the js library. Hard to use and documentation is not great.
- thanksforfish 7y ago> You dig into CLOUD WATCH logs. We have three sets of logs all intermixed together. This is unreadable. I'm not following this. You don't read logs one by one. You filter by source, error code, time range, etc. Its been a while, but I'm also pretty sure CW has log groups.
- trill123 7y ago> Its been a while, but I'm also pretty sure CW has log groups. This. If you have three sets of logs intermixed together, things have been architected incorrectly.
- katzgrau 7y agoI thought for sure I was going to find Elastic Beanstalk. It's great when it works, but when it doesn't, the list of ways it makes life hell is a long one. Elasticache though? Pricey, but I recently got an alert at 1am that one of my core services was down and it was because Redis was out of memory. With Elasticache I doubled the size of the instance in place from my phone so I could go back to sleep. Fixed the key leak in the am and returned to the original instance size.
- ramraj07 7y agoDo you have any details on EB failing?
- senderista 7y agoIME EB often gets into unrecoverable bad states that necessitate nuking the environment and starting over.
- katzgrau 7y agoThis is the problem. The ways in which it can fail can vary (sometimes you can't even pull logs from the web interface), and if it fails during a production deploy you may be left at half or no capacity. When something inextricably effs up, rebuild the environment and more often than not, the problem dissapears.
- monkey34 7y agoI'm not sure I'd consider this a "failure," but related to GP, I have had a number of issues maintaining Elastic Beanstalk environments, including: - The single container Docker platform (not sure if this is an issue with other platforms) can cause the CloudWatch agent on the environments' EC2 instances to stop streaming logs to CloudWatch. This seems to occur when a Docker container fails, for example if the process it's managing stops (e.g., if a Node.js application triggers an exception that is not caught and exits). A new Docker container will be started, but the new container's log file sometimes does not automatically get attached to/monitored by the CloudWatch agent. - The default CloudWatch alarms created by the environment can create a "boy who cried wolf" situation. For example, when updating the application version for an environment, EB will transition the environment's state from "OK" to "Info" or even "Warning," depending on the deployment policy. This is a regular operation, but CloudWatch will still send an email to the designated notification email address about the state change. If you monitor those emails for environment issues, this normal operation could cause overload, which might lead to ignoring the emails outright. This could be problematic if the environment state transitions to an actual problem state. You can create email client rules for this, but the structure of the alarm email doesn't make this very easy, at least in Outlook 365. An annoying example of this is when your EB environment auto-scales up due to, for example, an increase in traffic. When the auto-scaling policy scales down your instances (due to normal operation of the policy), you'll get an email that your environment has transitioned into a "Warning" state because one or more of your environment's EC2 instances are being terminated. This looks scary in the CloudWatch email that is delivered, but you have to learn that it's just the ASG doing its thing, terminating unused instances as it's been configured to do. The emails, however, do not provide good context into what has led to the "Warning" state. - The way environments handle configuration files stored in your application's .ebextensions/ directory can cause inconsistent application state between version deployments on existing/new EC2 instances. For example, if your auto-scaling policy creates a new EC2 instance, but your recently deployed application version doesn't specify some of the commands/settings applied during a previous update to your .ebextensions/ files that might have been deployed to existing EC2 instances, you run the risk of having inconsistent state across your application's EC2 instances. This can be solved by using the "immutable" deployment type, but that's not the default deployment type. It's an edge case, but it's still something that requires you to SSH into your EC2 instances, and possibly manually terminate older instances when you eventually figure out what's going on. Having said all of that, I think EB is still a reasonable choice for small/beginner workloads: It gives you a number of things (automated deployment, auto-scaling, load balancing, logs, etc.) that you can get by doing things on your own, but lets you get to production quickly. For mature applications, I think you could be better off managing these individual services yourself (EB is mostly just wiring together a number of AWS services with a a few deployment and monitoring agents running on each EC2 instance). If you're comfortable with the components EB is managing for you and if you have a stable CI/CD pipeline, you can get more flexibility than bending EB against its will.
- peterwwillis 7y agoYou should use Cognito as a quick way to provide SAML/OAuth to a lot of apps, but it's not going to solve all your authentication woes. Nothing does (not even Auth0). You should use CloudFormation if it's there and you're just trying to stand something up quickly. Shell scripts with AWS CLI works too. For more robust long term stuff, use Terraform, Pulumi, or custom Boto3 code. The point is to just start using the Infrastructure as Code pattern early, not to make it perfect. Terraform is surprisingly painful after a while, but it's easier to standardize your organization on large-scale. There's no great solutions in this space. You should absolutely throw money at ElastiCache to quickly scale up and down a cache. If you're not a very experienced sysadmin (and fuck the entire tech industry for making that term a dirty word - if you are not experienced at administrating systems) you'll be spending unnecessary time and energy to stand up and maintain it. Your caching should ideally just be in your service and scaling your services themselves should be sufficient, but whatever, Redis wasn't invented for people who write good code. > But, not all data/records/events should go into Kinesis Not all of anything should go into anything. Use it if it's convenient. Lambda is very useful for batch jobs and CloudWatch-triggered maintenance tasks. I wouldn't rely on it for anything more than that; I would rather just deploy the same code to a Fargate Spot Instance and not deal with all of Lambda's bullshit.
- ufmace 7y agoI have experience with a few of these. For those, the comments seem kind of "duh" quality. So you say Kinesis is bad for message streams where each message should be handled by only one machine? Well duh, that's how it works. Kinesis is meant for every listener to be assured to get the entire stream. It was never designed for that, and I expect you'll have a bad time if you try to use it that way. SQS is obviously what you want if you're interested in guarantees that each message is handled once and only once. One Lambda per route in a decently-sized restful webapp is ugly and unmanageable? Well duh. Don't do that. Do pretty much anything else instead. Seriously, anything else at all. This sounds more like, if you want to deploy services on AWS, do either pay somebody who knows what they're doing, or spend some time checking out the services to make sure they are designed for what you want. Don't just pick a random AWS service and usage pattern from a Google search and start building around it without ever checking if it matches what you're trying to do.
- ignoramous 7y ago> SQS is obviously what you want if you're interested in guarantees that each message is handled once and only once. You mean SQS FIFO? SQS classic cannot guarantee exactly-once delivery or ingestion, either due to failure at the client or server. SQS FIFO, however; can but requires both the client and the server to work in-tandem to ensure exactly-once processing.
- awinder 7y agoAnd your queue throughput also gets destroyed by flipping that FIFO option, it’s a mess. How’s amazon’a kafka offering going?
- isugimpy 7y agoI'm not a fan of it, personally. The interface is lacking, configurability after launch is minimal at best, and the observability is EXTREMELY poor because you can't get per-topic metrics without paying extra.
- Aeolun 7y ago
- scoot_718 7y agoAthena is actively worse than hosting your own Hive on EC2. It'll timeout queries and give you no results beyond charging you. Given the variable performance of S3, that's terrible.
- aeyes 7y agoWait? Why are you comparing Athena to Hive? Athena is basically managed Presto.
- kbumsik 7y ago> CloudFormation I lack experience in AWS or clouds but I wonder what is the alternative to CF if I should avoid it? I can imagine alternatives to the rest but I cannot come up with one to CF.
- 1123581321 7y agoThe article suggests Terraform.
- acd 7y agoUse Terraform instead or create infrastructure with Pulumi using conventional programming languages such as Typescript, Javascript and Python. Terraform https://www.terraform.io/ https://www.terraform.io/ Pulumi https://www.pulumi.com/ https://www.pulumi.com/ Hand written Cloudformation is on Hold on thoughtworks technology radar. https://www.thoughtworks.com/radar/tools/handwritten-cloudformation https://www.thoughtworks.com/radar/tools/handwritten-cloudfo...
- kbumsik 7y ago> thoughtworks technology radar I've never heard of this magazine but it like a very useful curated list. Thanks for mentioning this!
- rantwasp 7y agouse CloudFormation (or Deployment Manager if on gcp). Terraform is a disaster and you’ll learn the hard way you cannot trust it and you’ll have to understand the underlying cloud no matter what you do.
- marcrosoft 7y agoThe lambda part it’s just wrong. You can use apigateway as a proxy to avoid the mess of one lambda per route.
- acd 7y agoAlso avoid EFS and please use S3 instead. Elastic Filesystem which is not that "elastic".
- rantwasp 7y agoyeah no. cloudformation is awesome and it works as advertised. I have literally not seen it fail catastrophically ever since it came out. it just works. terraform? or terrafail as it’s known. it’s a disaster. it cannot keep track of the resources it creates. when in trouble, it throws its hands up in the air and you’re on your own. do yourself a favor and don’t learn the hard way - in production - that terraform cannot take your infra from point A to point B (and rollback in case something went wrong).
- simlevesque 7y agoYeah I never got the CloudFormt1ion hate. I write them for a living without any issue.
- rantwasp 7y agoit’s not cool to use it. why use a service that’s written, ran and maintained by a dedicated team that also builds all the other services you use? (if in AWS). they don’t know what they’re doing, right? better use some randotool and talk big about Iaac when in reality the only people that I have ran across and loved Terraform either 1) used it on a really small scale or 2) don’t really have a lot of experience with the cloud. the only reason to not use Cloudformation is that you’re not using AWs. Use whatever your cloud provides. For everyone that has objections, an appeal to authority: I was building stuff on the cloud before you knew the cloud was a thing. I worked with AWS, GCP, Azure and [sad face] OpenStack extensively.
- cosmiccatnap 7y agoNothing you said was an actual argument for why you should use cloud formation or not. We don't need infmaitory Facebook-esk posts here about how a product makes you "feel"
- thanksforfish 7y agoIf you needed to setup an S3 bucket that triggers a Lambda function when an object is added, how would you do that? Last I looked, the S3 resource would get created lazily, the Lambda wouldn't get an S3 ARN, as the S3 bucket didn't exist yet and the stack would fail. It's a super obvious way to use S3 and Lambda, but the docs recommend something insane. https://aws.amazon.com/premiumsupport/knowledge-center/cloudformation-s3-notification-lambda/ https://aws.amazon.com/premiumsupport/knowledge-center/cloud... https://stackoverflow.com/questions/38752985/create-a-lambda-notification-in-an-s3-bucket-with-cloudformation https://stackoverflow.com/questions/38752985/create-a-lambda...
- alexnewman 7y agoI agree. My personal list of stuff I like using at Amazon 1. S3 +cloudfront (but I still use cloudflare for ddos) 2. Ec2 3. Pg aurora 4. Sqs is a few cases, but generally i use redis for queueing 5. Lambda for tiny notifications usually to sqs if s3 gets dinged Unlike the author I'm not afraid of lambda but generally prefer containers for apps
- zxcvbn4038 7y agoSomeone’s shotgun of frustration post, sounds like the author is mad at everything in AWS. Lambda has its uses - I definitively wouldn’t follow AWS’s SAM model and I wouldn’t try to do serve my critical web site from it - but lambda does have a convenience factor. Cloudformation isn’t perfect but it is well integrated within AWS and you can get support for it around the clock. Hashicorp quoted me $13,000 just for support for Terraform. AWS support isn’t cheap either but covers everything in AWS and their support is fantastic - like IBM in their heyday.
- dom96 7y agoAm I the only one that finds AWS' pricing structure as a whole too risky to use? Why use a service that can charge me a lot of money without being up front about it?
- cosmiccatnap 7y agoYea I think you might be. You're not wrong that they should be more up front in some ways but in general if you are a big enough company to eat a month or so of testing costs and you have a simi compitent IT/Ops group the cost savings will outweigh or at least be on par with other options over time. Combine that with the time saved in coming up with backup and deployment solutions, not to mention auditing, and it's kind of a no brainier unless your management are a bunch of seat warmers.
- caogecym 7y agoI’ve been using Okta for authentication in my SaaS app, the doc and support experience had been great for me! Free 1000 MAU. Haven’t tried Cognito or Auth0.
- catchmeifyoucan 7y agoYou can definitely use one Lambda function to serve all of the endpoints that the author mentions. You need a wildcard route in API gateway, and can easily use koa-serverless or express-serverless to get the route information when any request is received. That way you only need one function for your APIs. Even with a regular api not on lambda, you’ll still have to be aware of db connections.
- acdha 7y ago> the new YAML CF is better than the old JSON CF, but it’s still tough to read long complex stacks. These days I would recommend using CDK if you can’t switch to Terraform but I would never under any circumstances recommend YAML since the odds approach certainty that the magic in the parser will cause a problem. Beyond the usual confusion around typing I’ve seen things like significant whitespace breaking functionality[1] and the shorthand types in a few cases make diffs messier. If you find an example in YAML, it takes a second to run it through cfn—flip and you’ll never have to deal with any of that. For anyone using CF seriously, I highly recommend the cfn-python-lint tool and associated editor support. It will catch many of these cases before you burn a lengthy update cycle. 1. Fun fact: AWS’ own security alert CF stack fails the Security Hub CIS scanner because it adds extra white space the metric filters.
- karmakaze 7y agoThis would be great as a Markdown repo of "When to avoid each AWS Service." Many more than the listed services have serious usage limitations or gotchas and the listed ones have more limitations. Workarounds would also be useful to know.
- dehrmann 7y agoThere's also ones that aren't in active development like SimpleDb. It's not that it doesn't work, but you probably don't want to build anything new with it.
- karmakaze 7y agoI don't get this attitude when applied to small things. Can't it just be considered complete and not in need of enhancements? If it's not supported to the point of not fixing security issues that's completely valid.
- duskwuff 7y agoSimpleDB has some significant limitations which don't exist in its successor, DynamoDB. For example, SimpleDB tables are limited to a size of 10 GB; no such limitation exists in DynamoDB.
- karmakaze 7y agoWell stated. It's very clear to say it has a 10 GB limit rather than a vague "don't use this". SimpleDB and DynamoDB are at opposite ends of the scaling spectrum.
- dehrmann 7y agoI actually agree with this in general. One of the open questions in my mind is what's a single-product business supposed to do when its product is feature complete.
- philliphaydon 7y ago
- Yhippa 7y ago> With Cognito, they are presented with an embedded WebView, prompting the user to log into Facebook again in the embedded WebView. Off-topic, but this is an antipattern right?
- marta_morena_24 7y agoDeep analysis of the situation! Guys, avoid those 5 services haha...
- raverbashing 7y ago"What’s the problem (with Elasticache)? It is expensive, and no one notices for months." So you just click on stuff without knowing how much it costs? Really? > Secondly that cache.* prefix means this instance costs $0.216/hr instead of $0.126/hr, a 71.4% premium. Then you might think you need one for dev, qa, and prod Ok and you pick a cache of the same size for dev/qa/prod? Again, really? > But, not all data/records/events should go into Kinesis. It is not a general purpose enterprise event bus or queue. Yes, there's SQS for that. But it seems another case of clicky-go-lucky. "Hey it says queue here so we'll just use this right?!" > Lambdas are great for the following tasks: Yes, agreed. I would generalize it as: it is good for specific small tasks, especially when plugging into the rest of the AWS infrastructure (reacting to/from SQS, S3 events, etc) > Lambda is horrible for: > A replacement for REST API endpoints. Sigh General advice: if something feels weirdly hard, then you're probably doing it wrong.
- StreamBright 7y ago>> With the Lambda server-less paradigm, you end up with 1 lambda function per route. Factually incorrect. It takes as many routes as you want. I mostly ok what he is saying but some of his claims are just factually wrong.
- philwelch 7y agoTo my surprise this is mostly solid advice despite the clickbaity title. I've never used Cognito or ElasticCache, but for the other three (CloudFormation, Kinesis, and Lambda), I would agree with what he's actually saying, which is to use these services judiciously. CloudFormation has some advantages over Terraform, but having used both, Terraform is much more usable for long-lived static resources. I have yet to try Pulumi, and I don't see any reason why I should to be honest. However, CloudFormation is required for automated build and deployment of Lambda functions--both Serverless Framework and AWS SAM use CloudFormation. I've also used CloudFormation StackSets to build out infrastructure for multi-account governance. In normal use though, Terraform is typically more usable, though both solutions can get pretty hairy. By "Kinesis" the author seems to be referring to Kinesis Streams. In fairness I haven't built anything where Kinesis Streams would be a better use case than SNS/SQS, but that's just as much a statement about the projects I have happened to work on than it is about Kinesis Streams. I have used Kinesis Firehose, which provides a very scalable mechanism for, "multiple clients are intermittently spitting out a massive volume of data points, all of which need to be faithfully logged into S3 somewhere". And finally, we come to Lambda. Lambda is a good fit for use cases where either you need to run some code on an intermittent basis or when you want to prototype faster and don't want to mess around with provisioning and deploying to servers. Lambda is a great place for little pieces of glue code that get triggered on a x-minute cron or based on a CloudWatch event or from SNS or SQS. It's good for serving an HTTP API that gets less than one request per second. These use cases are extremely common and I've encountered them a lot more than the author seems to, though again maybe that's just me. But for anything high traffic, where the Lambda container is just going to stay hot, just deploy it to EC2 (perhaps with some container magic in the middle), it's cheaper in the long run.
- x3n0ph3n3 7y agoMy team has invested a sizable amount of code to manage CloudFormation. At this point, we have a rather mature interface for dealing with CloudFormation, including: - defining dependencies between stacks - taking outputs from stacks and feeding them in as parameters to other stacks (not using that awful Import/Export Value crap they implemented) - deploying as many stacks in parallel as possible and waiting for them to complete before deploying dependent stacks - dealing with common failures and rollbacks, including handling known "continue update rollback" steps with predefined resources to skip - pre/post stack create/update/delete actions to make API calls and perform other actions outside of CloudFormation We basically built the missing pieces of CloudFormation ourselves and have managed to keep CloudFormation holding the definitive state rather than managing it ourselves (or paying Terraform to do it). We have about 500 stacks for a single deployment of our product. We also implemented a large number of Transform Macro lambdas to composable templates _much_ easier. There's a lot you can do with CloudFormation, but it takes some investment. A few things about it that really drive me nuts, though: - Lagging support for new features/resources - Parameter count and parameter size limits - Certain bugs with some resource types that are slow to get fixed. (Redshift cluster password management and issues with elasticsearch resizing triggering blue/green deployment come to mind. - No ability to modify timeouts for some actions. For example, the timeout for a CustomResource is fixed and cannot be tuned -- if your Lambda never responds, CloudFormation will hang up to 2 hours. We wrote our own Lambda wrapper just to guarantee a response if an unexpected failure occurs
- samstokes 7y agoThis makes some good points about misuses of these AWS services, but the title is misleading. The article is actually more like "tempting but inadvisable use cases for AWS services". My employer uses three of these heavily (ElastiCache, Kinesis and Lambda) and we get quite a bit of leverage out of them. ElastiCache in particular surprised me. At first glance I mistook it for a transparent (and expensive) wrapper around sticking Redis on an EC2 instance, but if your usage is heavy enough to need multi-node clusters (e.g. read replicas or full Redis Cluster), its orchestration features are pretty useful. We can resize instances, fail over to a replica, and reshard clusters, with zero downtime, by clicking a button (or a one-line Terraform change). And never having to install security patches is nice too. It certainly is expensive, though. (But if you're not willing to pay a premium for managed infra, what are you doing on AWS in the first place?)
- dnprock 7y agoThere's another one that I think qualifies for this list: Elastic Container Service. It's a half baked Docker service. It seems like it's abandoned now for Kubernetes.
- shanth 7y agoAbandoned as in? I'm considering ECS for some projects. Will EKS a better sword than ECS?
- deleted 7y ago[deleted]
- vegardx 7y agoI don't really agree that it's been abandoned, nothing suggest that this is true. They're not known for just abandoning established services, there's a public roadmap[0] that you can follow. Like all services there are pros and cons. Up until very recently EKS lacked more or less everything ECS has in tight integration with other AWS services, so there's that. [0] https://github.com/aws/containers-roadmap/projects/1 https://github.com/aws/containers-roadmap/projects/1
- Sytten 7y agoUsed Kinesis in an ETL pipeline and we ended up writing our own custom client for it because we didn't want to use the Java one. The product itself is not too horrible, but the fact that they refuse to create native clients for other languages makes it a no go for any company that is not JVM based. A managed Kafka on GCP or directly a kafka on Kubernetes (there is a controller for that if you pay confluence that makes the whole thing painless) is much better.
- manigandham 7y agoYou can also use Strimzi for Kafka on Kubernetes: https://strimzi.io/ https://strimzi.io/
- calcifer 7y agoThere are excellent community KCLs for other languages, like Twitch's kinsumer [1] for Go. We've been using it in production with a high througput stream for almost a year now without any issues. [1] https://github.com/twitchscience/kinsumer https://github.com/twitchscience/kinsumer
- typenil 7y agoWith regards to Cognito, I'd recommend reading through this GitHub issue before committing to it: https://github.com/aws-amplify/amplify-js/issues/3495 https://github.com/aws-amplify/amplify-js/issues/3495 While I haven't personally had the opportunity to run into these issues, the feedback there shows a serious lack of ownership that I've never encountered elsewhere with AWS.
- pinkskip 7y agoWe use the above mentioned services and I don't agree with the assessment as we are cloud native and use lambda heavily and we think its an amazing service to use to move faster!
- anthony_barker 7y agoHmm. 1) their VPN solution is overly expensive 2) Mongodb document database similar 3) Lambda I want to use but other than the sns/sqs connected tool I agree as an api endpoint google's offering is better 4) Cognito horrible docs...
- ashtonkem 7y agoThe section on Lamba missed the biggest problem: latency and cost. For low regularity calls, Lambda suffers badly from the cold start problem. The first call to Lambda must actually create the lambda which can take hundreds of milliseconds. This problem also shows up when the level of requests exceeds the number of lambdas currently active, causing a cold start as the new lambda instance is created. It’s not uncommon to see some services inject fake requests with some regularity to insure that there are enough warm lambda instances to avoid the cold start problem, which is silly. Secondly, all lambda calls are charged by the millisecond, but the all calls are rounded up to 100ms. So if your typical lambda call is 5ms, you might be paying for 20x more time than you’re actually using. Both of these issues led my former team to use a regular ECS app rather than lambdas.
- bastawhiz 7y agoYour mileage may vary. I process tens of millions (if not hundreds of millions) of requests on Lambda each month. If you have meaningful volume, cold starts aren't a problem. And when with my volume of traffic, lambda use is a tiny fraction of my AWS bill ($30 max?). I'd be interested to know who is dissatisfied with the cost and paying more than $100/mo and what you're running (and how that would be expensive for you, considering the kind of operation you must be running). But also, if you don't have highly variable traffic, why would you use Lambda in the first place? If you have negligible traffic (enough to sit on the free tier), why not just use a single cheap EC2 instance? Lambda trades start time for lack of a server—it's shared resource utilization taken to the extreme. You're lowering your cost by letting AWS use the bare minimum to keep your service available, and that means turning your code off when it's not running. If you want to keep your code available at a hundred millisecond's notice, just have a server running. I assume most of the folks running into this just couldn't be bothered to pay the $7/mo for a hobby Heroku dyno or run a dirt cheap EC2 instance. Really interested to hear from folks that find Lambda impractical for serious use cases.
- jeremyjh 7y agoThat doesn’t sound right. A hundred million requests would cost $350 a month just in API Gateway charges.
- virmundi 7y agoYou can use one lambda to handle multiple API gateway requests. Route to ANY. Then have a bridge that connects to your favorite language router. We do this all the time with NestJS. The down side is you burn some cash when someone calls a 404 resource since your NestJS has to say that it’s missing and that requires at least 100ms of lambda time. You could probably get around this by generating a route maker off the NestJS controller files.
- timmit 7y agoI sort of disagree with the EC opinion. you pay what you get. AWS's service is always more expensive than the service you built, but at the same time more realible too
- abrookewood 7y agoI'd add Route 53 Resolver [0]. $90 a month for a few DNS queries seems completely mispriced. https://aws.amazon.com/route53/pricing/ https://aws.amazon.com/route53/pricing/
- paxys 7y agoI have no experience with Cognito and the CloudFormation point I completely agree with. The rest of them don't make sense though. ElasticCache is expensive but offers a ton of features that a single Redis process on an EC2 instance will not. And while Kinesis and Lambda are bad for the use cases called out in the article, they each fit their niche pretty well.
- toredash 7y agoAuthor misses one important point with Redis: you don't pay for traffic. If you host it yourself, you pay for traffic both way if traffic crosses AZs. Dynamic scaling, no downtime upgrades,æ. It works really well. I do hope they will get tiered RIs for ElasticCache like they have for RDS.
- Scaevus 7y agoJust like the Redis replacement, AWS's MongoDB replacement of DocumentDB comes us when you search for Mongo, but it is not an actual replacement even though the name indicates "compatibility." As a rule of thumb, we always try to avoid such proprietary services for potential vendor lock-in, whether AWS or not.
- bshanks 7y agoSummary of the article's recommendations: Don't use: - Cognito (authentication). Because: on mobile social login it isn't native. Instead: use Auth0, OneLogin, Okta or roll your own - Cloudformation (programmatically configure AWS). Because: various complexities. Instead: use Terraform. - ElastiCache (managed Redis). Because: expensive. Instead: run Docker Redis in EC2. - KINESIS (queue), as a general-purpose data queue. Good for: streaming data such as video processing/uploading. Bad for: generic data queue, because its's difficult to route each event in the queue to one of multiple workers. Kinesis is meant for every listener to be assured to get the entire stream. Instead: SNS/SQS (SQS FIFO) or a queuing framework that sits on top of Redis or traditional databases. - Lambda (server-less), to implement a REST API. Good for: Serving/Redirecting requests to CloudFront, also reacting to events from SNS or SQS by running small asynchronous tasks. Bad for: replacement for REST API endpoints, because it's too hard to work with a zillion lambdas. Instead: use a regular web framework (or, other commentators say, route all requests to one lambda and then do more routing within than lambda) I do not agree or disagree, I'm just summarizing; although I did add in some information from other comments here, too.