8 ms·

Libaws: A simpler way to declare AWS infrastructure

87 points by ithkuil 4y ago

deanCommie 4y ago
Why should it be? is Linux easy? Is Windows? AWS (as is GCP and Azure) can basically be thought of as an entire operating system, with extremely low level concepts (VPCs, EC2 machines), mid-level (Lambda, StepFunctions, EventBridge), and high-level (Translate, Comprehend, Rekognition, etc) I am not sure if this library helps. It presents its own opinionated flavour of AWS but doesn't really hide enough of the details. If you want to use AWS and not think of underlying infra like networks and security groups, build on high-level services like EventBridge and StepFunctions. Alternatively, if you don't want to think about services at all, use CDK constructs: https://docs.aws.amazon.com/cdk/v2/guide/constructs.html https://docs.aws.amazon.com/cdk/v2/guide/constructs.html For example, ehre is a NetworkLoadBalancedFargateService: https://docs.aws.amazon.com/cdk/api/v1/docs/@aws-cdk_aws-ecs-patterns.NetworkLoadBalancedFargateService.html https://docs.aws.amazon.com/cdk/api/v1/docs/@aws-cdk_aws-ecs...
- jgust 4y ago
  > Why should it be? There exists a cohort of internal customers at medium-to-large sized software companies that need to ship new applications without being bogged down by the decisions of how to get that thing running and playing nicely with the rest of the ecosystem. For the same reason they don't give their end users CRUD APIs to use their products, it doesn't always make sense to give our developers the entire suite of AWS and say "figure it out". There's need here but I'm not sure we've figured out what the right answer is.
  - masukomi 4y ago
    as an indie dev who is NOT a DevOps person i'd love more tooling that enabled me to configure AWS and have good safe defaults preset for me. Right now you really need to be a pro or spend a lot of time learning the nitty gritty configuration stuff of services you need but don't actually care about.
    - jgust 4y ago
      Or make a ton of mistakes and scramble to recover from them...
    - devonkim 4y ago
      Any decent DevOps / SRE / cloud ops type person has been striving to build something like this for a long time and has been frustrated at the kind of experience developers get out of the investment they put into all these crazy tools. The problems I’ve seen is that like most software in general there is no one size fits all approach to scaffolding and management infrastructure in AWS or any other cloud, especially when we add in the realities of business pivoting being likely. On the other hand, there are offerings such as AWS Landing Zones that sorta help if you’re running a big ol’ sprawling company but it’s totally overkill and costs more than anyone really wants to pay for if they’re just getting started. IMO, the AWS journey is probably better off for most folks using Lightsail and scaling with several of those instances making sure they can be containerized and have a brain dead easy deployment workflow that can withstand instance failures before getting more complicated. A lot of people are pushing serverless at first but it’s cumbersome enough for application developers that it’s distracting from the most important thing early on - building your dang product, silly. Otherwise I’d suggest going hard into containers and using a PaaS until it really, really hurts to where you can afford to pay the eye watering compensation of most decent SREs (expect $15k+ USD / month total spend if you have few advantages in terms of access to the market). I am not a fan of over-planning infrastructure at all but have also seen what happens when developers take too long to hire someone specialized in infrastructure and has seen the common mistakes that can gut company growth hard as they’re trying to scale up and out.
  - ghaff 4y ago
    That was sort of the idea with the various PaaS implementations out there. But invariably they either go complicated (basically transitioning to a Kubernetes-based container platform). Or they were missing some capabilities, making them less interesting to enterprises in particular.
    - jgust 4y ago
      Is there a PaaS out there that enables some guardrails? An example would be "we only want to use Fargate, don't manually provision EC2" and that option just doesn't show up as something you can self-serve.
  - pantulis 4y ago
    > a cohort of internal customers at medium-to-large sized software companies that need to ship new applications without being bogged down by the decisions Well, if it's a medium-to-large sized company they will be bogged down by governance processes anyway. Who approves the budget for this? Who signs off the cybersecurity and data protection compliance policies?
    - 015a 4y ago
      Right, and I think that strikes at the core of the issue; AWS allows you to configure things in insecure, poorly architected ways. Then they'll tell you "oh this is bad, you should fix that", as if they weren't the ones who built it to do that! The classic example is public-read S3 buckets. My favorite-er example is actually a security guidance which reads: nothing should use the default VPC (usually phrased from the perspective of, all the SGs on the default VPC should restrict all traffic). Follow that line of logic: why did you give me a default VPC then? Well, they'd argue, it makes onboarding easier, you can just deploy all these cool resources without worrying about the complexities of the networking. Wait hold on. All of this is within their power to control, they made the networking insanely complex, they made the networking layer integral to the Cool Product, they created the idea of a default VPC, they created the security rule that says its bad, and it really paints this picture of a house of cards where no-one is in control, no one has vision, its just a bunch of heads arguing with each other about their worldview on how things should be. Of course, one head yells "backwards compatibility, legacy, more knobs more knobs" and because its Amazon (and Microsoft) that head gets a megaphone. Because of this, as the OP wishes, AWS accounts are an extraordinarily useless thing to hand over to a development team. They're nothing like a Heroku or DigitalOcean account; at best, they're so complex that you need a domain expert to understand what to do, so all that ends up living in the CloudOps Org. At worst, the team will shoot themselves in the foot, and maybe take the company's bank account (or worse, customer data) with them. Thus, the New Hotness in BigBoyTech is to build a PaaS on top of AWS. Because Amazon isn't capable of doing it! Lightsail is a joke. Lambda is probably the closest they've gotten. This PaaS asserts the security, compliance, performance, architecture, etc opinions the company wants, and dev teams can build on it. Fine, whatever, but at some point you have to question why we're so obsessed with inefficiency that we tolerate knobs we won't use, or we'll miss turning to Secure, when alternatives do exist. Do you know what a compliance official's wet dream is? "Hey man I need somewhere to store this file in the cloud" "Use Google Drive" DONE! They bought Google Drive. Maybe tweaked a couple settings. Secure by default. AWS is nothing like that, and it can take millions in engineering to get an active AWS account to that state (let alone delegated management of multiple accounts across an org). It's actually collective insanity that we put up with it, when the only half-reason why it is so is because JP Morgan has a COBOL service built in 1987 that can only run if its hardwired with a telephone jack directly to an ISP interchange in Ashburn, and they're too cheap to modernize it so just tax the rest of the world by abusing Amazon's "never say No to a paying customer" policy. I'm being dramatic, but that describes 98% of the knobs on AWS.
  - unethical_ban 4y ago
    At my old job, SecDevOps built a metric fuckton of monitoring of all AWS accounts in the Organizations, as well as some pretty complex pipelines that checked the typs of infrastructure being built. Everything built in AWS was done with Terraform (minus some bootstrapping of accounts and orgs). Custom terraform modules were built which were guardrails of underlying raw Terraform providers. I'm not saying it's ideal, I'm just throwing it out there.
    - jgust 4y ago
      I'm not really familiar with terraform, I'll check it out.
- Mo3 4y ago
  100% agreed "Piloting a passenger plane should be easy" "Performing brain surgery should be easy" None of that should be easy. AWS is not meant to be focused on ease of use in the first place. It offers brutal power to those who know how to operate it and brutal ways to fuck up if you don’t. The shipping of applications mentioned by the comment in this thread is not a valid statement either.. there are countless services by now attempting exactly this (Firebase e.g), yet, demanding that AWS or GCP should be child's play to use shows a complete disregard for what they actually offer.. I feel like this is similar to saying "we want to build a spacecraft to transport satellites into space, but we'd love to just have two buttons to start and land the rocket" And then, even if you manage to abstract everything away to just those two buttons, someone will press it in the wrong moment and the rocket hits a plane flying above and you realize the expertise required is always required and someone who has this expertise doesn't need all of the abstraction in the first place.
  - vosper 4y ago
    > "Piloting a passenger plane should be easy" > "Performing brain surgery should be easy" > None of that should be easy. Why shouldn't those things be easy? Wouldn't it be better if they were? It would mean more people could do it well (I'm not saying let anyone do it!) and it would presumably have come about as a result of better understanding and tooling. Imagine if people didn't die on the waitlist for the best neurosurgeon in the country, because we'd made brain surgery easier and so now all the neurosurgeons could do that one formerly-super-difficult surgery.
    - Mo3 4y ago
      Because anyone trying to perform brain surgery still needs to have phenomenal understanding of the (brain) matter at hand. Sure, super-accurate robots to do the cuts and whatever are really cool, but I'd reckon they still have to be operated by someone who knows what to do.....
    - SkipperCat 4y ago
      If surgery was made "easy" then it would be a one size fits all and anyone with an outlier condition would not get medical care. We train doctors to have a very deep knowledge of their discipline because when they do encounter that edge case, they need to know immediately what to do. Medical care is sometimes extremely time sensitive and you don't want the doc to have to ask his superior how to stop your bleeding.
- pid-1 4y ago
  My anecdotal evidence is developers struggle big time to deploy simple stuff to AWS. "Easy things should be easy" isn't part of their philosophy. Not complaining thought, I make good money understanding AWS so others don't need to. > Why should it be? Because if they don't focus on developer experience, they might end up being treated as a commodity. Eventually someone will eat their lunch.
  - deanCommie 4y ago
    > isn't part of their philosophy. There is no one AWS philosophy. The philosophy of foundation networking services is going to be different from higher-abstraction services. They are for different customers with different needs. > Because if they don't focus on developer experience, they might end up being treated as a commodity. Eventually someone will eat their lunch. That's certainly true.
- maccard 4y ago
  > if you don't want to think about services at all, use CDK constructs This is the level that it makes sense to use AWS at if you're a small team. 10 lines of code/5 minutes and you have a deployment of https terminated load balanced containers that scale with your usage. It just works, it scales, and for small teams it saves a huge amount of time (and Dev effort - instead of focusing on setting up the correct nginx or haproxy settings I can write my app).
- rglover 4y ago
  Because they're literally wasting your existence making it overly-complicated to either justify their own egos, or, to steal your/your company's money for an inferior product. Linux is easy/well-designed/well-documented. Windows is a hackjob mess. There's a difference.
- 0xbadcafebee 4y ago
  We need AWS Distributions, like Linux distributions, and pre-made Apps that work with that distro.
SkyPuncher 4y ago
IMO, this is seeing everything as a nail when you're a hammer. AWS provides extreme flexibility. This really only becomes valuable once you hit a scale that you can optimize your infrastructure around the shape of your needs. * Need some heavy workload to run in the most cost effective data center at the most cost effective time. Yep, AWS is great for that. * Need a specialized GPU set. Yep, AWS is great for that. * Need some crazy storage or memory setup. Yep, AWS is great for that. For everything else, you should really consider using a PaaS with a good ol' Docker container.
- omegalulw 4y ago
  > AWS provides extreme flexibility. This really only becomes valuable once you hit a scale that you can optimize your infrastructure around the shape of your needs. This is upside down? When you hit the scale where you get value from modelling your infra around your business use cases that's when you go on-prem or use services that let you rent bare metal. Before that you use cloud.
- onphonenow 4y ago
  Docker containers are pretty simple, and I've got AWS App Runner / Fargate / ECS / ECS Anywhere to work pretty well for them. You get something working in a docker, push to your AWS repo, and App Runner grabs it and goes.
- thomaslord 4y ago
  It's interesting to see everyone's different views on Docker. Here you (at least to my reading) suggest that people who need simplicity should just chuck their app in a Docker container, but elsewhere I've seen it portrayed as something you shouldn't touch until you have an infrastructure team. Maybe I'm interpreting this wrong and the HN-consensus path is something like bare VPS => Docker => use AWS directly and tweak everything how you want it?
- scarface74 4y ago
  Yes because Kubernetes is simple…
bvanderveen 4y ago
> X should be easy > proposes solution that involves YAML Am I crazy, ignorant, or is YAML the most tedious and error-prone format edit?
- Marazan 4y ago
  I have grown to loathe YAML over the years. It is a giant foot gun of a mess of syntax. I lile python, I like significant whitespace, I hate how YAML does it.
- mountainriver 4y ago
  Yes! YAML is great at visualizing data structures or producing small configs. The way it’s being used today in devops is terrible
- shitlord 4y ago
  Agreed, trial & error with YAML sucks. People hated it so much that Amazon created CDK, a tool that basically generates/executes Cloudformation templates.
  - tormeh 4y ago
    Using a proper programming language seems to be the obvious end-game of configuration. Everything as code. Not quite sure why we keep trying and failing to use non-Turing complete configuration languages.
    - gjvc 4y ago
      ...and doing that allows one to interface to existing sources of truth for inventory and whatnot.
      - gtirloni 4y ago
        Just like with every other tool, right? Ansible dynamic inventories, Terraform plugins, etc.
      - 4y ago
kristianpaul 4y ago
I think AWS is getting there, but hiding either with the cost of hiding complexity with tools like AWS CDK or even actually with services like AWS Lightsail designed for people that doesn't care about building blocks (ec2, s3, route53, sqs, sns, etc...)
jsiaajdsdaa 4y ago
AWS is very easy for me when all I have to do is launch ec2 instances ^.^
OJFord 4y ago
Terraform modules? In particular, https://github.com/terraform-aws-modules/ https://github.com/terraform-aws-modules/ is a better solution for anyone looking for this IMO. (Yes there is a comparison to 'terraform' in the Readme - I'm not convinced it's considering modules, versus just using the AWS provider directly. Another bonus of using tf modules over this: you can eventually realise you need to graduate to using the AWS provider directly, and do so quite easily (cf. `terraform state mv`).)
- mountainriver 4y ago
  Or better yet Pulumi AWS https://www.pulumi.com/docs/get-started/aws/ https://www.pulumi.com/docs/get-started/aws/
  - emptysongglass 4y ago
    Or even better, Crossplane for auto-reconciling infra: https://aws.amazon.com/blogs/opensource/introducing-aws-blueprints-for-crossplane/ https://aws.amazon.com/blogs/opensource/introducing-aws-blue...
- freeqaz 4y ago
  We've had a lot of success using the AWS CDK (our code[0]). We wrote a tonnn of stuff with Terraform back we were building a low-code platform, but it was a ton of work to manage. The AWS CDK has higher level "constructs"[1] that are really nice to use. How does that compare to the state of the world for things like Pulumi or Terraform these days? 0: https://github.com/lunasec-io/lunasec/tree/master/lunatrace/bsl/backend-cdk/lib https://github.com/lunasec-io/lunasec/tree/master/lunatrace/... Any of these files represents the infrastructure. The "bin" folder is the actual entry point. 1: https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_ecs_patterns.ApplicationLoadBalancedFargateService.html https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_... We're using this as well as the Fargate SQS worker one to scale up backend job processing. Lambda is a pain and this works quite magically.
  - gtirloni 4y ago
NathanKP 4y ago
Some feedback on this idea from someone who works at AWS and has worked on similar tools to make AWS easier, such as AWS Copilot (https://aws.amazon.com/blogs/containers/introducing-aws-copilot/ https://aws.amazon.com/blogs/containers/introducing-aws-copi...): The hard part isn't making AWS easier, it is making a tool which is both easy and flexible enough that a user can do what they need (within reason), then outgrow your tool and move on. In specific any tool based abstraction has the downside that if the only way to interact with your infrastructure is through the tool then you severely limit the options for a user to make customizations. AWS will add features faster than you can keep up, and there will always be someone who can't use your tool unless you have feature X, Y, and Z. At that point you can either add more and more features until your tool becomes as complex to use as all of AWS itself, or you can draw a line in the sand and say "you have outgrown my tool, it is time to move on". When they reach that point you should try to make it easy for them to avoid having to recreate everything from scratch themselves. For AWS Copilot we tackled this problem in part by adding a "copilot svc package" command which spits out a full CloudFormation stack for your application and all its infrastructure, so that you don't have to restart from scratch if you decide you have outgrown Copilot. Copilot will help you deploy simple Docker based container applications quickly and easily, but then if you have more complex needs you can "graduate" on to the next experience by exporting things up into a stable, productionized CloudFormation template that lets you customize the full range of AWS settings and resources to your hearts content. I think your tooling will need something similar. The "outgrowing" section (https://github.com/nathants/libaws#outgrowing https://github.com/nathants/libaws#outgrowing) is a bit weak right now, and that's what you need to really nail down for people to have the trust to start adopting a tool for serious usage. They need to know that once they outgrow the tool they will have a viable path that won't be a huge headache. Otherwise I like your ideas here, and I think the YAML based DSL here is a fascinating abstraction of many common, simple use cases. Great job!
- pojzon 4y ago
  In case of cloud providers its quite often that Terraform Providers are updated faster by dedicated teams than their own Cloud solutions. Cloudformation or AWS CDK are upgraded slower than providers for example. So this argument is mot.
datalopers 4y ago
is this a new infrastructure-as-code/config attempt?
cptaj 4y ago
I am convinced that AWS is intentionally obtuse in order to promote job security for engineers and thus be the preferred platform
- sharpy 4y ago
  As someone who has worked within AWS, and also used AWS as a customer at multiple companies, I totally understand that sentiment. But I wouldn't want to solve some of the problems myself, rather than using their services.
  - tluyben2 4y ago
    > But I wouldn't want to solve some of the problems myself, rather than using their services. Sure, but vanilla setups should be trivial and optimised for cost. That is not to their or their consultants’ benefit. I mean, whenever I asked someone ‘how do I do this in aws’, they tell me ‘oh that is a bog standard blah setup, very Simple’. So if that is simple, why is it so hard to set up as a novice (with 20 years of rack server bare metal hosting experience). I had to learn everything about aws in order to figure out that all experts saying things are ‘simple’ were still not providing optimal solutions for my vanille, and for decades unchanged across many different applications, hosting wishes. If you don’t know literally everything about aws and hired a consultant or a team to set it up, you are probably (massively) overpaying. And often, as I see at clients, for vanilla stuff that should be 1 click setup. But yes, I guess tool providers should provide this on top, not aws itself.
- wnolens 4y ago
  A lot of effort is put into the customer experience, however often it's by the feature devs. So it's good once you grok it. But obtuse to start and not good across products. It really needs some love from a higher level thinker who's not steeped in the solution, but in the problem. Presumably this is a PM but there's just not a lot of good ones in the org to be frank. They're almost not worth consulting.
- tluyben2 4y ago
  My experience as well; I had to become really intimate with aws over the years because I noticed more and more that consultants we hire (we don’t need fulltime) for aws or other cloud setups, pull wool over your eyes by throwing out terms (you generally wouldn’t know outside these setups), saying the word ‘security’ a lot and showing extremely complex deployment charts which look great but contain many things you didn’t know you needed and probably do not need. I cannot count the times I found 1000-10000$/month of unneeded stuff at clients in these setups. First it was just ec2 (‘in the beginning’) which was more expensive than our vps/baremetal setups, but that price jumped up for only marginally more complex setups and so I decided to learn it all and see what was happening. Almost never justified in my experience.
lnwlebjel 4y ago
As stated earlier this week, AWS's CF is your opportunity. Seriously, there is a business opportunity here. Make a multi-cloud interface that is easy to use, prevents lock-in, has reasonable defaults ... please?
- FredPret 4y ago
  You'll always be beholden to egress costs
- pojzon 4y ago
  Orchestrations that promise they are “cloud-agnostic” like Kubernetes in fact are also bound to vendor lock-ins due to services they are run on. This makes a pretty sound argument that “cloud-agnostic” solution is never going to be built. Its bad for cloud-providers business and so they will never unify.
0xbadcafebee 4y ago
Saying AWS should be easy is like saying a Lego parts bin should be easy. Easy for what? Easy to build a robot monkey with? Even if legos are really easy to snap together, you still have to do the work of putting them together in the shape of a robot monkey. This project seems to be a collection of pre-assembled lego parts, that can eventually be put together in the form of a robot monkey. But I think all of us actually just want the robot monkey, not several clumps of legos. We should have more feature-complete projects that we can push a button and it pops up for you. But that's very different than saying AWS should be easy. I think we should stop obsessing over "code", and start focusing on robot monkeys. In other words, stop thinking you need more and more and more abstractions around calling AWS APIs, and focus more on publishing a complete working solution for a specific problem. I don't give a shit about SQS queues and CloudWatch alerts and Lambdas and IAM role policies! I give a shit about a file-upload-website that automatically triggers a python program to process a file and deliver the result into an S3 bucket or Postgres database. That complete thing is the robot monkey. That's what we should be publishing; not a million lines of boilerplate that I still have to spend time and energy cobbling together into a robot monkey. I have built the same robot monkey over and over and over again in 5 different abstractions. I don't care what the abstraction is, I just want the robot monkey! I think the reason we don't have more robot monkeys is that they're built within companies out of "code", and so we aren't allowed to open-source them, because it's "intellectual property". We need a new abstraction that is not code, and is not considered super secret proprietary information, so we can publish it as open source without thinking. For example: what if you could point a tool at some part of your AWS account, and it exports a declarative configuration file that is a snapshot of how everything is plumbed together? The tool could strip out any "literals" and anonymize all strings, so that there is nothing proprietary in the end result except a lot of API calls to functions with random names. The end result would essentially be an unlabeled architectural diagram that you could execute. You could then publish that on the Robot Monkey Database with a description of what it does.
billllll 4y ago
I'm a bit lost as to how learning a new infrastructure-as-code tool makes AWS easier and more "fun." At best, you're still exposed to all the same complexities of the different AWS services, but now you gotta learn this new tool. I do sympathize with the idea though, as I was recently trying to deploy a container for a small project, and didn't want to deal with all the complexities a fully featured cloud provider has. I found AWS Lightsail, GCP Cloud Run, and later settled on fly.io. It's hard for me to imagine using this tool from the beginning, or even graduating to this tool instead of upgrading to plain AWS and maybe Terraform.
gkoberger 4y ago
I don't think AWS should be easy, any more than a compiler should be simple. It's a different layer of the stack. I do think most people should use Heroku/Render/etc on top of AWS, and I bet AWS agrees... because either way, AWS is making money. AWS is a low-level tool, and that's okay.
r00tanon 4y ago
A lot of Go code using the Go AWS SDK that's useful as a non-trivial example in and of itself. Is this IaC "light" for purposes of setting up and tearing down small applications for experiments or demos? Obviously a lot of effort spent. Of course, once you know AWS well enough to "make it easy", by that time, you've found the need for tooling which supports the harder stuff. And, frankly, someone new to AWS needs a lot of context just to understand the canonical ways to wire stuff together in AWS to even comprehend which pattern will be useful. For example, S3 events are really backed by SQS under the covers, with implied retries on failure, but no DLQ support, and a lambda that triggers on an arriving S3 event may need to account for "duplicate" events, etc. That's just one wrinkle that even a simplified configuration might require more complex understanding of the underlying AWS ecosystem.
pbalau 4y ago
There is no tool in the world that will make you good at something you don't know. If you allow me a story, I sail on a kind of boat, called a squib. For the foresail, eg the sail that is in front of the mast, it has the regular sheets, a Cunningham, a halyard tensioner and barber haulers. The squib is a very easy boat to sail, notoriously hard to sail fast. Complaining about AWS being complicated, is the same as complaining the squib is complicated, just because it gives you options. I don't have to use all those controls, I still can get from a to b without using them. But if I want to win the race, I have to understand what they do and use them appropriately.
- aaaaaaaaata 4y ago
  Sometimes the race is "not burning your whole raise".
tomatowurst 4y ago
It's funny to me why people aren't aware of CDK. It's literally no different than writing code. You don't have to depend on anyone, you can customize this to no end, its part of your codebase. from aws_cdk import aws_s3 as s3,naws_iam as iam,bcore class S3Template(core.Stack): def __init__(self, app: core.App, id: str, **kwargs) -> None: super().__init__(app, id) myBucket = s3.Bucket(self, 'MyFirstBucket', bucket_name='balkaran-aws-cdk-s3-demo-bucket') myBucket.add_to_resource_policy( iam.PolicyStatement( actions=['s3:GetObject'], resources= [myBucket.arn_for_objects('*')], principals= [iam.AccountPrincipal(account_id=core.Aws.ACCOUNT_ID)] ) ) myUser = iam.User(self,'s3User') myBucket.grant_write(myUser) app = core.App() S3Template(app, "S3Template", env={'region':'us-east-1'}) app.synth()
- battle_hardened 4y ago
  Careful, sarcasm is hard to detect online
  - tomatowurst 4y ago
    are you sure you can offline?
vivegi 4y ago
I have used AWS CDK for IaC and for the most part, it works great. It also has escape hatches for constructs that are not yet implemented in the CDK API that you could fall back on CloudFormation templates or whip out a custom resource, if you are so inclined. For the most part, the constructs provided out of the box has good defaults and the documentation helps. The aws-cdk support team is also responsive on issues/bugs reported. This is just based on my experience and I started off as a total noob on AWS.

87 points by ithkuil 4y ago

Why should it be? is Linux easy? Is Windows? AWS (as is GCP and Azure) can basically be thought of as an entire operating system, with extremely low level concepts (VPCs, EC2 machines), mid-level (Lambda, StepFunctions, EventBridge), and high-level (Translate, Comprehend, Rekognition, etc) I am not sure if this library helps. It presents its own opinionated flavour of AWS but doesn't really hide enough of the details. If you want to use AWS and not think of underlying infra like networks and security groups, build on high-level services like EventBridge and StepFunctions. Alternatively, if you don't want to think about services at all, use CDK constructs: https://docs.aws.amazon.com/cdk/v2/guide/constructs.html https://docs.aws.amazon.com/cdk/v2/guide/constructs.html For example, ehre is a NetworkLoadBalancedFargateService: https://docs.aws.amazon.com/cdk/api/v1/docs/@aws-cdk_aws-ecs-patterns.NetworkLoadBalancedFargateService.html https://docs.aws.amazon.com/cdk/api/v1/docs/@aws-cdk_aws-ecs...

jgust 4y ago
> Why should it be? There exists a cohort of internal customers at medium-to-large sized software companies that need to ship new applications without being bogged down by the decisions of how to get that thing running and playing nicely with the rest of the ecosystem. For the same reason they don't give their end users CRUD APIs to use their products, it doesn't always make sense to give our developers the entire suite of AWS and say "figure it out". There's need here but I'm not sure we've figured out what the right answer is.
- masukomi 4y ago
  as an indie dev who is NOT a DevOps person i'd love more tooling that enabled me to configure AWS and have good safe defaults preset for me. Right now you really need to be a pro or spend a lot of time learning the nitty gritty configuration stuff of services you need but don't actually care about.
  - jgust 4y ago
    Or make a ton of mistakes and scramble to recover from them...
  - devonkim 4y ago
    Any decent DevOps / SRE / cloud ops type person has been striving to build something like this for a long time and has been frustrated at the kind of experience developers get out of the investment they put into all these crazy tools. The problems I’ve seen is that like most software in general there is no one size fits all approach to scaffolding and management infrastructure in AWS or any other cloud, especially when we add in the realities of business pivoting being likely. On the other hand, there are offerings such as AWS Landing Zones that sorta help if you’re running a big ol’ sprawling company but it’s totally overkill and costs more than anyone really wants to pay for if they’re just getting started. IMO, the AWS journey is probably better off for most folks using Lightsail and scaling with several of those instances making sure they can be containerized and have a brain dead easy deployment workflow that can withstand instance failures before getting more complicated. A lot of people are pushing serverless at first but it’s cumbersome enough for application developers that it’s distracting from the most important thing early on - building your dang product, silly. Otherwise I’d suggest going hard into containers and using a PaaS until it really, really hurts to where you can afford to pay the eye watering compensation of most decent SREs (expect $15k+ USD / month total spend if you have few advantages in terms of access to the market). I am not a fan of over-planning infrastructure at all but have also seen what happens when developers take too long to hire someone specialized in infrastructure and has seen the common mistakes that can gut company growth hard as they’re trying to scale up and out.
- ghaff 4y ago
  That was sort of the idea with the various PaaS implementations out there. But invariably they either go complicated (basically transitioning to a Kubernetes-based container platform). Or they were missing some capabilities, making them less interesting to enterprises in particular.
  - jgust 4y ago
    Is there a PaaS out there that enables some guardrails? An example would be "we only want to use Fargate, don't manually provision EC2" and that option just doesn't show up as something you can self-serve.
- pantulis 4y ago
  > a cohort of internal customers at medium-to-large sized software companies that need to ship new applications without being bogged down by the decisions Well, if it's a medium-to-large sized company they will be bogged down by governance processes anyway. Who approves the budget for this? Who signs off the cybersecurity and data protection compliance policies?
  - 015a 4y ago
    Right, and I think that strikes at the core of the issue; AWS allows you to configure things in insecure, poorly architected ways. Then they'll tell you "oh this is bad, you should fix that", as if they weren't the ones who built it to do that! The classic example is public-read S3 buckets. My favorite-er example is actually a security guidance which reads: nothing should use the default VPC (usually phrased from the perspective of, all the SGs on the default VPC should restrict all traffic). Follow that line of logic: why did you give me a default VPC then? Well, they'd argue, it makes onboarding easier, you can just deploy all these cool resources without worrying about the complexities of the networking. Wait hold on. All of this is within their power to control, they made the networking insanely complex, they made the networking layer integral to the Cool Product, they created the idea of a default VPC, they created the security rule that says its bad, and it really paints this picture of a house of cards where no-one is in control, no one has vision, its just a bunch of heads arguing with each other about their worldview on how things should be. Of course, one head yells "backwards compatibility, legacy, more knobs more knobs" and because its Amazon (and Microsoft) that head gets a megaphone. Because of this, as the OP wishes, AWS accounts are an extraordinarily useless thing to hand over to a development team. They're nothing like a Heroku or DigitalOcean account; at best, they're so complex that you need a domain expert to understand what to do, so all that ends up living in the CloudOps Org. At worst, the team will shoot themselves in the foot, and maybe take the company's bank account (or worse, customer data) with them. Thus, the New Hotness in BigBoyTech is to build a PaaS on top of AWS. Because Amazon isn't capable of doing it! Lightsail is a joke. Lambda is probably the closest they've gotten. This PaaS asserts the security, compliance, performance, architecture, etc opinions the company wants, and dev teams can build on it. Fine, whatever, but at some point you have to question why we're so obsessed with inefficiency that we tolerate knobs we won't use, or we'll miss turning to Secure, when alternatives do exist. Do you know what a compliance official's wet dream is? "Hey man I need somewhere to store this file in the cloud" "Use Google Drive" DONE! They bought Google Drive. Maybe tweaked a couple settings. Secure by default. AWS is nothing like that, and it can take millions in engineering to get an active AWS account to that state (let alone delegated management of multiple accounts across an org). It's actually collective insanity that we put up with it, when the only half-reason why it is so is because JP Morgan has a COBOL service built in 1987 that can only run if its hardwired with a telephone jack directly to an ISP interchange in Ashburn, and they're too cheap to modernize it so just tax the rest of the world by abusing Amazon's "never say No to a paying customer" policy. I'm being dramatic, but that describes 98% of the knobs on AWS.
- unethical_ban 4y ago
  At my old job, SecDevOps built a metric fuckton of monitoring of all AWS accounts in the Organizations, as well as some pretty complex pipelines that checked the typs of infrastructure being built. Everything built in AWS was done with Terraform (minus some bootstrapping of accounts and orgs). Custom terraform modules were built which were guardrails of underlying raw Terraform providers. I'm not saying it's ideal, I'm just throwing it out there.
  - jgust 4y ago
    I'm not really familiar with terraform, I'll check it out.
Mo3 4y ago
100% agreed "Piloting a passenger plane should be easy" "Performing brain surgery should be easy" None of that should be easy. AWS is not meant to be focused on ease of use in the first place. It offers brutal power to those who know how to operate it and brutal ways to fuck up if you don’t. The shipping of applications mentioned by the comment in this thread is not a valid statement either.. there are countless services by now attempting exactly this (Firebase e.g), yet, demanding that AWS or GCP should be child's play to use shows a complete disregard for what they actually offer.. I feel like this is similar to saying "we want to build a spacecraft to transport satellites into space, but we'd love to just have two buttons to start and land the rocket" And then, even if you manage to abstract everything away to just those two buttons, someone will press it in the wrong moment and the rocket hits a plane flying above and you realize the expertise required is always required and someone who has this expertise doesn't need all of the abstraction in the first place.
- vosper 4y ago
  > "Piloting a passenger plane should be easy" > "Performing brain surgery should be easy" > None of that should be easy. Why shouldn't those things be easy? Wouldn't it be better if they were? It would mean more people could do it well (I'm not saying let anyone do it!) and it would presumably have come about as a result of better understanding and tooling. Imagine if people didn't die on the waitlist for the best neurosurgeon in the country, because we'd made brain surgery easier and so now all the neurosurgeons could do that one formerly-super-difficult surgery.
  - Mo3 4y ago
    Because anyone trying to perform brain surgery still needs to have phenomenal understanding of the (brain) matter at hand. Sure, super-accurate robots to do the cuts and whatever are really cool, but I'd reckon they still have to be operated by someone who knows what to do.....
  - SkipperCat 4y ago
    If surgery was made "easy" then it would be a one size fits all and anyone with an outlier condition would not get medical care. We train doctors to have a very deep knowledge of their discipline because when they do encounter that edge case, they need to know immediately what to do. Medical care is sometimes extremely time sensitive and you don't want the doc to have to ask his superior how to stop your bleeding.
pid-1 4y ago
My anecdotal evidence is developers struggle big time to deploy simple stuff to AWS. "Easy things should be easy" isn't part of their philosophy. Not complaining thought, I make good money understanding AWS so others don't need to. > Why should it be? Because if they don't focus on developer experience, they might end up being treated as a commodity. Eventually someone will eat their lunch.
- deanCommie 4y ago
  > isn't part of their philosophy. There is no one AWS philosophy. The philosophy of foundation networking services is going to be different from higher-abstraction services. They are for different customers with different needs. > Because if they don't focus on developer experience, they might end up being treated as a commodity. Eventually someone will eat their lunch. That's certainly true.
maccard 4y ago
> if you don't want to think about services at all, use CDK constructs This is the level that it makes sense to use AWS at if you're a small team. 10 lines of code/5 minutes and you have a deployment of https terminated load balanced containers that scale with your usage. It just works, it scales, and for small teams it saves a huge amount of time (and Dev effort - instead of focusing on setting up the correct nginx or haproxy settings I can write my app).
rglover 4y ago
Because they're literally wasting your existence making it overly-complicated to either justify their own egos, or, to steal your/your company's money for an inferior product. Linux is easy/well-designed/well-documented. Windows is a hackjob mess. There's a difference.
0xbadcafebee 4y ago
We need AWS Distributions, like Linux distributions, and pre-made Apps that work with that distro.

SkyPuncher 4y ago

IMO, this is seeing everything as a nail when you're a hammer. AWS provides extreme flexibility. This really only becomes valuable once you hit a scale that you can optimize your infrastructure around the shape of your needs. * Need some heavy workload to run in the most cost effective data center at the most cost effective time. Yep, AWS is great for that. * Need a specialized GPU set. Yep, AWS is great for that. * Need some crazy storage or memory setup. Yep, AWS is great for that. For everything else, you should really consider using a PaaS with a good ol' Docker container.

omegalulw 4y ago
> AWS provides extreme flexibility. This really only becomes valuable once you hit a scale that you can optimize your infrastructure around the shape of your needs. This is upside down? When you hit the scale where you get value from modelling your infra around your business use cases that's when you go on-prem or use services that let you rent bare metal. Before that you use cloud.
onphonenow 4y ago
Docker containers are pretty simple, and I've got AWS App Runner / Fargate / ECS / ECS Anywhere to work pretty well for them. You get something working in a docker, push to your AWS repo, and App Runner grabs it and goes.
thomaslord 4y ago
It's interesting to see everyone's different views on Docker. Here you (at least to my reading) suggest that people who need simplicity should just chuck their app in a Docker container, but elsewhere I've seen it portrayed as something you shouldn't touch until you have an infrastructure team. Maybe I'm interpreting this wrong and the HN-consensus path is something like bare VPS => Docker => use AWS directly and tweak everything how you want it?
scarface74 4y ago
Yes because Kubernetes is simple…

bvanderveen 4y ago

> X should be easy > proposes solution that involves YAML Am I crazy, ignorant, or is YAML the most tedious and error-prone format edit?

Marazan 4y ago
I have grown to loathe YAML over the years. It is a giant foot gun of a mess of syntax. I lile python, I like significant whitespace, I hate how YAML does it.
mountainriver 4y ago
Yes! YAML is great at visualizing data structures or producing small configs. The way it’s being used today in devops is terrible
shitlord 4y ago
Agreed, trial & error with YAML sucks. People hated it so much that Amazon created CDK, a tool that basically generates/executes Cloudformation templates.
- tormeh 4y ago
  Using a proper programming language seems to be the obvious end-game of configuration. Everything as code. Not quite sure why we keep trying and failing to use non-Turing complete configuration languages.
  - gjvc 4y ago
    ...and doing that allows one to interface to existing sources of truth for inventory and whatnot.
    - gtirloni 4y ago
      Just like with every other tool, right? Ansible dynamic inventories, Terraform plugins, etc.
    - 4y ago

kristianpaul 4y ago

I think AWS is getting there, but hiding either with the cost of hiding complexity with tools like AWS CDK or even actually with services like AWS Lightsail designed for people that doesn't care about building blocks (ec2, s3, route53, sqs, sns, etc...)

jsiaajdsdaa 4y ago

AWS is very easy for me when all I have to do is launch ec2 instances ^.^

OJFord 4y ago

Terraform modules? In particular, https://github.com/terraform-aws-modules/ https://github.com/terraform-aws-modules/ is a better solution for anyone looking for this IMO. (Yes there is a comparison to 'terraform' in the Readme - I'm not convinced it's considering modules, versus just using the AWS provider directly. Another bonus of using tf modules over this: you can eventually realise you need to graduate to using the AWS provider directly, and do so quite easily (cf. `terraform state mv`).)

mountainriver 4y ago
Or better yet Pulumi AWS https://www.pulumi.com/docs/get-started/aws/ https://www.pulumi.com/docs/get-started/aws/
- emptysongglass 4y ago
  Or even better, Crossplane for auto-reconciling infra: https://aws.amazon.com/blogs/opensource/introducing-aws-blueprints-for-crossplane/ https://aws.amazon.com/blogs/opensource/introducing-aws-blue...
freeqaz 4y ago
We've had a lot of success using the AWS CDK (our code[0]). We wrote a tonnn of stuff with Terraform back we were building a low-code platform, but it was a ton of work to manage. The AWS CDK has higher level "constructs"[1] that are really nice to use. How does that compare to the state of the world for things like Pulumi or Terraform these days? 0: https://github.com/lunasec-io/lunasec/tree/master/lunatrace/bsl/backend-cdk/lib https://github.com/lunasec-io/lunasec/tree/master/lunatrace/... Any of these files represents the infrastructure. The "bin" folder is the actual entry point. 1: https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_ecs_patterns.ApplicationLoadBalancedFargateService.html https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_... We're using this as well as the Fargate SQS worker one to scale up backend job processing. Lambda is a pain and this works quite magically.
- gtirloni 4y ago

NathanKP 4y ago

Some feedback on this idea from someone who works at AWS and has worked on similar tools to make AWS easier, such as AWS Copilot (https://aws.amazon.com/blogs/containers/introducing-aws-copilot/ https://aws.amazon.com/blogs/containers/introducing-aws-copi...): The hard part isn't making AWS easier, it is making a tool which is both easy and flexible enough that a user can do what they need (within reason), then outgrow your tool and move on. In specific any tool based abstraction has the downside that if the only way to interact with your infrastructure is through the tool then you severely limit the options for a user to make customizations. AWS will add features faster than you can keep up, and there will always be someone who can't use your tool unless you have feature X, Y, and Z. At that point you can either add more and more features until your tool becomes as complex to use as all of AWS itself, or you can draw a line in the sand and say "you have outgrown my tool, it is time to move on". When they reach that point you should try to make it easy for them to avoid having to recreate everything from scratch themselves. For AWS Copilot we tackled this problem in part by adding a "copilot svc package" command which spits out a full CloudFormation stack for your application and all its infrastructure, so that you don't have to restart from scratch if you decide you have outgrown Copilot. Copilot will help you deploy simple Docker based container applications quickly and easily, but then if you have more complex needs you can "graduate" on to the next experience by exporting things up into a stable, productionized CloudFormation template that lets you customize the full range of AWS settings and resources to your hearts content. I think your tooling will need something similar. The "outgrowing" section (https://github.com/nathants/libaws#outgrowing https://github.com/nathants/libaws#outgrowing) is a bit weak right now, and that's what you need to really nail down for people to have the trust to start adopting a tool for serious usage. They need to know that once they outgrow the tool they will have a viable path that won't be a huge headache. Otherwise I like your ideas here, and I think the YAML based DSL here is a fascinating abstraction of many common, simple use cases. Great job!

pojzon 4y ago
In case of cloud providers its quite often that Terraform Providers are updated faster by dedicated teams than their own Cloud solutions. Cloudformation or AWS CDK are upgraded slower than providers for example. So this argument is mot.

datalopers 4y ago

is this a new infrastructure-as-code/config attempt?

cptaj 4y ago

I am convinced that AWS is intentionally obtuse in order to promote job security for engineers and thus be the preferred platform

sharpy 4y ago
As someone who has worked within AWS, and also used AWS as a customer at multiple companies, I totally understand that sentiment. But I wouldn't want to solve some of the problems myself, rather than using their services.
- tluyben2 4y ago
  > But I wouldn't want to solve some of the problems myself, rather than using their services. Sure, but vanilla setups should be trivial and optimised for cost. That is not to their or their consultants’ benefit. I mean, whenever I asked someone ‘how do I do this in aws’, they tell me ‘oh that is a bog standard blah setup, very Simple’. So if that is simple, why is it so hard to set up as a novice (with 20 years of rack server bare metal hosting experience). I had to learn everything about aws in order to figure out that all experts saying things are ‘simple’ were still not providing optimal solutions for my vanille, and for decades unchanged across many different applications, hosting wishes. If you don’t know literally everything about aws and hired a consultant or a team to set it up, you are probably (massively) overpaying. And often, as I see at clients, for vanilla stuff that should be 1 click setup. But yes, I guess tool providers should provide this on top, not aws itself.
wnolens 4y ago
A lot of effort is put into the customer experience, however often it's by the feature devs. So it's good once you grok it. But obtuse to start and not good across products. It really needs some love from a higher level thinker who's not steeped in the solution, but in the problem. Presumably this is a PM but there's just not a lot of good ones in the org to be frank. They're almost not worth consulting.
tluyben2 4y ago
My experience as well; I had to become really intimate with aws over the years because I noticed more and more that consultants we hire (we don’t need fulltime) for aws or other cloud setups, pull wool over your eyes by throwing out terms (you generally wouldn’t know outside these setups), saying the word ‘security’ a lot and showing extremely complex deployment charts which look great but contain many things you didn’t know you needed and probably do not need. I cannot count the times I found 1000-10000$/month of unneeded stuff at clients in these setups. First it was just ec2 (‘in the beginning’) which was more expensive than our vps/baremetal setups, but that price jumped up for only marginally more complex setups and so I decided to learn it all and see what was happening. Almost never justified in my experience.

lnwlebjel 4y ago

As stated earlier this week, AWS's CF is your opportunity. Seriously, there is a business opportunity here. Make a multi-cloud interface that is easy to use, prevents lock-in, has reasonable defaults ... please?

FredPret 4y ago
You'll always be beholden to egress costs
pojzon 4y ago
Orchestrations that promise they are “cloud-agnostic” like Kubernetes in fact are also bound to vendor lock-ins due to services they are run on. This makes a pretty sound argument that “cloud-agnostic” solution is never going to be built. Its bad for cloud-providers business and so they will never unify.

0xbadcafebee 4y ago

Saying AWS should be easy is like saying a Lego parts bin should be easy. Easy for what? Easy to build a robot monkey with? Even if legos are really easy to snap together, you still have to do the work of putting them together in the shape of a robot monkey. This project seems to be a collection of pre-assembled lego parts, that can eventually be put together in the form of a robot monkey. But I think all of us actually just want the robot monkey, not several clumps of legos. We should have more feature-complete projects that we can push a button and it pops up for you. But that's very different than saying AWS should be easy. I think we should stop obsessing over "code", and start focusing on robot monkeys. In other words, stop thinking you need more and more and more abstractions around calling AWS APIs, and focus more on publishing a complete working solution for a specific problem. I don't give a shit about SQS queues and CloudWatch alerts and Lambdas and IAM role policies! I give a shit about a file-upload-website that automatically triggers a python program to process a file and deliver the result into an S3 bucket or Postgres database. That complete thing is the robot monkey. That's what we should be publishing; not a million lines of boilerplate that I still have to spend time and energy cobbling together into a robot monkey. I have built the same robot monkey over and over and over again in 5 different abstractions. I don't care what the abstraction is, I just want the robot monkey! I think the reason we don't have more robot monkeys is that they're built within companies out of "code", and so we aren't allowed to open-source them, because it's "intellectual property". We need a new abstraction that is not code, and is not considered super secret proprietary information, so we can publish it as open source without thinking. For example: what if you could point a tool at some part of your AWS account, and it exports a declarative configuration file that is a snapshot of how everything is plumbed together? The tool could strip out any "literals" and anonymize all strings, so that there is nothing proprietary in the end result except a lot of API calls to functions with random names. The end result would essentially be an unlabeled architectural diagram that you could execute. You could then publish that on the Robot Monkey Database with a description of what it does.

billllll 4y ago

I'm a bit lost as to how learning a new infrastructure-as-code tool makes AWS easier and more "fun." At best, you're still exposed to all the same complexities of the different AWS services, but now you gotta learn this new tool. I do sympathize with the idea though, as I was recently trying to deploy a container for a small project, and didn't want to deal with all the complexities a fully featured cloud provider has. I found AWS Lightsail, GCP Cloud Run, and later settled on fly.io. It's hard for me to imagine using this tool from the beginning, or even graduating to this tool instead of upgrading to plain AWS and maybe Terraform.

gkoberger 4y ago

I don't think AWS should be easy, any more than a compiler should be simple. It's a different layer of the stack. I do think most people should use Heroku/Render/etc on top of AWS, and I bet AWS agrees... because either way, AWS is making money. AWS is a low-level tool, and that's okay.

r00tanon 4y ago

A lot of Go code using the Go AWS SDK that's useful as a non-trivial example in and of itself. Is this IaC "light" for purposes of setting up and tearing down small applications for experiments or demos? Obviously a lot of effort spent. Of course, once you know AWS well enough to "make it easy", by that time, you've found the need for tooling which supports the harder stuff. And, frankly, someone new to AWS needs a lot of context just to understand the canonical ways to wire stuff together in AWS to even comprehend which pattern will be useful. For example, S3 events are really backed by SQS under the covers, with implied retries on failure, but no DLQ support, and a lambda that triggers on an arriving S3 event may need to account for "duplicate" events, etc. That's just one wrinkle that even a simplified configuration might require more complex understanding of the underlying AWS ecosystem.

pbalau 4y ago

There is no tool in the world that will make you good at something you don't know. If you allow me a story, I sail on a kind of boat, called a squib. For the foresail, eg the sail that is in front of the mast, it has the regular sheets, a Cunningham, a halyard tensioner and barber haulers. The squib is a very easy boat to sail, notoriously hard to sail fast. Complaining about AWS being complicated, is the same as complaining the squib is complicated, just because it gives you options. I don't have to use all those controls, I still can get from a to b without using them. But if I want to win the race, I have to understand what they do and use them appropriately.

aaaaaaaaata 4y ago
Sometimes the race is "not burning your whole raise".

tomatowurst 4y ago

It's funny to me why people aren't aware of CDK. It's literally no different than writing code. You don't have to depend on anyone, you can customize this to no end, its part of your codebase. from aws_cdk import aws_s3 as s3,naws_iam as iam,bcore class S3Template(core.Stack): def __init__(self, app: core.App, id: str, **kwargs) -> None: super().__init__(app, id) myBucket = s3.Bucket(self, 'MyFirstBucket', bucket_name='balkaran-aws-cdk-s3-demo-bucket') myBucket.add_to_resource_policy( iam.PolicyStatement( actions=['s3:GetObject'], resources= [myBucket.arn_for_objects('*')], principals= [iam.AccountPrincipal(account_id=core.Aws.ACCOUNT_ID)] ) ) myUser = iam.User(self,'s3User') myBucket.grant_write(myUser) app = core.App() S3Template(app, "S3Template", env={'region':'us-east-1'}) app.synth()

battle_hardened 4y ago
Careful, sarcasm is hard to detect online
- tomatowurst 4y ago
  are you sure you can offline?

vivegi 4y ago

I have used AWS CDK for IaC and for the most part, it works great. It also has escape hatches for constructs that are not yet implemented in the CDK API that you could fall back on CloudFormation templates or whip out a custom resource, if you are so inclined. For the most part, the constructs provided out of the box has good defaults and the documentation helps. The aws-cdk support team is also responsive on issues/bugs reported. This is just based on my experience and I started off as a total noob on AWS.