10 ms·

How I use the good parts of AWS

357 points by DVassallo 7y ago

deleted 7y ago
[deleted]
fovc 7y ago
Here's the thread link more readably: https://tttthreads.com/thread/1154516910265884672.html https://tttthreads.com/thread/1154516910265884672.html
rjurney 7y ago
I can’t really imagine going back to scripting servers remotely to boot up without docker. It sucked. And if you’ve got docker you want something like ECS, cause why manage your own servers? It’s a bit rough to learn but man life sucked before PaaS. You had to build your own. Now I write a simple install script for docker on my machine, edit a menu and as many servers as I want run that app. Otherwise I dig.
- DVassallo 7y ago
  Author here. Docker is the only one in that list I’m on the fence on. However, I feel that the EC2 AMI can be the equivalent of the container image, and Docker would only be adding another layer and another OS to deal with. Sure, the AMI is not as portable as a docker image, but all in all I prefer working directly on the EC2 VM just for the sake of reducing layers.
  - sien 7y ago
    Very interesting point. But what about maintenance? Working in a place where we have hundreds of small apps, the issue of maintaining umpteen servers is a great pain. Docker reduces the footprint of what has to be maintained substantially.
    - DVassallo 7y ago
      Yes I can see that. I was never in a situation of maintaining more than a couple of apps, so my perspective might be shaped through that.
    - greyskull 7y ago
      Curious, what does "maintenance" mean for you?
      - sien 7y ago
        Patching servers can be part of it. Also fixing minor issues when no major development is being paid for.
        greyskull 7y ago
        FWIW, there is tooling in place such that you can just recycle the hosts in your fleet and they'll be brought up with the latest image. so you can become just another scheduled and automated event. Don't quite follow the latter.
    - antpls 7y ago
      Stop me if I'm wrong, but there is technically nothing blocking AWS to have the same management in EC2. Scaling docker containers is the same allocation problem than scaling VMs, so it could be done with EC2 too. Maybe AWS spins up an actual EC2 instance for each container, it's an implementation detail. EC2 is a legacy tool with different abstractions, but if you replace "VM" by "hypervisor" and "container" by "microkernel-app", you could do the same thing with one less layer and the same tech we always used
  - finaliteration 7y ago
    What about development-wise? One of the big advantages I’ve found with Docker (at least on my teams) is that we’ve mostly eliminated discussions about things breaking on local dev machines due to environment differences. When things break it’s almost always due to code and configuration within our application itself rather than some OS package being a wrong version or something similar. How do you achieve that just using EC2 AMIs without maintaining remote EC2 instances for each developer and keeping them up to date?
    - DVassallo 7y ago
      That was exactly the reason I was very close to adopting Docker recently. But then I realized it was a very rare problem, in my experience. Maybe it’s because I always worked in small teams of <= 10 people, or maybe the stuff I work on tends to not cause local dev problems frequently. But I definitely see how Docker can help a lot there.
      - eatonphil 7y ago
        Even on teams of small people (< 10 people) you can quickly run into issues if you don't develop in the same environment you deploy. For example at one company I worked for, developers would use pypi packages but the operations team required us to deploy using the version of the python package available on the Debian our hosts ran. This would create not infrequent discrepancies that certainly could have been dealt with a number of ways, but would have been pretty simple if we could just bundle a single environment and use it locally and in production. Vagrant and VirtualBox VMs are another alternative (that we did use too!) but Docker allows you to get even closer more easily if teams can agree on it.
        takeda 7y ago
        Being in operations and as a dev, I can tell you that is something operations people are doing wrong. They make their own life harder. What about dependencies that are not in the repo? Will they be building their own packages? What if one of packages has a bug discovered in production, will they create their own package with a bugfix risking breaking system components that might depend on the old version. What when the OS version is EOL to migrate now your app needs to be tested with the new OS, because it is tied to it. I have seen an upgrade from Python 2.6 to 2.7 being unnecessarily painful because of this, even though Python 2.6 code can run on Python 2.7 without changes. You absolutely want your application to be disconnected from the system you running it on. Languages like Python or Java were designed to run the same not only on different OS but also on different architecture, why would you be ruining that? With Python I highly recommend if you use redhat or CentOS to use IUS repo which let's you choose exact Python version you want to use. Then create a virtualenv with the exact dependencies you need. If your ops are standing on the way of this they are misinformed, because what they are suggesting ads a lot of unnecessary work for themselves as well with no tangible gain.
  - evilmushroom 7y ago
    Not even in the same universe. Containers using some managed k8s (EKS, or GKE) cluster with helm is ridiculously easier to manage a dynamic sets of applications running than bare amis. I have a k8s cluster that runs across >1k ec2 instances where I have thousand of containers running various jobs. I can apply/delete any massive deployment across that literally in seconds. with a single command. The layers make it a lot lot easier here.
    - DVassallo 7y ago
      My perspective is shaped by my experience developing and maintaining mostly monolithic apps on fleets of servers. I never had the problem of having to manage dozens/hundreds of independently deployable jobs/services. And I hope I never will :) I can see how K8 can be super useful there, but it’s a situation I try to avoid :)
      - evilmushroom 7y ago
        You know, that makes sense. I was missing that context. If you ever do, please give container orchestration a look. It makes it honestly easy.
      - nostrebored 7y ago
        Weird -- the services you're utilizing are developed exactly the way you're going to avoid! When developing a monolithic application your choices make sense. When you're taking an approach designed to allow for feature development by multiple teams in an Enterprise setting they start to be a bottleneck.
        DVassallo 7y ago
        I’ve seen AWS from inside and I helped build a small part of it. Each individual product is closer to a monolith than a constellation of indispensably deployable jobs/services. The 3 products I worked on were definitely monoliths because we intentionally wanted them to be so. Obviously I don’t mean that they didn’t horizontally scale. Only that the entire app could run from 1 process, the build system produced 1 build artifact, every deployment deploys to all server, and there’s only 1 version of the app running in prod.
  - squirrelicus 7y ago
    Don't be discouraged by the docker love in tech right now. I'm not saying it's a fad (it's basically critical for local dev these days), but nothing will replace the joy of simplicity and empowerment of working with the Linux virtual machine directly instead of through endless PaaS or compose or k8s abstractions.
    - greyskull 7y ago
      If we assume there's a way to quickly build an image for a vm (like docker build), are there workflows to quickly spin up and throw away those images?
      - scarface74 7y ago
        Fargate.
        greyskull 7y ago
        1. That's docker. 2. I should have specified "in local development".
      - scaryclam 7y ago
        I think Packer (https://www.packer.io/ https://www.packer.io/) might what you want there.
      - eatonphil 7y ago
        From a thread a while ago I know of these that work on Dockerfiles but build fully bootable disk images: https://godarch.com/ https://godarch.com/ https://github.com/imc-trading/dock2box https://github.com/imc-trading/dock2box https://buildroot.org/ https://buildroot.org/ https://github.com/ottomatica/slim https://github.com/ottomatica/slim
      - takeda 7y ago
        There is packer I highly recommend to use it together with salt module to do all the configuration. There is also NixOS together with NixOps it has some steep learning curve, but I think it is worth it. You edit configuration.nix which is like Dockerfile, and then you can either use standard AMI and provide the configuration, that will result in setup similar to salt, ansible, chef etc. You can also use NixOps to provision such machine, you can then place /etc/NIXOS_LUSTRATE file which removes state after next reboot, stop it and generate AMI that essentially emulates what packer is doing. You can also use nix build to generate an image file and upload that to AWS. NixOps is also a great tool to test, you can deploy your configuration to local virtual box, or public cloud provider.
  - mnutt 7y ago
    The AMI build process is also incredibly slow compared to docker. When worked into the build process, creating AMIs probably accounts for an extra ~30 minutes for every build, as you have to spin up a separate ec2 instance to create the AMI, snapshot it, copy full AMIs around between regions...
    - DVassallo 7y ago
      I don’t build AMIs on every build. In fact, nowadays, I just use the latest Amazon Linux AMI and set up the necessary stuff on instance boot using a UserData script. Example: https://github.com/encrypted-dev/proof-of-concept/blob/af60b014ef38a2b4f2ca54b80ef114bbb29bdd92/deploy/cfn/stage.yml#L29 https://github.com/encrypted-dev/proof-of-concept/blob/af60b...
      - mnutt 7y ago
        That certainly works and is faster. But doesn't it turn UserData scripts into (part of) a provisioning management system? For instance in that example, you could have a machine die in the middle of the night, the autoscaling group replace it, and end up with a different version of node.js than the one you had before. Or the package server goes down and you can't bring up new instances at all. I suppose you could bake everything into the AMI except your code, and hardcode the git sha into UserData to ensure your builds are reproducible. It just seems like it might get complex when coordinating lock-step code/dependency changes.
  - takeda 7y ago
    I started playing with Nix and recently also tried NixOS and NixOps and I must say that it actually did correctly what other tools (salt, chef, ansible, docker, vagrant, packer and others) failed to do. Nix approach is to build everything from ground up without depending on anything outside of it. It caches results so you don't have to rebuild the whole system when building your application. This approach makes fully reproducible builds[1], because the entire environment is known. Nix by itself can be used for deployments, you can install it on any machine that has nix installed and don't need to worry about any other dependencies, since nix will take care of all of them. It can generate a docker image of you need it, and it will only contain your application with is dependencies. You can use nix-shell to define CDE with all developer tools installed with exact same versions that way developers only need to have nix installed and nix will take care of all other dependencies you need. NixOS takes what Nix does and takes it one step further and uses a configuration that similarly describes an entire operating system. The single configuration.nix describes what your system supposed to have installed, and configured. You can either deploy that and have nix configure machine on boot, configure machine create /etc/NIXOS_LUSTRATE file which removes all state on next reboot and create AMI out of it (equivalent to what packer does). Or have nix generate an image file and upload that to AWS. NixOps supposed to be for deployments, but to me it replaces vagrant and docker you can create configuration.nix and deploy it with local vbox, ec2 and other cloud providers. The great thing is that your configuration file will just work fine no matter which provider you use. There are some rough edges though, for example I needed to update NixOps to use boto3 so it works with assume role and MFA, I hope it will be merged soon. I believe the issue is that what they are doing is very ambitious and they have limited number of developers to handle all of the work, but from all devops tooling I used they seem to have the right approach. They provide the immutability reproducibility at the right level and doing it the right way (declarative (through use of pure, functional, lazily evaluated language starting with a fully known state) vs an iterative language with a partially known state) [1] your need to pin your build down to specific version of nixpkgs
- squirrelicus 7y ago
  I don't agree life sucked before PaaS. I've been responsible for responding to far too many production issues where the PaaS tooling got in the way of rapid response. PaaS is a problem for me because so many developers don't understand (or own) the tradeoffs of choosing PaaS, so i have to constantly have the "no-PaaS" argument to people who have never dealt with the responsibility of uptime response and can't fathom why I'd want to "do more work when the PaaS does it for you"
  - viraptor 7y ago
    I think if you start a new project, PaaS is definitely the way to go. You don't have that many infra people available, you don't have time to deal with building your own deployment details, etc. Once you have time, staff, and real users - move from PaaS to something you can prove is both a better fit and more reliable than (for example) Heroku. But in the meantime? PaaS services solved a lot of issues for you for less money than it would take you to interview a sysops person. Use them as much as you can, whether it's complete app hosting, abstracted container deployment, or something else.
    - squirrelicus 7y ago
      There's a third option: as you grow in your career as a developer, learn the ops stuff so you can make good choices from square one.
- takeda 7y ago
  If your site is simple sure PaaS will work, but anything advanced and even simple things are becoming complicated.
benjaminwootton 7y ago
“Step 1: Forget that all these things exist: Microservices, Lambda, API Gateway, Containers, Kubernetes, Docker. Anything whose main value proposition is about “ability to scale” will likely trade off your “ability to be agile & survive”. That’s rarely a good trade off.” I don’t see these as being about scalability. Rather, they’re about fast time to market and ability to change. Moving up the stack and adding managed services such as API Gateway will definetly give your product a better chance of survival.
- DVassallo 7y ago
  The disadvantages I’m highlighting are about the restrictions of the Lambda abstraction. What if you want to send telemetry to a third party? Or use a cache? Or deploy something bigger than 250MB? Or handle WebSockets (without having to read/write state in DDB on every message)? Or buffer something on the filesystem? Or run something for more than 15 mins? etc etc. How do all those things not impact agility? (In the web app/service space at least.)
  - nexuist 7y ago
    I've been working with Lambda for about two years now, I'd like to answer all of your concerns. I don't work for AWS but I do love Lambda and I think it has its place amongst everything else. Previously I built node.js+Postgres apps hosted on DigitalOcean for ~4 years. >What if you want to send telemetry to a third party? Can't you do this from your own Lambda code? Sure, your code could crash before it can reach your telemetry service - but isn't this a concern on a server based app as well? >Or use a cache? Elasticache or Mongo or whatever NoSQL 3rd party service you want to use works straight from Lambda. If you're talking about caching Lambda responses, you can again add custom code, which you would also have to do in a server environment. >Or deploy something bigger than 250MB? Yeah, you're SOL here. 250MB is huge for any non-GUI software and it would take a long time to get set up on Lambda's containers. If you're in this spot, I wholeheartedly recommend ditching Lambda for EC2. However, don't count Lambda out entirely - you can still have it take over repetitive, simple tasks so the server hosting your monster 250MB backend doesn't get overwhelmed! >Or handle WebSockets (without having to read/write state in DDB on every message)? I'm sure when PHP introduced sessions there were a bunch of devs complaining about having to maintain a MySQL table for session keys. Ultimately, if you're making a web app, 95% of its routes are probably glorified Excel formulas, so you'd need to pull and push state through a database anyways. Where else do you store WebSocket state? In memory? What happens when you got millions of connections at once (think slither.io scale)? At the end of the day you have to put it into something that can scale. I'm assuming if you're using Lambda you care about scaling up - otherwise you could literally run your backend on an IoT toaster with a MySQL database hosted on some IoT coffee maker that had "admin" as its root password and nobody would tell the difference. Again, if this is you - Lambda wasn't meant for your use case. Go buy a coffee maker. >Or buffer something on the filesystem? I'm not really sure how to answer this one. There is obviously no permanent file system on Lambda unless you count S3 (although I'm sure you know that). I did some searching and found this: https://stackoverflow.com/a/31660175 https://stackoverflow.com/a/31660175 If you're talking about uploading huge files, direct upload to S3 seems like your best bet. >Or run something for more than 15 mins? Lambda wasn't meant for this. Set up a server and schedule a cron job. I'm guessing if it takes >15min it's probably some kind of backup, statistical analysis, ML model training, database dump parsing, yadda yadda yadda...Lambda is for handling events. Everything I mentioned seems like an internal business operation the users have no part in, so that also seems like a good candidate for just having one server instance floating around and throwing all your odd long jobs onto it. I haven't used Lambda Step Functions, but I vaguely recall hearing something about being able to run long tasks with those? Not sure. I wouldn't bother, though, I'd just head straight for a server (the example Amazon gives is starting a job to retrieve a specific item in a warehouse, sending an order to an inbox, and waiting for a warehouse worker to mark the item as retrieved...I don't know why Amazon chose that specific example, but it sounds like they have a very specific target audience!) >How do all those things not impact agility? I'm of the opinion that tools do not impact agility, decisions do. If you decide to use Lambda in a situation where a server would prevail, you're wasting time on the wrong thing. If you decide to a use a server in a situation where Lambda would prevail, you're...not really doing anything wrong, I think. It'll likely cost you more than a Lambda function but those numbers only start to matter once you actually have to care about scale. Like I said, it's ultimately about what you're trying to do. Don't put a square hole through a rectangular screw, or something like that.
obulpathi 7y ago
Or use Google Cloud, which has 90% good parts. Documentation can be a bit pain, but the services themselves are rocksolid. There are no 3/4 queuing services, just one. GKE rocks! Cloud Console is a breath of fresher, compared to AWS. Cloud Shell makes it easy to bypass firewalls for logging into instances and no messing with public keys. It's all managed for you. Use firebase if you are looking specifically for Web and Mobile Apps. Scaling to millions of users or Petabytes of data is no big deal and you don't have to rearchitect everytime your customer base grows by 10x.
- rjurney 7y ago
  The google services are just way easier to use but once you’ve got expert level proficiency at AWS it’s tough to let go.
  - evilmushroom 7y ago
    I have to use both. I find google easier to do simple things.. but I find it lacks some of the flexibility AWS has for less simple things. Well, that and Google's managed k8s solution was down for multiple days awhile back when I was doing a comparison. Another reason I use EKS atm.... despite I think GKE is a bit better.
    - obulpathi 7y ago
      I don't know when this happened. Give GKE a try, it's really amazing. Blows EKS out of water. As per the flexibility is concerned, once you learn how to use Google Services, you can get the flexibility with simplicity. AWS services are too complex and even things like billing require a PhD degree in finance to optimize for anything non-trivial.
  - zaphirplane 7y ago
    Can you provide examples, in what way they are easier
  - flurdy 7y ago
    The lock-in with not just all your infrastructure with one provider but also all the skills and proficiencies with one provider's stack is a high risk that I ranted about recently: https://blog.flurdy.com/2019/06/all-eggs-in-one-cloud-basket.html https://blog.flurdy.com/2019/06/all-eggs-in-one-cloud-basket... Having your teams doing little POCs with other providers will ease your project/company cloud bus-factor significantly.
jamestimmins 7y ago
Perhaps I'm alone here, but I'd say 50% of my time with AWS goes to (attempting to) properly configure the VPC as well as the IAM roles. For the majority of small-ish projects, the hardware isn't nearly as important or difficult as properly configuring access rules between services and outside parties.
- rjurney 7y ago
  IAM is a killer. You’re absolutely right. Are GCP permissions easier? I haven’t used them as much.
  - chishaku 7y ago
    night and day imho
  - mey 7y ago
    https://github.com/tilfin/aws-extend-switch-roles/blob/master/README.md https://github.com/tilfin/aws-extend-switch-roles/blob/maste... may be interesting to you. The fact that this has to exist is also interesting...
  - usmannk 7y ago
    I'll hate on GCP all day long but the one thing it has going for it is the permissions.
    - wodenokoto 7y ago
      Wow, I was gonna say “haven’t tried IAM on other platforms than GCP but I can’t imagine it being more difficult than this”, but damn ...
  - jsmeaton 7y ago
    I’ve stopped worrying about minimising IAM permissions and tend to just use the built in AWS roles for most things now.
    - 7y ago
whycombagator 7y ago
And if you want something even easier (albeit with more fixed/upfront costs) just use Heroku.
- DVassallo 7y ago
  I agree. I have no direct experience with Heroku so I can’t comment on the experience/restrictions, but if a PaaS works well for what you’re doing, I’d go for it.
  - bubble_talk 7y ago
    I was also thinking the same, especially considering that you start with this as your main reason: "Anything whose main value proposition is about “ability to scale” will likely trade off your “ability to be agile & survive”."
cosmotic 7y ago
Step one: don't use Twitter for this sort of thing
- xtracto 7y ago
  Thank You! I thought I was the only one that hated the format of these things. Why not writing a proper article and linking it on Twitter? It would be way better
mgamache 7y ago
All depends on what your goal is. Are you looking for large scale out? Or are you iterating fast and don't want to mess with managing databases, CDN's, KV stores etc...? Note: A version of premature optimization is premature scaling. So beware of those offerings that require you to use a certain technology to enable AWS scale-out it can slow you down.
merlincorey 7y ago
I am in agreement with everything generally except for CloudFormation and setting up Route53 zones manually. I use Terraform to setup my infrastructure as service on AWS, including Route53 zones. After the latest 0.12 upgrade to the language, Terraform is quite a bit more user friendly than CloudFormation, and importantly, not locked down to just Amazon -- it supports multiple clouds and on-premises solutions for declarative orchestration of resources.
- DVassallo 7y ago
  Author here. Fair. I never used Terraform beyond playing with it briefly, so I can’t really comment on the differences. I tend to choose CFN mostly because I’m familiar with it.
  - oppositelock 7y ago
    I'm in charge of cloud infrastructure at my company, and I've found that sticking with CloudFormation on AWS is more powerful and versatile than TerraForm, since TF isn't always a perfect mapping, where CFN is, with the caveat that it's sometimes months behind what their API can do. To that end, over the years, I've come across a couple of tools which help write CFN templates and manage rollout keeping cross-stack dependencies in mind. If anyone does a lot of CFN, I highly recommend Troposphere (python abstraction for cloudformation) [1] and Stacker (manager stack rollout via dependency DAG) [2] 1: https://github.com/cloudtools/troposphere https://github.com/cloudtools/troposphere 2: https://github.com/cloudtools/stacker https://github.com/cloudtools/stacker
    - philsnow 7y ago
      Fwiw terraform also lags behind what the underlying apis can do -- but it's a community effort and you can contribute to add the functionality you need, whereas with CFN you can only ask your TAM (iff you have one!) to ask the internal team(s) (not sure if only CFN team or only the service team or both have to implement something) to prioritize your feature request.
- evilmushroom 7y ago
diminoten 7y ago
Sort of related to this, but can someone explain to me why Zappa[1] isn't a bigger deal? It completely delivers, as far as I can tell, on the promise of deploying a WSGI app into a Lambda in a way that requires minimal configuration. And yet, the project seems to have stagnated somewhat. Maybe it just did everything it set out to do? I'm not sure, but when people say "forget about Lambda because you'll have to roll your own shit" I wonder if they've heard of Zappa or not. [1] https://github.com/Miserlou/Zappa https://github.com/Miserlou/Zappa
- KaiserPro 7y ago
  YES. Zappa got me out of a massive hole several times. I'm not a webdev, and we arn't a web company, so we don't have a web deployment pipeline. Having Zappa makes our life very simple and cheap. The best part is that local and "prod" are equally easy to spin up. The simple integration to DNS, SSL, api gateway, is a wonder. The only downside is the permissions management is a massive pain, and assumes that you have admin permissions.
- takeda 7y ago
  I tried to use Zappa and my experience was that it didn't support tires of callbacks I needed it to. After I patched it up to do what I want I realized that handling calls directly was actually simpler than using Zappa and Zappa made my application huge. I understand that primary use of Zappa is a web application, but I have reservation about that as well. The "serverless" applications aren't really server less, they just have VMs spun on demand and it's only good idea when you have a service that most of the time is idle.
- The_Amp_Walrus 7y ago
  I've tried to use Zappa for a non-trivial personal project to run a Django app and I found: - Super nice, low config API - easy to deploy - great out-of-the-box support for async tasks (no Celery! woo!) but: - It was a nightmare getting certain Python libraries (eg. Pillow, psycopg) to work in the AWS Lambda environment - It really sucked having to deploy to AWS in order to debug issues which cannot be reproduced locally (eg. library) - It seems hard to get away from using AWS tools to observe your code in prod (eg. CloudWatch for logs) - I still needed a database to maintain state, DynamoDB didn't work with Django's ORM and was surprisingly expensive, and if I'm going to shell out $10/mo for a Postgres RDS instance then I may as well run the whole thing on EC2 anyway I think Zappa is a really nice tool for some niche use cases, and I'd definitely turn to it if I needed to stand up some small, stateless serverless service, but I would hate to support and debug it as a web app in prod. project: https://memories.ninja https://memories.ninja zappa version: https://github.com/MattSegal/family-photos/tree/lambda-hosting https://github.com/MattSegal/family-photos/tree/lambda-hosti... ec2 version: https://github.com/MattSegal/family-photos https://github.com/MattSegal/family-photos
philwelch 7y ago
I wouldn’t recommend using single EC2 instances in production ever, but the rest seems solid.
greyskull 7y ago
The CDK[0] will hopefully help with the cloudformation story. [0] https://github.com/aws/aws-cdk https://github.com/aws/aws-cdk
manigandham 7y ago
Completely agree on just using a server instead of the various lambda-style systems. All modern languages have great web/app frameworks that make it incredibly easy to build, whether it's a single endpoint or a giant app. The ability to just include whatever code you need and deploy it atomically is massively underrated. Also agreed on scale, servers are fast and cheap and the savings from Lambda rarely pays off in the extended effort. When you do scale, Lambda becomes more expensive anyway. I do recommend using Docker though. Containers are more portable and easier to deploy and replace on a running server, along with the ability to run multiple instances, mount volumes, setup ports and local networks, and eventually migrate to something like ECS/K8S if you really need it.
- tus88 7y ago
  Extended effort to push up a lambda function and not have to worry about automating deployment and configuration and patching and monitoring and upgrading and fail over-ring and, yes, scaling? Maybe its just me but I'd rather not see the backend of a server ever again for anything other than development.
  - bharam 7y ago
    > not have to worry about automating deployment How do you validate your code before deploying to production? If you test in environments besides production how do you manage configuration settings for the different environments (i.e. a db connection string)? How do you avoid patching? Almost any code I've written takes dependencies on 3rd party libraries and those will often have security vulnerabilities (usually some time after I wrote the code).
    - tus88 7y ago
      I mean of infrastructure, servers etc. I.e. the days of Puppet and Ansible are behind me. And patching a Lambda is as simple as pushing up a new version. No downtime, restarting services etc. And no patching of OS or Docker containers. Or building them. Configuration is as simple as environment vars or SSM for secrets.
  - 7y ago
etaioinshrdlu 7y ago
I find the characterization of RDS Aurora as not proven, or unstable, as odd. I thought it looked like one of the more mature-seeming choices on the market. Any thoughts?
- vasco 7y ago
  From the point of view of someone who's used Aurora Postgres since it came out, though you sometimes are exposed to bugs, AWS support has always been great and we never faced anything super serious. It's been almost a year since we had any problems though. This on a somewhat large aurora instance at around 5TB, so it's already representative for a lot of people.
  - takeda 7y ago
    Right now if you use Aurora Postgresql 9.6 you don't have an easy way to migrate to 10, and 11 is not even available. They supposedly are working on a solution, but won't disclose when it will be available.
- sciurus 7y ago
  It's way less mature than vanilla MySQL or postgresql.
foobar_ 7y ago
Docker is bullshit. AWS images are alright IME. Spinning multiple large instances will give you more bang for the buck than dealing with tiny docker containers.
abiro 7y ago
I don't think is good advice in general. OP is obviously used to doing things in a certain way and good for him for working the way he is most productive. But looking at things objectively, everything has a learning a curve and once you get past that, Lambda is way more efficient even when working alone due to the agility it allows and time saved on maintenance.
k__ 7y ago
Anything whose main value proposition is about “ability to scale” will likely trade off your “ability to be agile & survive”. That’s rarely a good trade off. Interesting that he sees micromanaging infra on AWS as more agile than using managed services. Also, serverless systems don't cost you money when you(r customers) don't use them, so that's a huge plus on survival. Finally, code you write is always a liability, if you got a good architecture (hexagonal, etc.) you can just swap out one service that saved you time in the past with another service that will save you time in the future.
- awinder 7y ago
  Yeah for launching startup level traffic I totally agree, lambda feels like the right arch at the right cost. Also it will engrain setups that, like you said, you can swap out. You’ll need an alb / api gateway, you’ll need to get logs flowing through cloudwatch or some other non-host logging mechanisms. Transferring onto these kinds of architectures from not-these-architectures has been a royal pain and a time drain in my experience.
- gtirloni 7y ago
  Yeah, kind of made me cringe a bit (considering his experience).
- freehunter 7y ago
  >serverless systems don't cost you money when you(r customers) don't use them All the horror stories I see surrounding huge surprise Lambda bills always bring me back to this point. If I have to pay $5/mo for a server I only use for two hours once every few months, that's something that should go on Lambda. If it's something that I'm using constantly, all day every day, a server will be cheaper. If I only use my car once every few weeks, Uber makes a lot of sense. If I use it every day back and forth to work and the grocery store, Uber's gonna be a lot more expensive. Lambda is for small tasks that don't execute often. And using it right can save startups gobsmackingly large amounts of money.
  - k__ 7y ago
    Yes, it comes all down to risk assessment. Maybe using serverless technology is too hard for your corp, because you don't have the skills, so it could lead to problems in the future (surprise Lambda bills). But it could also be that your competitors get a huge advantage by investing in the serverless paradigm and run you away in the future. To me that Twitter thread sounded too much like a guy who invested in some tech over the last 11 years and now tries to convince potential customers of him to use the tech he knows about. He could be right, he could be wrong. I don't know. I started back-end development with serverless, so I'm biased in the other direction, haha.
ookblah 7y ago
I think this is really great for getting stuff up and running as quick as possible, maybe if you're just starting out, but I'm really surprised at some of the things said. Maybe if this is a site that you run on the side. Doing test/staging on a nano and then pushing to production on an m5? What? Like get ready to troubleshoot random issues completely unrelated to your code. And then equating different AMI's to containers is vastly oversimplifying things. With docker I can just get up and running on my local box and every dev I share this with has the exact same environment and config. I can package up those images and send them up to ECS/Kubernetes and not deal with the same headaches. The learning curve is a bit steeper, but absolutely worth it. I walked the same path here (starting out with single VMs for environments, installing services locally on my laptop) and the headaches down the line are not worth it and it isn't even a case of over-thinking the solution. You don't need to ride the cutting edge but modern tooling saves a ton of time. It was a PITA migrating our old VM stuff over, but absolutely worth it. If you don't want to deal with maintaining systems there are other solutions mentioned ranging from full PaaS to something like GKE.
- mpmpmpmp 7y ago
  My favorite part about using ECS was never being able to deploy because our nodes had jacked up ecs agents on them. So the application was still running but you couldn’t touch it. We switched to GKE last year and haven’t had a single problem with infra since. YMMV but for simpler docker/k8s apps GKE has been so easy in my experience.
  - eatonphil 7y ago
    ECS Fargate is an option for fully managed container runtime/scheduling on AWS today. We're looking into using this for deploying applications. Anyone here had experience with it?
    - cle 7y ago
      I’ve used it for both running a website and backend batch processing, and it has worked great so far.
foxhop 7y ago
I'm a core dev of stacker, it's worth taking a look at if you maintain Cloudformation templates. Makes life a lot better. At Remind (remind.com) we have nearly 600 separate Cloudformation stacks which build and maintain our stage and prod environments. This would be insane without stacker. As for this tweet, I'm fairly confident he is talking about a side projects or early startups and I agree with most of what he said. Personal, I don't use AWS for side projects, I use offerings from Linode, Digital Ocean, and vultr. They are cheaper and scale up vertically with a click of a button, which is really what you need for the first couple of years, unless you hit the growth/scaling lottery, which isn't typically the case. For example, over the last two weekends I was able to use Digital Ocean Spaces (alternative to AWS S3) to build my wife a secure digital downloads store. All uploads and downloads use presigned POST and GET urls created via Boto3! I was suprised how perfect Digital Ocean implemented S3's API in their Spaces offering. https://russell.ballestrini.net/pre-signed-get-and-post-for-digital-ocean-spaces/ https://russell.ballestrini.net/pre-signed-get-and-post-for-...
nijave 7y ago
Lambda is absolutely essential for glueing together Amazon services and many Amazon articles recommend it (for instance, getting your ASG to drain ECS instances before scaling down). It's also helpful for building alerting/monitoring workflows since Cloudwatch/SNS is pretty simplistic on its own I think the sentiment is more "don't build your entire app on lambda"
- wongarsu 7y ago
  I think "don't have any customer facing Lamdas" is a good rule of thumb. It's great for glue code, if-this-then-that code and cron jobs.
cwyers 7y ago
Why not just use Linode or Digital Ocean at this point?
nijave 7y ago
I tend to disagree with the EC2 sentiment. Any cost savings using VMs over a higher abstraction will likely be wiped out by the time wasted becoming a sysadmin unless you just like doing sysadmin stuff (patching, access management, config)
- ericcholis 7y ago
  This is a very solid point and something to consider inside of the entire infrastructure planning stage. Opportunity cost could be huge, or net zero depending on your needs.
otabdeveloper2 7y ago
> 1/25 Is there a more user-hostile webapp than Twitter? I think not.
fnord77 7y ago
> Forget that all these things exist: ... Containers, Kubernetes, Docker. Not sure how this guy can give this advice. containers make rapid development so much easier. There's no distinction between your dev environment and your prod environment except for a couple properties/env variables and some scaling factors. Also "no" to cloudformation and "yes" to terraform. And if I were to write a simple application, I'd probably try with lambda next time around.
aequitas 7y ago
> But Autoscaling is still useful. Think of it as a tool to help you spin up or replace instances according to a template. If you have a bad host, you can just terminate it and AS will replace it with an identical one (hopefully healthy) in a couple of minutes. I really started to enjoy AWS when I got into the habit of deploying all EC2 instances using Autoscaling groups. Even single node instances and configurations that don't need any scaling, just everything. Autoscaling is free and it forces you into the immutable/disposable infrastructure paradigm, which just makes administration of the nodes just so much easier. And for all stateful stuff there is RDS, S3, dynamo, SQS, etc.
deleted 7y ago
[deleted]

357 points by DVassallo 7y ago

deleted 7y ago

[deleted]

fovc 7y ago

Here's the thread link more readably: https://tttthreads.com/thread/1154516910265884672.html https://tttthreads.com/thread/1154516910265884672.html

rjurney 7y ago

I can’t really imagine going back to scripting servers remotely to boot up without docker. It sucked. And if you’ve got docker you want something like ECS, cause why manage your own servers? It’s a bit rough to learn but man life sucked before PaaS. You had to build your own. Now I write a simple install script for docker on my machine, edit a menu and as many servers as I want run that app. Otherwise I dig.

DVassallo 7y ago
Author here. Docker is the only one in that list I’m on the fence on. However, I feel that the EC2 AMI can be the equivalent of the container image, and Docker would only be adding another layer and another OS to deal with. Sure, the AMI is not as portable as a docker image, but all in all I prefer working directly on the EC2 VM just for the sake of reducing layers.
- sien 7y ago
  Very interesting point. But what about maintenance? Working in a place where we have hundreds of small apps, the issue of maintaining umpteen servers is a great pain. Docker reduces the footprint of what has to be maintained substantially.
  - DVassallo 7y ago
    Yes I can see that. I was never in a situation of maintaining more than a couple of apps, so my perspective might be shaped through that.
  - greyskull 7y ago
    Curious, what does "maintenance" mean for you?
    - sien 7y ago
      Patching servers can be part of it. Also fixing minor issues when no major development is being paid for.
      - greyskull 7y ago
        FWIW, there is tooling in place such that you can just recycle the hosts in your fleet and they'll be brought up with the latest image. so you can become just another scheduled and automated event. Don't quite follow the latter.
  - antpls 7y ago
    Stop me if I'm wrong, but there is technically nothing blocking AWS to have the same management in EC2. Scaling docker containers is the same allocation problem than scaling VMs, so it could be done with EC2 too. Maybe AWS spins up an actual EC2 instance for each container, it's an implementation detail. EC2 is a legacy tool with different abstractions, but if you replace "VM" by "hypervisor" and "container" by "microkernel-app", you could do the same thing with one less layer and the same tech we always used
- finaliteration 7y ago
  What about development-wise? One of the big advantages I’ve found with Docker (at least on my teams) is that we’ve mostly eliminated discussions about things breaking on local dev machines due to environment differences. When things break it’s almost always due to code and configuration within our application itself rather than some OS package being a wrong version or something similar. How do you achieve that just using EC2 AMIs without maintaining remote EC2 instances for each developer and keeping them up to date?
  - DVassallo 7y ago
    That was exactly the reason I was very close to adopting Docker recently. But then I realized it was a very rare problem, in my experience. Maybe it’s because I always worked in small teams of <= 10 people, or maybe the stuff I work on tends to not cause local dev problems frequently. But I definitely see how Docker can help a lot there.
    - eatonphil 7y ago
      Even on teams of small people (< 10 people) you can quickly run into issues if you don't develop in the same environment you deploy. For example at one company I worked for, developers would use pypi packages but the operations team required us to deploy using the version of the python package available on the Debian our hosts ran. This would create not infrequent discrepancies that certainly could have been dealt with a number of ways, but would have been pretty simple if we could just bundle a single environment and use it locally and in production. Vagrant and VirtualBox VMs are another alternative (that we did use too!) but Docker allows you to get even closer more easily if teams can agree on it.
      - takeda 7y ago
        Being in operations and as a dev, I can tell you that is something operations people are doing wrong. They make their own life harder. What about dependencies that are not in the repo? Will they be building their own packages? What if one of packages has a bug discovered in production, will they create their own package with a bugfix risking breaking system components that might depend on the old version. What when the OS version is EOL to migrate now your app needs to be tested with the new OS, because it is tied to it. I have seen an upgrade from Python 2.6 to 2.7 being unnecessarily painful because of this, even though Python 2.6 code can run on Python 2.7 without changes. You absolutely want your application to be disconnected from the system you running it on. Languages like Python or Java were designed to run the same not only on different OS but also on different architecture, why would you be ruining that? With Python I highly recommend if you use redhat or CentOS to use IUS repo which let's you choose exact Python version you want to use. Then create a virtualenv with the exact dependencies you need. If your ops are standing on the way of this they are misinformed, because what they are suggesting ads a lot of unnecessary work for themselves as well with no tangible gain.
- evilmushroom 7y ago
  Not even in the same universe. Containers using some managed k8s (EKS, or GKE) cluster with helm is ridiculously easier to manage a dynamic sets of applications running than bare amis. I have a k8s cluster that runs across >1k ec2 instances where I have thousand of containers running various jobs. I can apply/delete any massive deployment across that literally in seconds. with a single command. The layers make it a lot lot easier here.
  - DVassallo 7y ago
    My perspective is shaped by my experience developing and maintaining mostly monolithic apps on fleets of servers. I never had the problem of having to manage dozens/hundreds of independently deployable jobs/services. And I hope I never will :) I can see how K8 can be super useful there, but it’s a situation I try to avoid :)
    - evilmushroom 7y ago
      You know, that makes sense. I was missing that context. If you ever do, please give container orchestration a look. It makes it honestly easy.
    - nostrebored 7y ago
      Weird -- the services you're utilizing are developed exactly the way you're going to avoid! When developing a monolithic application your choices make sense. When you're taking an approach designed to allow for feature development by multiple teams in an Enterprise setting they start to be a bottleneck.
      - DVassallo 7y ago
        I’ve seen AWS from inside and I helped build a small part of it. Each individual product is closer to a monolith than a constellation of indispensably deployable jobs/services. The 3 products I worked on were definitely monoliths because we intentionally wanted them to be so. Obviously I don’t mean that they didn’t horizontally scale. Only that the entire app could run from 1 process, the build system produced 1 build artifact, every deployment deploys to all server, and there’s only 1 version of the app running in prod.
- squirrelicus 7y ago
  Don't be discouraged by the docker love in tech right now. I'm not saying it's a fad (it's basically critical for local dev these days), but nothing will replace the joy of simplicity and empowerment of working with the Linux virtual machine directly instead of through endless PaaS or compose or k8s abstractions.
  - greyskull 7y ago
    If we assume there's a way to quickly build an image for a vm (like docker build), are there workflows to quickly spin up and throw away those images?
    - scarface74 7y ago
      Fargate.
      - greyskull 7y ago
        1. That's docker. 2. I should have specified "in local development".
    - scaryclam 7y ago
      I think Packer (https://www.packer.io/ https://www.packer.io/) might what you want there.
    - eatonphil 7y ago
      From a thread a while ago I know of these that work on Dockerfiles but build fully bootable disk images: https://godarch.com/ https://godarch.com/ https://github.com/imc-trading/dock2box https://github.com/imc-trading/dock2box https://buildroot.org/ https://buildroot.org/ https://github.com/ottomatica/slim https://github.com/ottomatica/slim
    - takeda 7y ago
      There is packer I highly recommend to use it together with salt module to do all the configuration. There is also NixOS together with NixOps it has some steep learning curve, but I think it is worth it. You edit configuration.nix which is like Dockerfile, and then you can either use standard AMI and provide the configuration, that will result in setup similar to salt, ansible, chef etc. You can also use NixOps to provision such machine, you can then place /etc/NIXOS_LUSTRATE file which removes state after next reboot, stop it and generate AMI that essentially emulates what packer is doing. You can also use nix build to generate an image file and upload that to AWS. NixOps is also a great tool to test, you can deploy your configuration to local virtual box, or public cloud provider.
- mnutt 7y ago
  The AMI build process is also incredibly slow compared to docker. When worked into the build process, creating AMIs probably accounts for an extra ~30 minutes for every build, as you have to spin up a separate ec2 instance to create the AMI, snapshot it, copy full AMIs around between regions...
  - DVassallo 7y ago
    I don’t build AMIs on every build. In fact, nowadays, I just use the latest Amazon Linux AMI and set up the necessary stuff on instance boot using a UserData script. Example: https://github.com/encrypted-dev/proof-of-concept/blob/af60b014ef38a2b4f2ca54b80ef114bbb29bdd92/deploy/cfn/stage.yml#L29 https://github.com/encrypted-dev/proof-of-concept/blob/af60b...
    - mnutt 7y ago
      That certainly works and is faster. But doesn't it turn UserData scripts into (part of) a provisioning management system? For instance in that example, you could have a machine die in the middle of the night, the autoscaling group replace it, and end up with a different version of node.js than the one you had before. Or the package server goes down and you can't bring up new instances at all. I suppose you could bake everything into the AMI except your code, and hardcode the git sha into UserData to ensure your builds are reproducible. It just seems like it might get complex when coordinating lock-step code/dependency changes.
- takeda 7y ago
  I started playing with Nix and recently also tried NixOS and NixOps and I must say that it actually did correctly what other tools (salt, chef, ansible, docker, vagrant, packer and others) failed to do. Nix approach is to build everything from ground up without depending on anything outside of it. It caches results so you don't have to rebuild the whole system when building your application. This approach makes fully reproducible builds[1], because the entire environment is known. Nix by itself can be used for deployments, you can install it on any machine that has nix installed and don't need to worry about any other dependencies, since nix will take care of all of them. It can generate a docker image of you need it, and it will only contain your application with is dependencies. You can use nix-shell to define CDE with all developer tools installed with exact same versions that way developers only need to have nix installed and nix will take care of all other dependencies you need. NixOS takes what Nix does and takes it one step further and uses a configuration that similarly describes an entire operating system. The single configuration.nix describes what your system supposed to have installed, and configured. You can either deploy that and have nix configure machine on boot, configure machine create /etc/NIXOS_LUSTRATE file which removes all state on next reboot and create AMI out of it (equivalent to what packer does). Or have nix generate an image file and upload that to AWS. NixOps supposed to be for deployments, but to me it replaces vagrant and docker you can create configuration.nix and deploy it with local vbox, ec2 and other cloud providers. The great thing is that your configuration file will just work fine no matter which provider you use. There are some rough edges though, for example I needed to update NixOps to use boto3 so it works with assume role and MFA, I hope it will be merged soon. I believe the issue is that what they are doing is very ambitious and they have limited number of developers to handle all of the work, but from all devops tooling I used they seem to have the right approach. They provide the immutability reproducibility at the right level and doing it the right way (declarative (through use of pure, functional, lazily evaluated language starting with a fully known state) vs an iterative language with a partially known state) [1] your need to pin your build down to specific version of nixpkgs
squirrelicus 7y ago
I don't agree life sucked before PaaS. I've been responsible for responding to far too many production issues where the PaaS tooling got in the way of rapid response. PaaS is a problem for me because so many developers don't understand (or own) the tradeoffs of choosing PaaS, so i have to constantly have the "no-PaaS" argument to people who have never dealt with the responsibility of uptime response and can't fathom why I'd want to "do more work when the PaaS does it for you"
- viraptor 7y ago
  I think if you start a new project, PaaS is definitely the way to go. You don't have that many infra people available, you don't have time to deal with building your own deployment details, etc. Once you have time, staff, and real users - move from PaaS to something you can prove is both a better fit and more reliable than (for example) Heroku. But in the meantime? PaaS services solved a lot of issues for you for less money than it would take you to interview a sysops person. Use them as much as you can, whether it's complete app hosting, abstracted container deployment, or something else.
  - squirrelicus 7y ago
    There's a third option: as you grow in your career as a developer, learn the ops stuff so you can make good choices from square one.
takeda 7y ago
If your site is simple sure PaaS will work, but anything advanced and even simple things are becoming complicated.

benjaminwootton 7y ago

“Step 1: Forget that all these things exist: Microservices, Lambda, API Gateway, Containers, Kubernetes, Docker. Anything whose main value proposition is about “ability to scale” will likely trade off your “ability to be agile & survive”. That’s rarely a good trade off.” I don’t see these as being about scalability. Rather, they’re about fast time to market and ability to change. Moving up the stack and adding managed services such as API Gateway will definetly give your product a better chance of survival.

DVassallo 7y ago
The disadvantages I’m highlighting are about the restrictions of the Lambda abstraction. What if you want to send telemetry to a third party? Or use a cache? Or deploy something bigger than 250MB? Or handle WebSockets (without having to read/write state in DDB on every message)? Or buffer something on the filesystem? Or run something for more than 15 mins? etc etc. How do all those things not impact agility? (In the web app/service space at least.)
- nexuist 7y ago
  I've been working with Lambda for about two years now, I'd like to answer all of your concerns. I don't work for AWS but I do love Lambda and I think it has its place amongst everything else. Previously I built node.js+Postgres apps hosted on DigitalOcean for ~4 years. >What if you want to send telemetry to a third party? Can't you do this from your own Lambda code? Sure, your code could crash before it can reach your telemetry service - but isn't this a concern on a server based app as well? >Or use a cache? Elasticache or Mongo or whatever NoSQL 3rd party service you want to use works straight from Lambda. If you're talking about caching Lambda responses, you can again add custom code, which you would also have to do in a server environment. >Or deploy something bigger than 250MB? Yeah, you're SOL here. 250MB is huge for any non-GUI software and it would take a long time to get set up on Lambda's containers. If you're in this spot, I wholeheartedly recommend ditching Lambda for EC2. However, don't count Lambda out entirely - you can still have it take over repetitive, simple tasks so the server hosting your monster 250MB backend doesn't get overwhelmed! >Or handle WebSockets (without having to read/write state in DDB on every message)? I'm sure when PHP introduced sessions there were a bunch of devs complaining about having to maintain a MySQL table for session keys. Ultimately, if you're making a web app, 95% of its routes are probably glorified Excel formulas, so you'd need to pull and push state through a database anyways. Where else do you store WebSocket state? In memory? What happens when you got millions of connections at once (think slither.io scale)? At the end of the day you have to put it into something that can scale. I'm assuming if you're using Lambda you care about scaling up - otherwise you could literally run your backend on an IoT toaster with a MySQL database hosted on some IoT coffee maker that had "admin" as its root password and nobody would tell the difference. Again, if this is you - Lambda wasn't meant for your use case. Go buy a coffee maker. >Or buffer something on the filesystem? I'm not really sure how to answer this one. There is obviously no permanent file system on Lambda unless you count S3 (although I'm sure you know that). I did some searching and found this: https://stackoverflow.com/a/31660175 https://stackoverflow.com/a/31660175 If you're talking about uploading huge files, direct upload to S3 seems like your best bet. >Or run something for more than 15 mins? Lambda wasn't meant for this. Set up a server and schedule a cron job. I'm guessing if it takes >15min it's probably some kind of backup, statistical analysis, ML model training, database dump parsing, yadda yadda yadda...Lambda is for handling events. Everything I mentioned seems like an internal business operation the users have no part in, so that also seems like a good candidate for just having one server instance floating around and throwing all your odd long jobs onto it. I haven't used Lambda Step Functions, but I vaguely recall hearing something about being able to run long tasks with those? Not sure. I wouldn't bother, though, I'd just head straight for a server (the example Amazon gives is starting a job to retrieve a specific item in a warehouse, sending an order to an inbox, and waiting for a warehouse worker to mark the item as retrieved...I don't know why Amazon chose that specific example, but it sounds like they have a very specific target audience!) >How do all those things not impact agility? I'm of the opinion that tools do not impact agility, decisions do. If you decide to use Lambda in a situation where a server would prevail, you're wasting time on the wrong thing. If you decide to a use a server in a situation where Lambda would prevail, you're...not really doing anything wrong, I think. It'll likely cost you more than a Lambda function but those numbers only start to matter once you actually have to care about scale. Like I said, it's ultimately about what you're trying to do. Don't put a square hole through a rectangular screw, or something like that.

obulpathi 7y ago

Or use Google Cloud, which has 90% good parts. Documentation can be a bit pain, but the services themselves are rocksolid. There are no 3/4 queuing services, just one. GKE rocks! Cloud Console is a breath of fresher, compared to AWS. Cloud Shell makes it easy to bypass firewalls for logging into instances and no messing with public keys. It's all managed for you. Use firebase if you are looking specifically for Web and Mobile Apps. Scaling to millions of users or Petabytes of data is no big deal and you don't have to rearchitect everytime your customer base grows by 10x.

rjurney 7y ago
The google services are just way easier to use but once you’ve got expert level proficiency at AWS it’s tough to let go.
- evilmushroom 7y ago
  I have to use both. I find google easier to do simple things.. but I find it lacks some of the flexibility AWS has for less simple things. Well, that and Google's managed k8s solution was down for multiple days awhile back when I was doing a comparison. Another reason I use EKS atm.... despite I think GKE is a bit better.
  - obulpathi 7y ago
    I don't know when this happened. Give GKE a try, it's really amazing. Blows EKS out of water. As per the flexibility is concerned, once you learn how to use Google Services, you can get the flexibility with simplicity. AWS services are too complex and even things like billing require a PhD degree in finance to optimize for anything non-trivial.
- zaphirplane 7y ago
  Can you provide examples, in what way they are easier
- flurdy 7y ago
  The lock-in with not just all your infrastructure with one provider but also all the skills and proficiencies with one provider's stack is a high risk that I ranted about recently: https://blog.flurdy.com/2019/06/all-eggs-in-one-cloud-basket.html https://blog.flurdy.com/2019/06/all-eggs-in-one-cloud-basket... Having your teams doing little POCs with other providers will ease your project/company cloud bus-factor significantly.

jamestimmins 7y ago

Perhaps I'm alone here, but I'd say 50% of my time with AWS goes to (attempting to) properly configure the VPC as well as the IAM roles. For the majority of small-ish projects, the hardware isn't nearly as important or difficult as properly configuring access rules between services and outside parties.

rjurney 7y ago
IAM is a killer. You’re absolutely right. Are GCP permissions easier? I haven’t used them as much.
- chishaku 7y ago
  night and day imho
- mey 7y ago
  https://github.com/tilfin/aws-extend-switch-roles/blob/master/README.md https://github.com/tilfin/aws-extend-switch-roles/blob/maste... may be interesting to you. The fact that this has to exist is also interesting...
- usmannk 7y ago
  I'll hate on GCP all day long but the one thing it has going for it is the permissions.
  - wodenokoto 7y ago
    Wow, I was gonna say “haven’t tried IAM on other platforms than GCP but I can’t imagine it being more difficult than this”, but damn ...
- jsmeaton 7y ago
  I’ve stopped worrying about minimising IAM permissions and tend to just use the built in AWS roles for most things now.
  - 7y ago

whycombagator 7y ago

And if you want something even easier (albeit with more fixed/upfront costs) just use Heroku.

DVassallo 7y ago
I agree. I have no direct experience with Heroku so I can’t comment on the experience/restrictions, but if a PaaS works well for what you’re doing, I’d go for it.
- bubble_talk 7y ago
  I was also thinking the same, especially considering that you start with this as your main reason: "Anything whose main value proposition is about “ability to scale” will likely trade off your “ability to be agile & survive”."

cosmotic 7y ago

Step one: don't use Twitter for this sort of thing

xtracto 7y ago
Thank You! I thought I was the only one that hated the format of these things. Why not writing a proper article and linking it on Twitter? It would be way better

mgamache 7y ago

All depends on what your goal is. Are you looking for large scale out? Or are you iterating fast and don't want to mess with managing databases, CDN's, KV stores etc...? Note: A version of premature optimization is premature scaling. So beware of those offerings that require you to use a certain technology to enable AWS scale-out it can slow you down.

merlincorey 7y ago

I am in agreement with everything generally except for CloudFormation and setting up Route53 zones manually. I use Terraform to setup my infrastructure as service on AWS, including Route53 zones. After the latest 0.12 upgrade to the language, Terraform is quite a bit more user friendly than CloudFormation, and importantly, not locked down to just Amazon -- it supports multiple clouds and on-premises solutions for declarative orchestration of resources.

DVassallo 7y ago
Author here. Fair. I never used Terraform beyond playing with it briefly, so I can’t really comment on the differences. I tend to choose CFN mostly because I’m familiar with it.
- oppositelock 7y ago
  I'm in charge of cloud infrastructure at my company, and I've found that sticking with CloudFormation on AWS is more powerful and versatile than TerraForm, since TF isn't always a perfect mapping, where CFN is, with the caveat that it's sometimes months behind what their API can do. To that end, over the years, I've come across a couple of tools which help write CFN templates and manage rollout keeping cross-stack dependencies in mind. If anyone does a lot of CFN, I highly recommend Troposphere (python abstraction for cloudformation) [1] and Stacker (manager stack rollout via dependency DAG) [2] 1: https://github.com/cloudtools/troposphere https://github.com/cloudtools/troposphere 2: https://github.com/cloudtools/stacker https://github.com/cloudtools/stacker
  - philsnow 7y ago
    Fwiw terraform also lags behind what the underlying apis can do -- but it's a community effort and you can contribute to add the functionality you need, whereas with CFN you can only ask your TAM (iff you have one!) to ask the internal team(s) (not sure if only CFN team or only the service team or both have to implement something) to prioritize your feature request.
evilmushroom 7y ago

diminoten 7y ago

Sort of related to this, but can someone explain to me why Zappa[1] isn't a bigger deal? It completely delivers, as far as I can tell, on the promise of deploying a WSGI app into a Lambda in a way that requires minimal configuration. And yet, the project seems to have stagnated somewhat. Maybe it just did everything it set out to do? I'm not sure, but when people say "forget about Lambda because you'll have to roll your own shit" I wonder if they've heard of Zappa or not. [1] https://github.com/Miserlou/Zappa https://github.com/Miserlou/Zappa

KaiserPro 7y ago
YES. Zappa got me out of a massive hole several times. I'm not a webdev, and we arn't a web company, so we don't have a web deployment pipeline. Having Zappa makes our life very simple and cheap. The best part is that local and "prod" are equally easy to spin up. The simple integration to DNS, SSL, api gateway, is a wonder. The only downside is the permissions management is a massive pain, and assumes that you have admin permissions.
takeda 7y ago
I tried to use Zappa and my experience was that it didn't support tires of callbacks I needed it to. After I patched it up to do what I want I realized that handling calls directly was actually simpler than using Zappa and Zappa made my application huge. I understand that primary use of Zappa is a web application, but I have reservation about that as well. The "serverless" applications aren't really server less, they just have VMs spun on demand and it's only good idea when you have a service that most of the time is idle.
The_Amp_Walrus 7y ago
I've tried to use Zappa for a non-trivial personal project to run a Django app and I found: - Super nice, low config API - easy to deploy - great out-of-the-box support for async tasks (no Celery! woo!) but: - It was a nightmare getting certain Python libraries (eg. Pillow, psycopg) to work in the AWS Lambda environment - It really sucked having to deploy to AWS in order to debug issues which cannot be reproduced locally (eg. library) - It seems hard to get away from using AWS tools to observe your code in prod (eg. CloudWatch for logs) - I still needed a database to maintain state, DynamoDB didn't work with Django's ORM and was surprisingly expensive, and if I'm going to shell out $10/mo for a Postgres RDS instance then I may as well run the whole thing on EC2 anyway I think Zappa is a really nice tool for some niche use cases, and I'd definitely turn to it if I needed to stand up some small, stateless serverless service, but I would hate to support and debug it as a web app in prod. project: https://memories.ninja https://memories.ninja zappa version: https://github.com/MattSegal/family-photos/tree/lambda-hosting https://github.com/MattSegal/family-photos/tree/lambda-hosti... ec2 version: https://github.com/MattSegal/family-photos https://github.com/MattSegal/family-photos

philwelch 7y ago

I wouldn’t recommend using single EC2 instances in production ever, but the rest seems solid.

greyskull 7y ago

The CDK[0] will hopefully help with the cloudformation story. [0] https://github.com/aws/aws-cdk https://github.com/aws/aws-cdk

manigandham 7y ago

Completely agree on just using a server instead of the various lambda-style systems. All modern languages have great web/app frameworks that make it incredibly easy to build, whether it's a single endpoint or a giant app. The ability to just include whatever code you need and deploy it atomically is massively underrated. Also agreed on scale, servers are fast and cheap and the savings from Lambda rarely pays off in the extended effort. When you do scale, Lambda becomes more expensive anyway. I do recommend using Docker though. Containers are more portable and easier to deploy and replace on a running server, along with the ability to run multiple instances, mount volumes, setup ports and local networks, and eventually migrate to something like ECS/K8S if you really need it.

tus88 7y ago
Extended effort to push up a lambda function and not have to worry about automating deployment and configuration and patching and monitoring and upgrading and fail over-ring and, yes, scaling? Maybe its just me but I'd rather not see the backend of a server ever again for anything other than development.
- bharam 7y ago
  > not have to worry about automating deployment How do you validate your code before deploying to production? If you test in environments besides production how do you manage configuration settings for the different environments (i.e. a db connection string)? How do you avoid patching? Almost any code I've written takes dependencies on 3rd party libraries and those will often have security vulnerabilities (usually some time after I wrote the code).
  - tus88 7y ago
    I mean of infrastructure, servers etc. I.e. the days of Puppet and Ansible are behind me. And patching a Lambda is as simple as pushing up a new version. No downtime, restarting services etc. And no patching of OS or Docker containers. Or building them. Configuration is as simple as environment vars or SSM for secrets.
- 7y ago

etaioinshrdlu 7y ago

I find the characterization of RDS Aurora as not proven, or unstable, as odd. I thought it looked like one of the more mature-seeming choices on the market. Any thoughts?

vasco 7y ago
From the point of view of someone who's used Aurora Postgres since it came out, though you sometimes are exposed to bugs, AWS support has always been great and we never faced anything super serious. It's been almost a year since we had any problems though. This on a somewhat large aurora instance at around 5TB, so it's already representative for a lot of people.
- takeda 7y ago
  Right now if you use Aurora Postgresql 9.6 you don't have an easy way to migrate to 10, and 11 is not even available. They supposedly are working on a solution, but won't disclose when it will be available.
sciurus 7y ago
It's way less mature than vanilla MySQL or postgresql.

foobar_ 7y ago

Docker is bullshit. AWS images are alright IME. Spinning multiple large instances will give you more bang for the buck than dealing with tiny docker containers.

abiro 7y ago

I don't think is good advice in general. OP is obviously used to doing things in a certain way and good for him for working the way he is most productive. But looking at things objectively, everything has a learning a curve and once you get past that, Lambda is way more efficient even when working alone due to the agility it allows and time saved on maintenance.

k__ 7y ago

Anything whose main value proposition is about “ability to scale” will likely trade off your “ability to be agile & survive”. That’s rarely a good trade off. Interesting that he sees micromanaging infra on AWS as more agile than using managed services. Also, serverless systems don't cost you money when you(r customers) don't use them, so that's a huge plus on survival. Finally, code you write is always a liability, if you got a good architecture (hexagonal, etc.) you can just swap out one service that saved you time in the past with another service that will save you time in the future.

awinder 7y ago
Yeah for launching startup level traffic I totally agree, lambda feels like the right arch at the right cost. Also it will engrain setups that, like you said, you can swap out. You’ll need an alb / api gateway, you’ll need to get logs flowing through cloudwatch or some other non-host logging mechanisms. Transferring onto these kinds of architectures from not-these-architectures has been a royal pain and a time drain in my experience.
gtirloni 7y ago
Yeah, kind of made me cringe a bit (considering his experience).
freehunter 7y ago
>serverless systems don't cost you money when you(r customers) don't use them All the horror stories I see surrounding huge surprise Lambda bills always bring me back to this point. If I have to pay $5/mo for a server I only use for two hours once every few months, that's something that should go on Lambda. If it's something that I'm using constantly, all day every day, a server will be cheaper. If I only use my car once every few weeks, Uber makes a lot of sense. If I use it every day back and forth to work and the grocery store, Uber's gonna be a lot more expensive. Lambda is for small tasks that don't execute often. And using it right can save startups gobsmackingly large amounts of money.
- k__ 7y ago
  Yes, it comes all down to risk assessment. Maybe using serverless technology is too hard for your corp, because you don't have the skills, so it could lead to problems in the future (surprise Lambda bills). But it could also be that your competitors get a huge advantage by investing in the serverless paradigm and run you away in the future. To me that Twitter thread sounded too much like a guy who invested in some tech over the last 11 years and now tries to convince potential customers of him to use the tech he knows about. He could be right, he could be wrong. I don't know. I started back-end development with serverless, so I'm biased in the other direction, haha.

ookblah 7y ago

I think this is really great for getting stuff up and running as quick as possible, maybe if you're just starting out, but I'm really surprised at some of the things said. Maybe if this is a site that you run on the side. Doing test/staging on a nano and then pushing to production on an m5? What? Like get ready to troubleshoot random issues completely unrelated to your code. And then equating different AMI's to containers is vastly oversimplifying things. With docker I can just get up and running on my local box and every dev I share this with has the exact same environment and config. I can package up those images and send them up to ECS/Kubernetes and not deal with the same headaches. The learning curve is a bit steeper, but absolutely worth it. I walked the same path here (starting out with single VMs for environments, installing services locally on my laptop) and the headaches down the line are not worth it and it isn't even a case of over-thinking the solution. You don't need to ride the cutting edge but modern tooling saves a ton of time. It was a PITA migrating our old VM stuff over, but absolutely worth it. If you don't want to deal with maintaining systems there are other solutions mentioned ranging from full PaaS to something like GKE.

mpmpmpmp 7y ago
My favorite part about using ECS was never being able to deploy because our nodes had jacked up ecs agents on them. So the application was still running but you couldn’t touch it. We switched to GKE last year and haven’t had a single problem with infra since. YMMV but for simpler docker/k8s apps GKE has been so easy in my experience.
- eatonphil 7y ago
  ECS Fargate is an option for fully managed container runtime/scheduling on AWS today. We're looking into using this for deploying applications. Anyone here had experience with it?
  - cle 7y ago
    I’ve used it for both running a website and backend batch processing, and it has worked great so far.

foxhop 7y ago

I'm a core dev of stacker, it's worth taking a look at if you maintain Cloudformation templates. Makes life a lot better. At Remind (remind.com) we have nearly 600 separate Cloudformation stacks which build and maintain our stage and prod environments. This would be insane without stacker. As for this tweet, I'm fairly confident he is talking about a side projects or early startups and I agree with most of what he said. Personal, I don't use AWS for side projects, I use offerings from Linode, Digital Ocean, and vultr. They are cheaper and scale up vertically with a click of a button, which is really what you need for the first couple of years, unless you hit the growth/scaling lottery, which isn't typically the case. For example, over the last two weekends I was able to use Digital Ocean Spaces (alternative to AWS S3) to build my wife a secure digital downloads store. All uploads and downloads use presigned POST and GET urls created via Boto3! I was suprised how perfect Digital Ocean implemented S3's API in their Spaces offering. https://russell.ballestrini.net/pre-signed-get-and-post-for-digital-ocean-spaces/ https://russell.ballestrini.net/pre-signed-get-and-post-for-...

nijave 7y ago

Lambda is absolutely essential for glueing together Amazon services and many Amazon articles recommend it (for instance, getting your ASG to drain ECS instances before scaling down). It's also helpful for building alerting/monitoring workflows since Cloudwatch/SNS is pretty simplistic on its own I think the sentiment is more "don't build your entire app on lambda"

wongarsu 7y ago
I think "don't have any customer facing Lamdas" is a good rule of thumb. It's great for glue code, if-this-then-that code and cron jobs.

cwyers 7y ago

Why not just use Linode or Digital Ocean at this point?

nijave 7y ago

I tend to disagree with the EC2 sentiment. Any cost savings using VMs over a higher abstraction will likely be wiped out by the time wasted becoming a sysadmin unless you just like doing sysadmin stuff (patching, access management, config)

ericcholis 7y ago
This is a very solid point and something to consider inside of the entire infrastructure planning stage. Opportunity cost could be huge, or net zero depending on your needs.

otabdeveloper2 7y ago

> 1/25 Is there a more user-hostile webapp than Twitter? I think not.

fnord77 7y ago

> Forget that all these things exist: ... Containers, Kubernetes, Docker. Not sure how this guy can give this advice. containers make rapid development so much easier. There's no distinction between your dev environment and your prod environment except for a couple properties/env variables and some scaling factors. Also "no" to cloudformation and "yes" to terraform. And if I were to write a simple application, I'd probably try with lambda next time around.

aequitas 7y ago

> But Autoscaling is still useful. Think of it as a tool to help you spin up or replace instances according to a template. If you have a bad host, you can just terminate it and AS will replace it with an identical one (hopefully healthy) in a couple of minutes. I really started to enjoy AWS when I got into the habit of deploying all EC2 instances using Autoscaling groups. Even single node instances and configurations that don't need any scaling, just everything. Autoscaling is free and it forces you into the immutable/disposable infrastructure paradigm, which just makes administration of the nodes just so much easier. And for all stateful stuff there is RDS, S3, dynamo, SQS, etc.

deleted 7y ago

[deleted]