9 ms·
Forget CDK and AWS's insane costs. Pulumi and DigitalOcean to the rescue
- mavdi 2y agoHi everyone, We've gone through a lot of pain to get this blueprint working since our AWS costs were getting out of hand but we didn't want to part ways with CDK. We've now got the same stack structure going with Pulumi and Digital ocean, having the same ease of development with at least 60% cost reduction.
- vundercind 2y agoKeep an eye on reachability and performance. I’ve seen DO consistently perform terribly and/or drop connections for months (that is, didn’t look like some brief routing glitch somewhere) for some US and Canadian routes (not, like, Sri Lanka or something) on excellent Internet connections. The fix was moving to AWS, problem gone. It felt like a shitty-peering-agreements issue.
- nostrebored 2y agoPeople will pretend that this quality difference doesn’t exist in networking, uptime, server quality. It’s not a drop in replacement. It might be worth it depending on what you’re doing.
- vundercind 2y agoFrustratingly, it’s also something that doesn’t meaningfully appear on any features list or comparison sheet.
- data_marsupial 2y agoHow do you monitor the connection quality?
- vundercind 2y agoFrom the client side. You can’t know what it should be like without knowing the client. I’m sure there are lots of DO clients seeing the same things we did, but not realizing it. We did see it (multiple DCs—we didn’t just not try to fix this before going to AWS) in multiple cases with tens of clients so if there’s good news it’s that if you can monitor like 100 clients distributed over a wide area and all of them behave as expected you may not be experiencing what we did. What we saw was closer to 5% with absurd slowness or frequently-dropped connections than to 0.01%. And if you are just operating a website and sticking Cloudflare or whatever in front of DO anyway, this doesn’t matter. I expect that’s why it’s not a more widely-reported issue.
- skywhopper 2y agoPlease change the title text unless you add some discussion of the cost differences to the page you linked. However useful your tool is, nothing on this page mentions AWS or costs.
- magamanlegends 2y ago[dead]
- mise_en_place 2y agoEKS has become a clusterf*ck to manage and provision. This looks very useful. Bare metal k8s, even running on EC2, might be another option.
- GauntletWizard 2y agoYou don't choose EKS because it's easy to manage. You choose it because you intend to use the bevy of other AWS hosted services. The clusterfuck of management is directly related to that. The alternative, which I feel is far too common (and I say this as someone who directly benefits from it): You choose AWS because it's a "Safe" choice and your incubator gets you a bunch of free credits for a year or two. You pay nothing for compute for the first year, but instead pay a devops guy a bunch to do all the setup - In the end it's about a wash because you have to pay a devops guy to handle your CI and deploy anyway, you're just paying a little more in the latter.
- trallnag 2y agoWhat's your issue with EKS? I operate several very simple and small single-tenant clusters, and I have to touch the infrastructure only once a year for updates
- petcat 2y agoKubernetes no thanks. Terraform + Kamal [1] on Digital Ocean is the way I deploy/run apps now. [1] https://kamal-deploy.org/ https://kamal-deploy.org/
- mati365 2y agoPlain Podman systemd integration is way more powerful and secure, as it does not mess with firewall and allows to run rootless containers using services. It's even possible to run healthchecks and enforce building images just before starting service making on-demand containers using systemd-proxyd possible. Check example: https://github.com/Mati365/hetzner-podman-bunjs-deploy https://github.com/Mati365/hetzner-podman-bunjs-deploy
- petcat 2y ago> way more powerful and secure I don't care about powerful. That's the opposite of what I want. I could just use k8s if I cared about that.
- mati365 2y agoIt looks like you don't even care about opening documentation before pressing reply. Podman is a simple hammer without any moving parts, that used properly can be used to build fancy stuff without much knowledge.
- ngrilly 2y ago
- lysace 2y agoPulumi is very neat with straight AWS, too. I suspect this is the primary use case.
- pmarreck 2y agoAnyone use Garnix? https://garnix.io/ https://garnix.io/
- mplewis 2y agoThis looks too experimental for me to trust with production deployments.
- turtlebits 2y agoI wish CDK was fully baked enough to actually use. It's still missing coverage for some AWS services (sometimes you have to do things in cloudformation, which sucks) and integrating existing infra doesn't work consistently. Oh and it creates cloudformation stacks behind the scenes and makes for troubleshooting hell.
- Aeolun 2y ago> sometimes you have to do things in cloudformation, which sucks All of CDK does things in cloudformation, which made the whole thing stillborn as far as I’m concerned. The CDK team goes to some lengths to make it better, but it’s all lambda based kludges.
- liveoneggs 2y agoso like every other aws "solution"
- LunaSea 2y agoThe biggest hurdle I've encountered is cross-stack resource sharing, especially in case of bidirectional dependencies like KMS keys and IAM roles.
- 8note 2y agoThe biggest hurdle is when you want to refactor your stacks, and you pretty well just can't, without risk of deleting everything
- irjustin 2y ago> you pretty well just can't, without risk of deleting everything This is one hyper annoying area. It is possible to get around it, but it's ugly, drop to L1 and override logical id: let vpc = new ec2.Vpc(this, 'vpc', { natGateways: 1 }) let cfnVpc = vpc.node.defaultChild as ec2.CfnVPC cfnVpc.overrideLogicalId('MainVpc') You have to do this literally for every resource that's refactored. For us, we run 2 stacks. One that basically cannot/should-not be deleted/refactored. VPC, RDS, critical S3 buckets - i.e. critical data. The 2nd stack runs the software and all those resources can be destroyed, moved whatever w/o any data loss.
- Aeolun 2y agoI don’t think Digital Ocean is all that much better for pricing, but using Pulumi over CDK is a pure win as far as I’m concerned.
- CSMastermind 2y agoYeah, I've been really disappointed with Digital Ocean so far. Not just from a pricing perspective but from a customer service perspective. Anyone using CDK should switch to Pulumi though.
- JamesSwift 2y agoAgreed. On the bright side, I was able to migrate managed k8s on DO to managed k8s in GCP with very minimal work since it was managed via pulumi.
- thelittleone 2y agoPerhaps Pulumi with Vultr is also worth a look.
- thinkindie 2y agoPulumi genAI-based documentation is trashed. I've moved to terraform and i was able to achieve much better results in shorter time thanks to higher documentation level for terraform.
- tholm 2y agoWorth noting that most of the terraform documentation for classic pulumi providers (providers build on top of TF providers) is still relevant to Pulumi.
- skywhopper 2y agoThis title text is nowhere on the linked page. Please get rid of the editorialization. DO is not that much cheaper for a baseline instance.
- jmspring 2y agoPulumi is really a royal piece of shit. Why the f*ck am I writing code to do "deployment". In C# --> new Dictionary<string, object> when dealing with a values.yaml for instance. The whole need to figure out when and when not to use Apply. Give me Terraform (as much as I hate it) any day.
- stackskipton 2y agoAs SRE dealing with former Pulumi, "Hey Devs can use code to deploy infrastructure" is not great idea you think it is. I've seen some real ugly conditional behavior where I'm like "Is this or is this not going to run? I honestly can't tell."
- hinkley 2y agoWe had so much conflict with the ops team over their choice of Terraform. The three colors of variable thing is just fucking bonkers. Getting tests wrapped around it that actually did what we thought they meant was a giant pain in the ass. I won't go as far as to say we burned bridges arguing back and forth about it but they were definitely significantly singed. Config files simply don't work until they do. And if it's your job to stare at them for hours and hours a day then maybe that's okay with you, but if you expect other people to 'just learn' it you're an idiot or an asshole. Or both. Ain't nobody got time for magic incantations. I also think it should tell you you're on the wrong path when your app is named after a verb and the data it deals with is all declarative.
- darkstar_16 2y agoEver thought that "Ops" needs a different mindset than the devs are used to ?
- deleted 2y ago[deleted]
- hinkley 2y agoAnd that’s why we don’t delegate that work to devs.
- jmspring 2y agoOne thing about managing EKS with Pulumi, Terraform, etc. if you deploy things like Istio that makes changes to infrastructure. Do a Terraform destroy - no luck, you are hunting down maybe some security groups or other assets Istio generated that TF doesn't know about. Good times.
- nixdev 2y agoDigital Ocean isn't really a "real" cloud. Maybe use Digital Ocean if you're hosting video game servers, but no serious business should be on it.
- Sohcahtoa82 2y agoI wouldn't even use DO for that, unless it's like a private server for just your friends. I won't touch DO after they took my droplet offline for 3 hours because I got DDoS'd by someone that was upset that I banned them from an IRC channel for spamming N-bombs and other racial slurs.
- aitchnyu 2y agoWhen was this? Now DO and Linode promise full DDOS protection.
- Dylan16807 2y agoWhat's your definition of real cloud? And can you name a real cloud that charges a half-reasonable price for bandwidth? I consider $10/TB to be half-reasonable.
- 15155 2y agoIdeally one that doesn't have these kinds of issues: https://news.ycombinator.com/item?id=6983097 https://news.ycombinator.com/item?id=6983097
- Dylan16807 2y agoThat was more than ten years ago, I don't think that tells us about current quality.
- nixdev 2y agoWhile yes, it was more than ten years ago, we can see that such stupidity is woven into their DNA as a company. TL;DR: where a cloud provider hosts customers for which there are real-world consequences for data leakage, not a single customer can be at-risk for data leakage. It's a different line of thinking, almost "a different world", to those who have this line of thinking vs those who do. "The thing about reputations is you only have one". By contrast even more than ten years before that, AWS was publishing whitepapers about how all contents of RAM to be used by a VM are initialized before a VM is provisioned, and other efforts to proactively scrub customer data. I worked at a niche cloud provider a bit over ten years ago. We used Intel QAT for client-side encryption for our network attached pools of SSD. We were able to offer all-SSD at low cost and without security blindspots by crypto key rotation implemented by compartmentalized teams and also physical infrastructure compartmentalization patterns. Which, about half a decade later we found we were second only to AWS and almost second (but ahead of in other ways) to some smaller cloud-style hosting provider.
- kristianpaul 2y agoIs this an Ad?
- nextworddev 2y agoGitHub has been littered with developer relations growth hacks recently.
- wordofx 2y agoIt’s only “insane costs” if you don’t know what you’re doing.
- yieldcrv 2y agoand even if you do, it’s usually a system design problem that you’re maintaining on one hand, I can see how this is an unfalsifiable standard, on the other hand I can see the utility of solving a friction for people that messed up
- postalrat 2y agoOr need a good amount of ram. Which should be really cheap these days.
- hinkley 2y agoMy life on AWS the last five or so years really would have been a lot simpler if every new generation of EC2 servers didn't have the exact same ratio of RAM to cores.
- zokier 2y agoAt this point the memory:vcpu ratio is the defining characteristic of main general purpose C/M/R series, I'd think it would be pretty disrupting to change that significantly anymore. And they got also the special extra-high memory X series available. I would say ec2 is pretty flexible in this regard, you have options for 2/4/8/16/32 gigabytes per vcpu. It's mostly problem if you need even less memory than what C series provide, or need some special features.
- hinkley 2y agoAs products age they tend to use more memory. Add in space/time tradeoffs asking to use more. You either get stuck applying the brakes trying to keep the memory creep at bay, or you give in and jump to 2x the memory pool which will disappear too. The old solution in on-prem was to populate machines with 2/3 to 3/4 of their max addressable memory and push back on the expensive upgrade as long as possible, or at least until memory prices came down for the most expensive modules. Then faster hard drives or new boxes are the next step.
- nextworddev 2y agoControversial opinion here: just use CDK. Learn cloud formation for advanced stuff. It’s really not that hard and pays dividends
- ptdorf 2y agoIn my experience AWS' CloudFormation is limited in the number of resources and exposed APIs than any of the CDK.
- nextworddev 2y agoAWS service teams provide cloud formation support before CDK support in many cases, so eventually CDK users run into situations where they need to look at CF
- coredog64 2y agoJust learn CloudFormation. It’s not that hard, and if you really want to write code, you can implement custom resources for all the times the service team let you down.
- __turbobrew__ 2y agoCDK is a second class citizen, it is missing implementations for many services and features. CDK was DOA as it should have been a requirement that when AWS added something to terraform it needed to be added to CDK as well.
- nothrabannosir 2y agoFor anyone deliberating between Pulumi and CDK let me recommend what I consider the best of both worlds: CDKTF, Hashicorp’s answer to Pulimi (my quote not theirs). It’s got everything you want: - strong type system (TS), - full expressive power of a real programming language (TS), - can use every existing terraform provider directly, - compiles to actual Terraform so you can always use that as an escape hatch to debug any problems or interface with any other tools, - official backing of Hashicorp so it’s a safe bet It’s a super power for infra. If you have strong software dev skills and you want to leverage the entire TF ecosystem without the pain of Terraform the language, CDKTF is for you. (No affiliation) https://developer.hashicorp.com/terraform/cdktf https://developer.hashicorp.com/terraform/cdktf
- ivantop 2y agoHow is compiling to terraform a positive? I'd rather debug python than python-compiled-to-terraform.
- nothrabannosir 2y agoBecause you can use that to interface with existing tooling. Terraform has a huge and established ecosystem and it’s an uphill battle to compete with it. It’s risky to bet your infra on a tech that tries to drink the ocean and supplant the entire thing. Meanwhile if you compile down to TF you get to use a different language without having to pay the cost of moving out of the tf ecosystem. And given that the language itself is by far the worst thing about terraform that’s a big win. It turns out terraform is actually quite acceptable when you slap a decent language on top of it. Passable, even :)
- ivantop 2y agoMakes sense! Except for one little thing.. We've been migrating off of Terraform at BigCo recently and it has been a tremendous success. The migration has saved countless hours. Before, I was jaded and routinely in the office until 8 or 9 or so manually running terraform deploys for our engineering teams in India. Now, thanks to Pulumi, I'm able to leave the office at 7:30-8 -- and I can tell you single handed that this has saved my relationship with my daughter and maybe even my marriage. I'm running the fastest for loops thanks to Pulumi. We actually compile our Python down to c and use the Pulumi C SDK for insane speed benefits when we loop over our datacenter arrays. Turns out, not having bounds checks shaves off valuable time that I would otherwise be spending with my daughter. Routinely I'd be waking up screaming at 4 in the morning due to Terraform (or, what we would refer to as Tearaform because all of the infra engineers were constantly in tears). Now, I can sleep soundly until 5:30.
- fulafel 2y agoWhy's everyone going away from declarative? Terraform, CloudFormation, AWS Copilot etc have a lot of virtues and are programming language agnostic. Using a complex programming language (C++ of the browser world) just for this has a big switching cost. Unless you're all in on TS. And/or have already built a huge complex IaC tower of babel where programming-in-the-large virtues justify it.
- pjmlp 2y agoBecause they like to spend endless hours debugging infrastructure builds.
- jnsaff2 2y ago> Why's everyone going away from declarative? If I had to guess it's because - more imperative background developers need to work with infrastructure and they bring over their mindset and ways of working - infrastructure is more and more available through API's and it saves a lot of effort to dynamically iterate over cattle than declaratively deal with pets - things like conditionals, loops and abstractions are very useful for a reason - in essence the declarative tools are not flexible enough for many use cases or ways of working, using a programming language brings infinite flexibility Personally I am more in the declarative camp and see the benefits of it, but there is certain amount of banging ones head against it's rigidity.
- 1dom 2y agoComplex programming languages for infrastructure code get used when people who are more comfortable using complex programming languages to solve their problems are given the problem of infrastructure and ops. It is classic "every problem is a nail to the person with a hammer". Complex languages - by definition - can solve a wider variety of problems than a simple declarative language but - by definition - are less simple. Complex languages for infra - IMO - are the wrong tool for the wrong job because of the wrong skills and the wrong person. The only reason why inefficiencies like this are ever allowed to happen is money. "Why hire a dev and an ops when we can hire a single devops for fractionally less?" - some excited business person or some broken dev manager, probably.
- nsonha 2y ago
- mythz 2y agoHetzner has been our "expensive AWS cloud costs" saviour We've also started switching our custom Docker compose + SSL GitHub Action deployments to use Kamal [1] to take advantage of its nicer remote monitoring features [1] https://kamal-deploy.org https://kamal-deploy.org
- KronisLV 2y agoI’ve been pretty happy with something like Docker Compose or Docker Swarm and Portainer, but honestly it’s nice that there are other alternatives that strive for something manageable and not too complex!
- nasmorn 2y agoMy DO K8S cluster ist bugging me every couple of months to do an upgrade. I am always scared to just run it but moving shit over to a new cluster instead is so much work that I simply gamble on it. AWS ECS is worth over penny
- katdork 2y agoDO's K8S is more equivalent to AWS's EKS offering, so of course ECS which abstracts away pretty much all of the other parts of K8s is going to require less maintenance. It's sort of a false equivalence to say ECS == that solution. On EKS, you need to do the same version updates with the same amount of terror. You do pay the extra for the further management to just run containers somewhere! (you might want to say "every" instead of over, "is" instead of "ist")
- nasmorn 2y agoI definitely want to say is instead of ist but it is bugging me every couple of months. You do the upgrade and 6 months later it needs another one. No LTS in sight
- icar 2y agoI strongly recommend sst.dev
- giorgioz 2y agoCDK APIs in JavaScript are very nice. It's a much much developer experience than Pulumi/Terra form and even Server less Framework. In our monorepo each service is in a separate folder with a folder called /infrastructure inside with a file called Stack.js that defines all the resources needed. When starting a new service we just copy one of the last similar services that we developed. We are able to deploy a new service in hours. Services are getting better and better with accumulation of nice to have features that you wouldn't have time to add to most services.
- lazzurs 2y agoThis doesn’t sound good to me. Would you do the same with some functional code rather than creating an external versioned library? Terraform or CDK I would want a simple shareable thing that did the boilerplate that I called with any variables I needed to change.
- RoxaneFischer1 2y agoI personally love terraform. It's easy to use and actually it's rigid framework allow to make less mistakes/way more readable than pulumi
- strzibny 2y agoYou can also simplify Kubernetes to just Kamal and things become instantly easier...