10 ms·

It's interesting how AWS can keep so high prices on these. But it's just the beginning, the real money comes from when they convince you to run over a dozen vm

It's interesting how AWS can keep so high prices on these. But it's just the beginning, the real money comes from when they convince you to run over a dozen vms/containers (all needing storage etc of course). You need to be triply redundant on 3 availability zones, (3x) both with the RDS db cluster and app containers (2x) . And then have separate dev/staging/prod envs (3x). That's 18x. You can then get a pat on the head ("pass AWS well-architected review"). Then they innovate "serverless" stuff where you can pour developer man months and years to save on this overpriced things by making everything more complex, hard to monitor/debug and learn new aws specific techs, so you can get faux savings. Here they're buying your long term lock-in by getting AWS specialists and advocates on your long term payroll. They're doing certifications and investing in AWS careers that can carry over to the next employer too. And don't even get me started on how much work by this time has gone to build the infra-as-code to manage the rube golderg machine, you'll (seriously) have more lines of CDK code than app logic (that was slower to develop & harder to debug than your actual app code per line). Just about now someone in your newly grown cloud herding engineering org champios the idea "I know what will help, let's start using Kubernetes". They've probably never costed / self administered a server in production use, or if they have, they know to keep quiet.

philliphaydon 4y ago
> You need to be triply redundant on 3 availability zones, (3x) both with the RDS db cluster and app containers (2x) . And then have separate dev/staging/prod envs (3x). That's 18x. Why would you care about the redundancy on staging / dev? Just making up things to inflate AWS costs now.
- fulafel 4y ago
  If you're not testing az failovers you're probably just wasting money, like untested backups... But it's true that most people don't know this and aren't testing it because they don't know that az failovers don't automatically just work(tm). And the second downside of course would be asymmetry between the envs and divergence in the IaC etc resulting in more complexity and engineering work. (Of course you're probably wasting money anyway since you don't actually business-wise need better uptime than the single AZ, and your complexity induced human fumbles will cause much more outages anyway, but this has been a main selling point of the decision to go to AWS, so the requirement needs to be defended) Yes, you can build automation to have the redundant stuff up only sometimes, if you eat the engineering effort and complexity in your IAC and build automation... in the general vein of justifying engineering spend to offset AWS operating costs where running containers is very expensive! TLDR: either way you end up paying for the very high markup in compute prices, it'll just be easier to excuse jumping through expensive hoops to "save money" on it.
  - philliphaydon 4y ago
    If you’re building your own infrastructure in a data center then sure you absolutely want to test your redundancy. But with AWS it’s a checkbox. It’s transparent to you and your applications. The infrastructure to host in multiple AZs is already in place. The only real issue with MultiAZ is the failover in RDS depending on the database you use could be seconds or 10s of seconds.
    - fulafel 4y ago
      IME not so in the real world, you'll have accidental state in your distributed system outside the DBs. You'll have some stuff that actually always runs in one AZ in normal circumstances and your integration partner in another org has whitelisted only its IP. Etc etc, everything that is untested will find ways to conspire to rot. Especially if you haven't learned by seeing these bugs so you can avoid them. Also you won't have a clear experience and understanding of what happens in the failover and you'll won't know to avoid failover breaking mistakes in your VPC configs, security groups, frontend-backend shared state etc. (And by "you" I mean "your dev team", it's not enough that one guy gets it). Also^2 if you read the news about all the outages it's very common for failover systems to fail generally, not just AWS - the general engineering wisdom is: always test your failovers. And there's no substitute for end-to-end testing it, instead of individually testing each layer/module. (Bad: "we can skip testing db failover", good: "let's test that the whole system works when there's a az failure")
      - russellendicott 4y ago
        Dealing with this now for a client. Can't test Redshift AZ relocation feature because there's no way to simulate AZ failure. Only safe bet is full multi region with DNS switcheroo.
    - moonchrome 4y ago
      >But with AWS it’s a checkbox. It’s transparent to you and your applications. The infrastructure to host in multiple AZs is already in place. The only real issue with MultiAZ is the failover in RDS depending on the database you use could be seconds or 10s of seconds. Have you actually seen this work on your project in practice ? Like a region go down and another region picks up automatically and it kept working just by switching a checkbox ?
- arinlen 4y ago
  > Why would you care about the redundancy on staging / dev? If you deploy to multiple regions then it wouldn't make no sense at all to have a single preprod stage, specially to run integration tests. Also, keep in mind that professional apps support localization, and localization has a significant afinity with regional deployments. I mean, if you support japanese and have a prod stage on Japan, would you think it was a good idea to run localization tests in a us deployment?
  - philliphaydon 4y ago
    Currently have a deployment in Oregon and cloudfront servicing most of the world. With localisation. No need to deploy in Japan to support Japan. The only tricky one is China because of the firewall. Tho that hasn’t been an issue for the last few years as they haven’t been blocking cloudfront completely. (This has been running for 10 years in AWS without issue)
    - arinlen 4y ago
      > Currently have a deployment in Oregon and cloudfront servicing most of the world. With localisation. So you're serving static assets through CloudFront with a single backing service. Congrats, you managed to have a service that doesn't offer regional services. Also, you definitely don't support clients in China, or enjoy shooting yourself in the foot. Most professional applications with a global deployment and paying customers don't have the benefit of uploading HTML and calling it done.
      - philliphaydon 4y ago
        So 10+ years of supporting broadcasters and creative agencies around the world from 1 region is not supporting those countries. Got it. I mean there’s services in those regions for the tasks performed but even then it doesn’t require the level of testing you’re assuming it does.
        sieabahlpark 4y ago
- raffraffraff 4y ago
  Even if you don't want it sometimes you're forced to run multiple AZs (eg: EKS requires 2x). But that 18x figure is nuts. VPCs, AZs, subnets, IAM etc are free. The cost comes from what you deploy into them. So separate environments don't have to be as expensive as production. You can scale them to zero, use smaller computer instances, run self-managed versions of expensive stuff (like DBs) or simply run small single-instances of RDS instead of large redundant clusters. Non-prod environments they're a great place to experiment with aggressive scale-down on cost while observing performance.
  - cutthegrass2 4y ago
    Whilst the VPC itself is free, they get you on the NATGW - which you probably need along with the VPC in most cases.
    - philliphaydon 4y ago
      Had to use a NATGW temporarly on AWS Lambda's to access the database and make remote http calls. But now it all works without the NATGW. Haven't had any other need for one.
      - ceejayoz 4y ago
        IIRC, you want one per AZ, or you'll get billed for extra cross-AZ traffic when instances in zone B's private subnet use the NAT gateway in zone A.
    - raffraffraff 4y ago
      You don't need NATGW at all if you use ipv6 egress-only gateway (which is free I believe)
blackoil 4y ago
Hire a better engineer/architect.
- fulafel 4y ago
  Indeed, this is often the next step. The new architect must have lots of AWS certifications. With the new architeture, the long term cost savings will be exponential, compared to the current projected cost curve! This can optionally combine with the Kubernetes scenario.
  - arinlen 4y ago
    > This can optionally combine with the Kubernetes scenario. Some people like to poke fun at the expense of kubernetes, but if they did any professional work, at the first failed deployment they would sell their firstborn to have something similar to kubernete's deployment rollback feature.
    - tpxl 4y ago
      We don't use kubes. Deployments sometimes fail and we deploy the old versions easily. No firstborns sold. I have used kubernetes blue/green deployments in an old job and it was beautiful. But to say kubernetes is that much better compared to rsyncing your executables to the server and restarting the service is plain wrong. It's a bit easier and a bit more declarative, sure, but manual rollbacks were a thing fifty years ago and still are.
  - russellendicott 4y ago
    Exactly, then you need at least 1 architect per app. I wrote an article about this problem: http://blog.bytester.net/posts/cloud-talent/ http://blog.bytester.net/posts/cloud-talent/
    - 90minuteAPI 4y ago
      This was a great read, thank you. I’m just now removing my application-focused blinders, but I already see the same technical, organizational, and financial issues.
throwaway787544 4y ago
Cloud Architecture: tech's version of a record label contract
moduspol 4y ago
This seems overwhelmingly pessimistic. VMs, serverless, and Kubernetes all panned without a suggestion of an alternative. The theme is knocking on AWS, so is your suggestion to go back to colos and self-hosting? Does that really sound better than infra-as-code?
- yakshaving_jgt 4y ago
  Self-hosting and infrastructure-as-code aren't mutually exclusive.
  - moduspol 4y ago
    This doesn't feel like a meaningful distinction. You can go from an empty cloud account to a fully-provisioned application stack in minutes with the right infra-as-code. Provisioning infrastructure at that level is mutually exclusive with self-hosting. But true, even with self-hosting, you can use infra-as-code at various levels (Ansible, OpenStack, self-hosted Kubernetes). It's just not really comparable to having your entire stack defined and provisioned that way.
    - yakshaving_jgt 4y ago
      Perhaps I am confused or we are talking about different things and consequently talking past each other, but I can literally open a new AWS/GCP/Hetzner/DO account, plug the credentials into my local configuration (in code), and then run a command with NixOps to provision an entire network of machines with custom specifications, and to automatically install all the software I need on those machines. Perhaps you aren't familiar with what NixOps and similar tools can give you?
      - moduspol 4y ago
        It sounds like we're using different definitions of the word "self-hosted." If you have your own on-premises lab or colo rental, you're not using AWS/GCP/Hetzner/DO, and you've got a lot of undifferentiated heavy lifting before your NixOps kick in (including maintenance going forward). If your point is that you can avoid a serverless architecture, still use cloud, and still use infra as code: of course you can. We've got to be disagreeing on what "self-hosted" means. OP criticized the cost and complexity of deploying EC2 instances and RDS databases across AZs, so presumably infra-as-code wouldn't help him here. OP didn't present an alternative solution, but reading between the lines is to not use cloud infrastructure (e.g. on-prem or colo).
von_lohengramm 4y ago
> And don't even get me started on how much work by this time has gone to build the infra-as-code to manage the rube golderg machine, you'll (seriously) have more lines of CDK code than app logic (that was slower to develop & harder to debug than your actual app code per line). Of all the AWS complaints, CDK is one of yours? I absolutely love CDK and its take on infra-as-code where you construct OO constructs imperatively in a sane language (please, please no more YAML-based DSLs...). I've found that CDK is the only one to have given us the code reusability that all of these solutions always promise while still being overtly hackable. Debugging CDK code? What kind of wacky stuff are you trying to do? I don't think I've ever had to debug CDK outside of typical "what IAM action am I missing?" or "how do I click some checkbox in CDK?". I'm curious what you would consider to be better alternatives to CDK.
- moduspol 4y ago
  CDK code also isn't executed at runtime. Even if it somehow is slower to develop and harder to debug (maybe if you're new to it), it's ideal that you have more of it than actual app code per line. That implies you have substantially less app code than you otherwise would, which (of course) is the entire purpose of these cloud-based abstractions. The amount of app code involved to stitch Kinesis, Lambda, and S3 together is minimal. The amount of app code to solve the same problems without them is orders of magnitude more plentiful and complex, which is why we avoid it.
  - tpxl 4y ago
    > The amount of app code to solve the same problems without them is orders of magnitude more plentiful and complex, which is why we avoid it. The amount of application code to write to a filesystem is... zero. The amount of application code required to setup a webserver in Java Spring is... zero. Not sure what Kinesis does, but the amount of application code required to connect to a database is zero (spring), to connect to a rabbitmq/kafka is zero (spring). And the best part is, it is less complex, mostly open source, and locally runnable compared to the Amazon rube goldberg machine.
    - 4y ago
rmbyrro 4y ago
This isn't the only way, though. Cloudfront can distribute static content and backends running on Lambda@Edge. DynamoDB supports global distribution (eventually consistent). If you don't have massive scale, they're quite cheaper than self managing (infra itself and ops people involved). Yes, these services require adaptation to your old habits of freely controlling your own servers. As for vendor lock-in, even on EC2 you'll end up with some. Difficult to argue how much, there's no reproducible studies on that. There are programming practices to minimize the costs of vendor switch as well.
- tommek4077 4y ago
  The original machine has a base-plate of prefabulated amulite, surmounted by a malleable logarithmic casing in such a way that the two spurving bearings were in a direct line with the pentametric fan. The latter consisted simply of six hydrocoptic marzelvanes, so fitted to the ambifacient lunar waneshaft that side fumbling was effectively prevented. The main winding was of the normal lotus-o-delta type placed in panendermic semiboloid slots in the stator, every seventh conductor being connected by a non-reversible tremie pipe to the differential girdlespring on the "up" end of the grammeters.
  - steveBK123 4y ago
    it's an older code sir, but it checks out
  - Frost1x 4y ago
    For anyone reading this for the first time, well worth the watch: https://m.youtube.com/watch?v=Ac7G7xOG2Ag https://m.youtube.com/watch?v=Ac7G7xOG2Ag
    - rmbyrro 4y ago
      Thanks for sharing, gave me context. But the parent comment and video were not worth the time at all. Maybe I'm failing to see the value? Happy to reconsider if you have more perspectives. It seems to me such video will only serve as fuel for cynic behavior. Which I wouldn't like to see more on HN.
kumarvvr 4y ago
At scale, perhaps they are costly. But i just cannot imagine deploying a webapp without cloud services. For the indie dev, cloud services have made side projects and experiments extremely cheap to spin up and scale.
- bigDinosaur 4y ago
  The idea of deploying an app to a machine sitting next to me, and serving the few requests per second just from that, is highly appealing. The only thing I'd always want is offsite backups which are best done to the cloud. You've inspired me!
steveBK123 4y ago
As an "onPrem" & "Big Data" guy I've been astonished at how fast the bill can run up. We are in the middle of a cloud adoption where the big proponent started with "its just money, devs time costs more.. do what you gotta do" to "hey so we need to consolidate these 3 DBs, and X is on vacation so lets spin this service down, and lets not worry about multi AZ/region yet" .. all before we even have a PROD launch, lol. In any case, was recently amazed at the cost of an RDS instance to host a meager 2B total records of vendor data. For the annual price I could buy one of the beefy servers I use to host 20% of my entire on-prem estate with 100s of vendor data sets. For my on-prem plant, 2B records is table stakes and we have datasets we receive that much data every day. Probably have something like 10T records of data across datasets if I had to speculate. Similarly on the ops/infra folks cost save, for every on-prem infra guy they think they can RIF, they've hired a cloud infra guy who for sure costs more. Likewise for the redundancy/backups/etc being "easy/free" in the cloud, we've already lost data in object store because of poorly execute combination of configuration & actions when try to do some permission changes/copy migration. It was completely unintentional and not noticed for weeks. No one actually executed a literal rm command at the time. Just because the object store advertises zero loss / versioning / etc, you really do still need to do backups. Clearly theres a lot of ephemeral & burst compute / variable cost stuff that totally belongs in the cloud. However for internal apps at a firm where the usage is largely scaled with staffing levels, the amount of fixed compute makes it hard to argue every app and every use case belongs in the cloud.
- Gravityloss 4y ago
  It's interesting to think about the process that lead to this: "its just money, devs time costs more.. do what you gotta do" Are cloud vendor salesmen doing jedi mind tricks? Or are these decisions just made by incompetent people? Who researches this kind of stuff? It's some kind of a management trends history subject.
  - shiftpgdn 4y ago
    Nobody got fired buying IBM^H^H^H Amazon.
brightball 4y ago
This is the playbook though. It's used by everybody. Microsoft, for example, has all of their certified developer programs. Company that use Microsoft development infrastructure like SQL Server end up getting huge discounts based on how many developers they have on staff with MS certs. This biases companies to hiring people with those certs, which reinforces the perception of value that developers have from getting the certs in the first place. Then when you have a ton of people on staff with MS certs you're NEVER going away from MS technology because at that point the company is so heavily invested in it. It's very much an evil genius situation. A few years back I worked for one of these companies and even though I wasn't doing much with the Microsoft stack in the company (they acquired a Rails company) I was still asked to go get an MS certification just to help with the discounts. I'm now a Microsoft certified HTML5/CSS3 developer, which after going through the IE6 years felt like the most ironic cert I could get. The lock in mentality is very real though. As part of an architecture committee at that company, we went through many months of analysis for some of the issues that were being faced at the company. The root cause was simply limitations of SQL Server combined with horizontal scale out limitations due to licensing costs. There was absolutely nothing that could be done though, because the company wouldn't move away from SQL Server for any reason. It was hard to watch.
- nojito 4y ago
  You'll be hard pressed to meet the business protections a MS contract gets your organization by using someone else.
  - brightball 4y ago
    What type of protections?
abofh 4y ago
You've got it backwards - which is to some extent why you have this problem: If you're fully utilizing your instances, you're working from the bottom of the pricing up, and all these things (3x redundancy, etc) are marginal costs of doing business because you're successful. What you're opting out of is the data-center workers, the sysadmins, the operators, and the expertise that comes with if it offered you value. Your hundreds of lines of CDK are people you're not employing to write them (it's your job I guess?), and applying the AWS systems and services to do that work for you - given you've chosen CDK, I have to assume you've bought into AWS anyhow, and someone in the chain sees the value extracted or savings approach. Where AWS burns you is when you don't know what you're doing - you have a PHP app, so you put in on an instance, you need it reliable so you have two, you need shared storage so you put it on EFS, and so on and so forth. If you're at the other end of the spectrum, and you want to buy the level of reliability and redundancy you get from three of the smallest instances in AWS spread in three AZ's, you're looking at a six figure investment. So yes, it's pricy if you use it like you use the desktop under you (8 hours a day, usually at 20% load), but you can also run millions of hits for pennies if you learn that most of what you're doing is overhead that gets built into, well, serverless.
voidr 4y ago
> You need to be triply redundant on 3 availability zones, (3x) both with the RDS db cluster and app containers (2x) . And then have separate dev/staging/prod envs (3x). That's 18x. Don't believe this has anything to do with the cloud, if you want the same thing with your own data center, it's going to be 18x there as well. > You can then get a pat on the head ("pass AWS well-architected review"). Or you can be pragmatic, the cloud will allow you to do whatever you want, nobody is stopping you. > Then they innovate "serverless" stuff Serverless does not replace everything, there are some things it's good for, but not everything, if you use it for something that it's not useful for then that's an issue with your judgement, not with the fact that it exists as an option. > And don't even get me started on how much work by this time has gone to build the infra-as-code to manage the rube golderg machine, you'll (seriously) have more lines of CDK code than app logic (that was slower to develop & harder to debug than your actual app code per line). I have thousands of lines of YAML and MD that I can give to anyone who is skilled enough to copy and paste and that person can 100% replicate the environment without me helping them once. If a component fails, I can just rebuild it very quickly and 100% accurately by just re-importing the Cloud Formation code, if I need an environment that is 100% same as the one that I already have, I just need to re-run a few scripts and I have it. What credible alternative can you propose that is better than infrastructure as code? > They've probably never costed / self administered a server in production use, or if they have, they know to keep quiet. I could make a similar snarky remark about sys-admins not understanding that the world is not static, however I would instead focus on the fact that acquisition of hardware is nowhere near the total cost of the infrastructure. According to a 3 second Google search, sys-admins in the US can cost around 75k-140k. Sys-admins can also slow down development by having to perform more manual actions and having to be more over protective, that adds additional costs, because you will still have to pay your idle developers while they are waiting for the sys-admin to action the ticket where they requested write access to their own home folder.
- fulafel 4y ago
  You could spend less than 18 x AWS cost, because AWS compute is so expensive, depending what hosting you chose. A self owned DC is isn't the best option for most. You might also skip the overkill (for most cases) AZ redundancy because it wasn't pushed on you. You might even go with a much more managed platform, like the Heroku like ones. Depends on what you build and for who. I'm not knocking IaC, but unnecessarily huge amounts of IaC to manage unnecessarily complex AWS service infra after you can't afford the monolith + db model and get roped to dynamodb, lambda@edge, API gateway, step functions, etc all requiring IAM roles, security groups, observability tooling, cicd & version control complexity, config services etc etc all the downsides you read about in microservice horror stories. And you can't even ssh in and strace or tcpdump the stuff like you could with your own microservices, they're black boxes. I also don't want separate sys admins, at least the bad kind you describe. But having your own servers, or knowing what they cost, doesn't mean you would.

philliphaydon 4y ago

fulafel 4y ago
If you're not testing az failovers you're probably just wasting money, like untested backups... But it's true that most people don't know this and aren't testing it because they don't know that az failovers don't automatically just work(tm). And the second downside of course would be asymmetry between the envs and divergence in the IaC etc resulting in more complexity and engineering work. (Of course you're probably wasting money anyway since you don't actually business-wise need better uptime than the single AZ, and your complexity induced human fumbles will cause much more outages anyway, but this has been a main selling point of the decision to go to AWS, so the requirement needs to be defended) Yes, you can build automation to have the redundant stuff up only sometimes, if you eat the engineering effort and complexity in your IAC and build automation... in the general vein of justifying engineering spend to offset AWS operating costs where running containers is very expensive! TLDR: either way you end up paying for the very high markup in compute prices, it'll just be easier to excuse jumping through expensive hoops to "save money" on it.
- philliphaydon 4y ago
  If you’re building your own infrastructure in a data center then sure you absolutely want to test your redundancy. But with AWS it’s a checkbox. It’s transparent to you and your applications. The infrastructure to host in multiple AZs is already in place. The only real issue with MultiAZ is the failover in RDS depending on the database you use could be seconds or 10s of seconds.
  - fulafel 4y ago
    IME not so in the real world, you'll have accidental state in your distributed system outside the DBs. You'll have some stuff that actually always runs in one AZ in normal circumstances and your integration partner in another org has whitelisted only its IP. Etc etc, everything that is untested will find ways to conspire to rot. Especially if you haven't learned by seeing these bugs so you can avoid them. Also you won't have a clear experience and understanding of what happens in the failover and you'll won't know to avoid failover breaking mistakes in your VPC configs, security groups, frontend-backend shared state etc. (And by "you" I mean "your dev team", it's not enough that one guy gets it). Also^2 if you read the news about all the outages it's very common for failover systems to fail generally, not just AWS - the general engineering wisdom is: always test your failovers. And there's no substitute for end-to-end testing it, instead of individually testing each layer/module. (Bad: "we can skip testing db failover", good: "let's test that the whole system works when there's a az failure")
    - russellendicott 4y ago
      Dealing with this now for a client. Can't test Redshift AZ relocation feature because there's no way to simulate AZ failure. Only safe bet is full multi region with DNS switcheroo.
  - moonchrome 4y ago
    >But with AWS it’s a checkbox. It’s transparent to you and your applications. The infrastructure to host in multiple AZs is already in place. The only real issue with MultiAZ is the failover in RDS depending on the database you use could be seconds or 10s of seconds. Have you actually seen this work on your project in practice ? Like a region go down and another region picks up automatically and it kept working just by switching a checkbox ?
arinlen 4y ago
> Why would you care about the redundancy on staging / dev? If you deploy to multiple regions then it wouldn't make no sense at all to have a single preprod stage, specially to run integration tests. Also, keep in mind that professional apps support localization, and localization has a significant afinity with regional deployments. I mean, if you support japanese and have a prod stage on Japan, would you think it was a good idea to run localization tests in a us deployment?
- philliphaydon 4y ago
  Currently have a deployment in Oregon and cloudfront servicing most of the world. With localisation. No need to deploy in Japan to support Japan. The only tricky one is China because of the firewall. Tho that hasn’t been an issue for the last few years as they haven’t been blocking cloudfront completely. (This has been running for 10 years in AWS without issue)
  - arinlen 4y ago
    > Currently have a deployment in Oregon and cloudfront servicing most of the world. With localisation. So you're serving static assets through CloudFront with a single backing service. Congrats, you managed to have a service that doesn't offer regional services. Also, you definitely don't support clients in China, or enjoy shooting yourself in the foot. Most professional applications with a global deployment and paying customers don't have the benefit of uploading HTML and calling it done.
    - philliphaydon 4y ago
      So 10+ years of supporting broadcasters and creative agencies around the world from 1 region is not supporting those countries. Got it. I mean there’s services in those regions for the tasks performed but even then it doesn’t require the level of testing you’re assuming it does.
      - sieabahlpark 4y ago
raffraffraff 4y ago
Even if you don't want it sometimes you're forced to run multiple AZs (eg: EKS requires 2x). But that 18x figure is nuts. VPCs, AZs, subnets, IAM etc are free. The cost comes from what you deploy into them. So separate environments don't have to be as expensive as production. You can scale them to zero, use smaller computer instances, run self-managed versions of expensive stuff (like DBs) or simply run small single-instances of RDS instead of large redundant clusters. Non-prod environments they're a great place to experiment with aggressive scale-down on cost while observing performance.
- cutthegrass2 4y ago
  Whilst the VPC itself is free, they get you on the NATGW - which you probably need along with the VPC in most cases.
  - philliphaydon 4y ago
    Had to use a NATGW temporarly on AWS Lambda's to access the database and make remote http calls. But now it all works without the NATGW. Haven't had any other need for one.
    - ceejayoz 4y ago
      IIRC, you want one per AZ, or you'll get billed for extra cross-AZ traffic when instances in zone B's private subnet use the NAT gateway in zone A.
  - raffraffraff 4y ago
    You don't need NATGW at all if you use ipv6 egress-only gateway (which is free I believe)

blackoil 4y ago

Hire a better engineer/architect.

fulafel 4y ago
Indeed, this is often the next step. The new architect must have lots of AWS certifications. With the new architeture, the long term cost savings will be exponential, compared to the current projected cost curve! This can optionally combine with the Kubernetes scenario.
- arinlen 4y ago
  > This can optionally combine with the Kubernetes scenario. Some people like to poke fun at the expense of kubernetes, but if they did any professional work, at the first failed deployment they would sell their firstborn to have something similar to kubernete's deployment rollback feature.
  - tpxl 4y ago
    We don't use kubes. Deployments sometimes fail and we deploy the old versions easily. No firstborns sold. I have used kubernetes blue/green deployments in an old job and it was beautiful. But to say kubernetes is that much better compared to rsyncing your executables to the server and restarting the service is plain wrong. It's a bit easier and a bit more declarative, sure, but manual rollbacks were a thing fifty years ago and still are.
- russellendicott 4y ago
  Exactly, then you need at least 1 architect per app. I wrote an article about this problem: http://blog.bytester.net/posts/cloud-talent/ http://blog.bytester.net/posts/cloud-talent/
  - 90minuteAPI 4y ago
    This was a great read, thank you. I’m just now removing my application-focused blinders, but I already see the same technical, organizational, and financial issues.

throwaway787544 4y ago

Cloud Architecture: tech's version of a record label contract

moduspol 4y ago

This seems overwhelmingly pessimistic. VMs, serverless, and Kubernetes all panned without a suggestion of an alternative. The theme is knocking on AWS, so is your suggestion to go back to colos and self-hosting? Does that really sound better than infra-as-code?

yakshaving_jgt 4y ago
Self-hosting and infrastructure-as-code aren't mutually exclusive.
- moduspol 4y ago
  This doesn't feel like a meaningful distinction. You can go from an empty cloud account to a fully-provisioned application stack in minutes with the right infra-as-code. Provisioning infrastructure at that level is mutually exclusive with self-hosting. But true, even with self-hosting, you can use infra-as-code at various levels (Ansible, OpenStack, self-hosted Kubernetes). It's just not really comparable to having your entire stack defined and provisioned that way.
  - yakshaving_jgt 4y ago
    Perhaps I am confused or we are talking about different things and consequently talking past each other, but I can literally open a new AWS/GCP/Hetzner/DO account, plug the credentials into my local configuration (in code), and then run a command with NixOps to provision an entire network of machines with custom specifications, and to automatically install all the software I need on those machines. Perhaps you aren't familiar with what NixOps and similar tools can give you?
    - moduspol 4y ago
      It sounds like we're using different definitions of the word "self-hosted." If you have your own on-premises lab or colo rental, you're not using AWS/GCP/Hetzner/DO, and you've got a lot of undifferentiated heavy lifting before your NixOps kick in (including maintenance going forward). If your point is that you can avoid a serverless architecture, still use cloud, and still use infra as code: of course you can. We've got to be disagreeing on what "self-hosted" means. OP criticized the cost and complexity of deploying EC2 instances and RDS databases across AZs, so presumably infra-as-code wouldn't help him here. OP didn't present an alternative solution, but reading between the lines is to not use cloud infrastructure (e.g. on-prem or colo).

von_lohengramm 4y ago

> And don't even get me started on how much work by this time has gone to build the infra-as-code to manage the rube golderg machine, you'll (seriously) have more lines of CDK code than app logic (that was slower to develop & harder to debug than your actual app code per line). Of all the AWS complaints, CDK is one of yours? I absolutely love CDK and its take on infra-as-code where you construct OO constructs imperatively in a sane language (please, please no more YAML-based DSLs...). I've found that CDK is the only one to have given us the code reusability that all of these solutions always promise while still being overtly hackable. Debugging CDK code? What kind of wacky stuff are you trying to do? I don't think I've ever had to debug CDK outside of typical "what IAM action am I missing?" or "how do I click some checkbox in CDK?". I'm curious what you would consider to be better alternatives to CDK.

moduspol 4y ago
CDK code also isn't executed at runtime. Even if it somehow is slower to develop and harder to debug (maybe if you're new to it), it's ideal that you have more of it than actual app code per line. That implies you have substantially less app code than you otherwise would, which (of course) is the entire purpose of these cloud-based abstractions. The amount of app code involved to stitch Kinesis, Lambda, and S3 together is minimal. The amount of app code to solve the same problems without them is orders of magnitude more plentiful and complex, which is why we avoid it.
- tpxl 4y ago
  > The amount of app code to solve the same problems without them is orders of magnitude more plentiful and complex, which is why we avoid it. The amount of application code to write to a filesystem is... zero. The amount of application code required to setup a webserver in Java Spring is... zero. Not sure what Kinesis does, but the amount of application code required to connect to a database is zero (spring), to connect to a rabbitmq/kafka is zero (spring). And the best part is, it is less complex, mostly open source, and locally runnable compared to the Amazon rube goldberg machine.
  - 4y ago

rmbyrro 4y ago

This isn't the only way, though. Cloudfront can distribute static content and backends running on Lambda@Edge. DynamoDB supports global distribution (eventually consistent). If you don't have massive scale, they're quite cheaper than self managing (infra itself and ops people involved). Yes, these services require adaptation to your old habits of freely controlling your own servers. As for vendor lock-in, even on EC2 you'll end up with some. Difficult to argue how much, there's no reproducible studies on that. There are programming practices to minimize the costs of vendor switch as well.

tommek4077 4y ago
The original machine has a base-plate of prefabulated amulite, surmounted by a malleable logarithmic casing in such a way that the two spurving bearings were in a direct line with the pentametric fan. The latter consisted simply of six hydrocoptic marzelvanes, so fitted to the ambifacient lunar waneshaft that side fumbling was effectively prevented. The main winding was of the normal lotus-o-delta type placed in panendermic semiboloid slots in the stator, every seventh conductor being connected by a non-reversible tremie pipe to the differential girdlespring on the "up" end of the grammeters.
- steveBK123 4y ago
  it's an older code sir, but it checks out
- Frost1x 4y ago
  For anyone reading this for the first time, well worth the watch: https://m.youtube.com/watch?v=Ac7G7xOG2Ag https://m.youtube.com/watch?v=Ac7G7xOG2Ag
  - rmbyrro 4y ago
    Thanks for sharing, gave me context. But the parent comment and video were not worth the time at all. Maybe I'm failing to see the value? Happy to reconsider if you have more perspectives. It seems to me such video will only serve as fuel for cynic behavior. Which I wouldn't like to see more on HN.

kumarvvr 4y ago

At scale, perhaps they are costly. But i just cannot imagine deploying a webapp without cloud services. For the indie dev, cloud services have made side projects and experiments extremely cheap to spin up and scale.

bigDinosaur 4y ago
The idea of deploying an app to a machine sitting next to me, and serving the few requests per second just from that, is highly appealing. The only thing I'd always want is offsite backups which are best done to the cloud. You've inspired me!

steveBK123 4y ago

As an "onPrem" & "Big Data" guy I've been astonished at how fast the bill can run up. We are in the middle of a cloud adoption where the big proponent started with "its just money, devs time costs more.. do what you gotta do" to "hey so we need to consolidate these 3 DBs, and X is on vacation so lets spin this service down, and lets not worry about multi AZ/region yet" .. all before we even have a PROD launch, lol. In any case, was recently amazed at the cost of an RDS instance to host a meager 2B total records of vendor data. For the annual price I could buy one of the beefy servers I use to host 20% of my entire on-prem estate with 100s of vendor data sets. For my on-prem plant, 2B records is table stakes and we have datasets we receive that much data every day. Probably have something like 10T records of data across datasets if I had to speculate. Similarly on the ops/infra folks cost save, for every on-prem infra guy they think they can RIF, they've hired a cloud infra guy who for sure costs more. Likewise for the redundancy/backups/etc being "easy/free" in the cloud, we've already lost data in object store because of poorly execute combination of configuration & actions when try to do some permission changes/copy migration. It was completely unintentional and not noticed for weeks. No one actually executed a literal rm command at the time. Just because the object store advertises zero loss / versioning / etc, you really do still need to do backups. Clearly theres a lot of ephemeral & burst compute / variable cost stuff that totally belongs in the cloud. However for internal apps at a firm where the usage is largely scaled with staffing levels, the amount of fixed compute makes it hard to argue every app and every use case belongs in the cloud.

Gravityloss 4y ago
It's interesting to think about the process that lead to this: "its just money, devs time costs more.. do what you gotta do" Are cloud vendor salesmen doing jedi mind tricks? Or are these decisions just made by incompetent people? Who researches this kind of stuff? It's some kind of a management trends history subject.
- shiftpgdn 4y ago
  Nobody got fired buying IBM^H^H^H Amazon.

brightball 4y ago

This is the playbook though. It's used by everybody. Microsoft, for example, has all of their certified developer programs. Company that use Microsoft development infrastructure like SQL Server end up getting huge discounts based on how many developers they have on staff with MS certs. This biases companies to hiring people with those certs, which reinforces the perception of value that developers have from getting the certs in the first place. Then when you have a ton of people on staff with MS certs you're NEVER going away from MS technology because at that point the company is so heavily invested in it. It's very much an evil genius situation. A few years back I worked for one of these companies and even though I wasn't doing much with the Microsoft stack in the company (they acquired a Rails company) I was still asked to go get an MS certification just to help with the discounts. I'm now a Microsoft certified HTML5/CSS3 developer, which after going through the IE6 years felt like the most ironic cert I could get. The lock in mentality is very real though. As part of an architecture committee at that company, we went through many months of analysis for some of the issues that were being faced at the company. The root cause was simply limitations of SQL Server combined with horizontal scale out limitations due to licensing costs. There was absolutely nothing that could be done though, because the company wouldn't move away from SQL Server for any reason. It was hard to watch.

nojito 4y ago
You'll be hard pressed to meet the business protections a MS contract gets your organization by using someone else.
- brightball 4y ago
  What type of protections?

abofh 4y ago

You've got it backwards - which is to some extent why you have this problem: If you're fully utilizing your instances, you're working from the bottom of the pricing up, and all these things (3x redundancy, etc) are marginal costs of doing business because you're successful. What you're opting out of is the data-center workers, the sysadmins, the operators, and the expertise that comes with if it offered you value. Your hundreds of lines of CDK are people you're not employing to write them (it's your job I guess?), and applying the AWS systems and services to do that work for you - given you've chosen CDK, I have to assume you've bought into AWS anyhow, and someone in the chain sees the value extracted or savings approach. Where AWS burns you is when you don't know what you're doing - you have a PHP app, so you put in on an instance, you need it reliable so you have two, you need shared storage so you put it on EFS, and so on and so forth. If you're at the other end of the spectrum, and you want to buy the level of reliability and redundancy you get from three of the smallest instances in AWS spread in three AZ's, you're looking at a six figure investment. So yes, it's pricy if you use it like you use the desktop under you (8 hours a day, usually at 20% load), but you can also run millions of hits for pennies if you learn that most of what you're doing is overhead that gets built into, well, serverless.

voidr 4y ago

> You need to be triply redundant on 3 availability zones, (3x) both with the RDS db cluster and app containers (2x) . And then have separate dev/staging/prod envs (3x). That's 18x. Don't believe this has anything to do with the cloud, if you want the same thing with your own data center, it's going to be 18x there as well. > You can then get a pat on the head ("pass AWS well-architected review"). Or you can be pragmatic, the cloud will allow you to do whatever you want, nobody is stopping you. > Then they innovate "serverless" stuff Serverless does not replace everything, there are some things it's good for, but not everything, if you use it for something that it's not useful for then that's an issue with your judgement, not with the fact that it exists as an option. > And don't even get me started on how much work by this time has gone to build the infra-as-code to manage the rube golderg machine, you'll (seriously) have more lines of CDK code than app logic (that was slower to develop & harder to debug than your actual app code per line). I have thousands of lines of YAML and MD that I can give to anyone who is skilled enough to copy and paste and that person can 100% replicate the environment without me helping them once. If a component fails, I can just rebuild it very quickly and 100% accurately by just re-importing the Cloud Formation code, if I need an environment that is 100% same as the one that I already have, I just need to re-run a few scripts and I have it. What credible alternative can you propose that is better than infrastructure as code? > They've probably never costed / self administered a server in production use, or if they have, they know to keep quiet. I could make a similar snarky remark about sys-admins not understanding that the world is not static, however I would instead focus on the fact that acquisition of hardware is nowhere near the total cost of the infrastructure. According to a 3 second Google search, sys-admins in the US can cost around 75k-140k. Sys-admins can also slow down development by having to perform more manual actions and having to be more over protective, that adds additional costs, because you will still have to pay your idle developers while they are waiting for the sys-admin to action the ticket where they requested write access to their own home folder.

fulafel 4y ago
You could spend less than 18 x AWS cost, because AWS compute is so expensive, depending what hosting you chose. A self owned DC is isn't the best option for most. You might also skip the overkill (for most cases) AZ redundancy because it wasn't pushed on you. You might even go with a much more managed platform, like the Heroku like ones. Depends on what you build and for who. I'm not knocking IaC, but unnecessarily huge amounts of IaC to manage unnecessarily complex AWS service infra after you can't afford the monolith + db model and get roped to dynamodb, lambda@edge, API gateway, step functions, etc all requiring IAM roles, security groups, observability tooling, cicd & version control complexity, config services etc etc all the downsides you read about in microservice horror stories. And you can't even ssh in and strace or tcpdump the stuff like you could with your own microservices, they're black boxes. I also don't want separate sys admins, at least the bad kind you describe. But having your own servers, or knowing what they cost, doesn't mean you would.