4 ms·

While terraform is not ideal it is much much more easy to deal with managed services in AWS than to deal with on premises baremetal servers. Most are biased be

While terraform is not ideal it is much much more easy to deal with managed services in AWS than to deal with on premises baremetal servers. Most are biased because they like dealing with the kind of issues in on premises. They like dealing with the performance regressions, heat maps, kernel issues etc. Because why not? You are a developer and you need some way to exercise your skills. AWS takes that away and makes you focus on the product. Issues arising from AWS only requires you talking to support. Most developers get into this industry for the love of solving these problems and not actually solving product requirements. AWS takes away what devs like and brings in more "actual" work.

ndriscoll 1y ago
> AWS takes that away and makes you focus on the product. Issues arising from AWS only requires you talking to support. Not my experience at all. e.g. NLBs don't support ICMP which has broken some clients of the application I work on. When we tried to turn on preserve-client-ip so we could get past the ephemeral port limit, it started causing issues with MSS negotiation, breaking some small fraction of clients. This stuff is insanely hard to debug because you can't get onto the loadbalancer to do packet captures (nor can AWS support). Loadbalancing for long-lived connections works poorly. Lambda runs into performance issues immediately for a web application server because it's just an entirely broken architecture for that use-case (it's basically the exact opposite of user-mode threads to scale: let's use an entire VM per request!). For some reason they encourage people to do it anyway. Lord help you if you have someone with some political capital in your org that wants to push for that. RDS also runs into performance issues the moment you actually have some traffic. A baremetal server is orders of magnitude more capable. ipv6 support is still randomly full of gaps (or has only very recently been fixed, except you might have to do things like recreate your production EKS cluster, oops) which leads to random problems that you have to architect around. Taken with NAT gateway being absurdly expensive, you end up having to invert sensible architectures or go through extra proxy layers that just complicate things. AWS takes basic skills around how to build/maintain backend systems and makes half of your knowledge useless/impossible to apply, instead upgrading all of your simple tuning tasks into architectural design problems. The summary of my last few years has basically been working around problems that almost entirely originate from trying to move software into EKS and dealing with random constraints that would take minutes to fix baremetal.
- icedchai 1y ago
  I agree that building your backend on Lambda is terrible for many reasons: slow starts, request / response size restrictions, limitations in "layer" sizes, etc. RDS, however, I have found to be rock solid. What have you run into?
  - mystifyingpoi 1y ago
    The parent compares RDS to baremetal, which I think isn't a fair comparison at all. Especially since we don't know the specs of either of these. I found RDS to be rock solid too, although performance issues are often resolved by developers by submitting a PR that bumps the instance size x2, because "why not". On baremetal it's often impossible to upgrade CPU just like that, so people have to fix performance issues elsewhere, which leads to better outcome at the end.
    - vidarh 1y ago
      RDS works great, but it's far easier to scale a bare metal setup to an extent that makes RDS look like an expensive toy because you have far more hardware options RDS is a good option if you want convenience and simplicity, though.
      - DanielHB 1y ago
        Managing database backups myself is something that gives me nightmares. I would refuse to use bare-metal dbs unless I have a dedicated team just to manage the database (or data that is okay to lose, like caching layers).
        vidarh 1y ago
        Managing database backups is fairly straightforward. Postgres + a base backup + long term wal archiving in a blob store is very easy to set up and monitor. It could be easier, and if you don't want to manage that using RDS is certainly a valid choice, but it's a tradeoff - I often have customers that help addressing performance issues with RDS they simply wouldn't have if they sized a bare metal setup with enough RAM and NVMe and configured it even halfway decently instead, and the end result is often that they end up paying more for devops help to figure out performance bottlenecks than they'd spend putting the same devops consultant on retainer ensuring they have a solid backup setup.
vanviegen 1y ago
Uh, no I don't in fact like dealing with any of that. And I've never had to in 20 years of managing some mid scale service on bare metal. (Though of course there have been issues of various sorts.) I think you may have it backwards: people like tinkering with complex cloud stuff, even if they don't need it.
bcrosby95 1y ago
We have both AWS and colocated servers. The server workload mostly scales with the machine count not the user count. And you can get a lot done with very few servers these days.
theamk 1y ago
> performance regressions, heat maps, kernel issues etc. > AWS takes that away and makes you focus on the product. ha ha ha no. Have been dealing with kernel issues on my AWS machines for a long time. They lock up under certain kinds of high load. AWS support is useless. Experimenting with kernel version leads to performance regressions. AWS is great if your IT/purchasing department is inefficient. Getting a new AWS machine is instant, compared to getting purchasing to approve new machine and IT allocating it to you. But all the low-level stuff is still there.
- geoduck14 1y ago
  >AWS is great if your IT/purchasing department is inefficient Fwiw, I think a lot of companies have this problem.
  - san1t1 1y ago
    I think the conversation has turned from "Can we spend more?" to "Can you please try and spend less?"
    - theamk 1y ago
      "Can you please try and spend less?" "Sure, we can get some on-prem machines. They'll pay for themselves in 6 months. I just need permissions from Finance to spend some CAPEX, and get IT and Facilities to cooperate" "Ugh, actually please keep using AWS. But try and spend less.. if you can and this does not compromise deadlines"
  - DanielHB 1y ago
    My wife is literally consulting for a big bank where it takes them 6+ months to get an on-premise VM setup and configured.
vidarh 1y ago
That was barely true a decade ago. It's total nonsense today when it's trivial to ensure all your servers have IPMI or similar and it cann all be automated apart from the couple-of-times a year component swap outs. But it's also the wrong comparison: there's rarely a reason to go on premises, and need to take responsibility for the hardware yourself - renting bare metal servers is usually the sweet spot and means someone else does the annoying bits for you but you still have the simplicity and lower cost. As someone contracted to manage systems for people, I consistently make more money from people who overengineer their cloud setups than from people with bare metal servers. It tends to require far more maintenance to keep an AWS setup same, secure, and not bankrupting you.
sgarland 1y ago
I have literally never met a dev who wanted to deal with any kind of infrastructure.
DanielHB 1y ago
> While terraform is not ideal it is much much more easy to deal with managed services in AWS than to deal with on premises baremetal servers. If you have one or two baremetal servers it is not, but yes once you start having a lot of infra it is way better. But you can get really, really far with one or two baremetal servers and some SaaS solutions...

ndriscoll 1y ago

> AWS takes that away and makes you focus on the product. Issues arising from AWS only requires you talking to support. Not my experience at all. e.g. NLBs don't support ICMP which has broken some clients of the application I work on. When we tried to turn on preserve-client-ip so we could get past the ephemeral port limit, it started causing issues with MSS negotiation, breaking some small fraction of clients. This stuff is insanely hard to debug because you can't get onto the loadbalancer to do packet captures (nor can AWS support). Loadbalancing for long-lived connections works poorly. Lambda runs into performance issues immediately for a web application server because it's just an entirely broken architecture for that use-case (it's basically the exact opposite of user-mode threads to scale: let's use an entire VM per request!). For some reason they encourage people to do it anyway. Lord help you if you have someone with some political capital in your org that wants to push for that. RDS also runs into performance issues the moment you actually have some traffic. A baremetal server is orders of magnitude more capable. ipv6 support is still randomly full of gaps (or has only very recently been fixed, except you might have to do things like recreate your production EKS cluster, oops) which leads to random problems that you have to architect around. Taken with NAT gateway being absurdly expensive, you end up having to invert sensible architectures or go through extra proxy layers that just complicate things. AWS takes basic skills around how to build/maintain backend systems and makes half of your knowledge useless/impossible to apply, instead upgrading all of your simple tuning tasks into architectural design problems. The summary of my last few years has basically been working around problems that almost entirely originate from trying to move software into EKS and dealing with random constraints that would take minutes to fix baremetal.

icedchai 1y ago
I agree that building your backend on Lambda is terrible for many reasons: slow starts, request / response size restrictions, limitations in "layer" sizes, etc. RDS, however, I have found to be rock solid. What have you run into?
- mystifyingpoi 1y ago
  The parent compares RDS to baremetal, which I think isn't a fair comparison at all. Especially since we don't know the specs of either of these. I found RDS to be rock solid too, although performance issues are often resolved by developers by submitting a PR that bumps the instance size x2, because "why not". On baremetal it's often impossible to upgrade CPU just like that, so people have to fix performance issues elsewhere, which leads to better outcome at the end.
  - vidarh 1y ago
    RDS works great, but it's far easier to scale a bare metal setup to an extent that makes RDS look like an expensive toy because you have far more hardware options RDS is a good option if you want convenience and simplicity, though.
    - DanielHB 1y ago
      Managing database backups myself is something that gives me nightmares. I would refuse to use bare-metal dbs unless I have a dedicated team just to manage the database (or data that is okay to lose, like caching layers).
      - vidarh 1y ago
        Managing database backups is fairly straightforward. Postgres + a base backup + long term wal archiving in a blob store is very easy to set up and monitor. It could be easier, and if you don't want to manage that using RDS is certainly a valid choice, but it's a tradeoff - I often have customers that help addressing performance issues with RDS they simply wouldn't have if they sized a bare metal setup with enough RAM and NVMe and configured it even halfway decently instead, and the end result is often that they end up paying more for devops help to figure out performance bottlenecks than they'd spend putting the same devops consultant on retainer ensuring they have a solid backup setup.

vanviegen 1y ago

Uh, no I don't in fact like dealing with any of that. And I've never had to in 20 years of managing some mid scale service on bare metal. (Though of course there have been issues of various sorts.) I think you may have it backwards: people like tinkering with complex cloud stuff, even if they don't need it.

bcrosby95 1y ago

We have both AWS and colocated servers. The server workload mostly scales with the machine count not the user count. And you can get a lot done with very few servers these days.

theamk 1y ago

> performance regressions, heat maps, kernel issues etc. > AWS takes that away and makes you focus on the product. ha ha ha no. Have been dealing with kernel issues on my AWS machines for a long time. They lock up under certain kinds of high load. AWS support is useless. Experimenting with kernel version leads to performance regressions. AWS is great if your IT/purchasing department is inefficient. Getting a new AWS machine is instant, compared to getting purchasing to approve new machine and IT allocating it to you. But all the low-level stuff is still there.

geoduck14 1y ago
>AWS is great if your IT/purchasing department is inefficient Fwiw, I think a lot of companies have this problem.
- san1t1 1y ago
  I think the conversation has turned from "Can we spend more?" to "Can you please try and spend less?"
  - theamk 1y ago
    "Can you please try and spend less?" "Sure, we can get some on-prem machines. They'll pay for themselves in 6 months. I just need permissions from Finance to spend some CAPEX, and get IT and Facilities to cooperate" "Ugh, actually please keep using AWS. But try and spend less.. if you can and this does not compromise deadlines"
- DanielHB 1y ago
  My wife is literally consulting for a big bank where it takes them 6+ months to get an on-premise VM setup and configured.

vidarh 1y ago

That was barely true a decade ago. It's total nonsense today when it's trivial to ensure all your servers have IPMI or similar and it cann all be automated apart from the couple-of-times a year component swap outs. But it's also the wrong comparison: there's rarely a reason to go on premises, and need to take responsibility for the hardware yourself - renting bare metal servers is usually the sweet spot and means someone else does the annoying bits for you but you still have the simplicity and lower cost. As someone contracted to manage systems for people, I consistently make more money from people who overengineer their cloud setups than from people with bare metal servers. It tends to require far more maintenance to keep an AWS setup same, secure, and not bankrupting you.

sgarland 1y ago

I have literally never met a dev who wanted to deal with any kind of infrastructure.

DanielHB 1y ago

> While terraform is not ideal it is much much more easy to deal with managed services in AWS than to deal with on premises baremetal servers. If you have one or two baremetal servers it is not, but yes once you start having a lot of infra it is way better. But you can get really, really far with one or two baremetal servers and some SaaS solutions...