5 ms·

I'm one of those people, and I don't agree. Before I drop 5 figures on a single server, I'd like to have some confidence in the performance numbers I'm likely

I'm one of those people, and I don't agree. Before I drop 5 figures on a single server, I'd like to have some confidence in the performance numbers I'm likely to see. I'd expect folk who are experienced with on-prem have a good intuition about this - after a decade of cloud-only work, I don't. Also, cloud networking offers a bunch of really nice primitives which I'm not clear how I'd replicate on-prem. I've estimated our IT workload would roughly double if we were to add physically racking machines, replacing failed disks, monitoring backups/SMART errors etc. That's... not cheap in staff time. Moving things on-prem starts making financial sense around the point your cloud bills hit the cost of one engineers salary.

esseph 7mo ago
> Also, cloud networking offers a bunch of really nice primitives which I'm not clear how I'd replicate on-prem. Like what?
- danielheath 7mo ago
  The biggest one for me is the way AWS security groups & IAM work. In AWS, it's straightforward to say e.g. "permit traffic on port X from instances holding IAM role Y". You can easily e.g. get the firewall rules for all your ec2 instances in a structured format. I really would not look forward to building something even 1/10th as functional as that.
  - tempaccount5050 7mo ago
    And you think just anyone can set that up? No sys admin/infra guy needed? Seems pretty risky.
    - vel0city 7mo ago
      I mean not just anyone, but its far less complicated than dealing with arcane iptables commands. And yet far more powerful, being able to just say "instances like this can talk to instances like this in these particular ways, reject everything else". Don't need subnet rules or whatever, its all about identity of the actual things. Meanwhile lots of enterprise firewalls barely even have a concept of "zones". Its practically not even close to comparing for most deployments. Maybe with extremely fancy firewall stacks with $ $MAX_INT service contracts one can do something similar. But I guess with on-prem stuff things are often less ephemeral, so there's slightly less need.
      - esseph 7mo ago
        > I guess with on-prem stuff things are often less ephemeral, so there's slightly less need Kubernetes is running on bare metal quite a lot of places.
      - ahartmetz 7mo ago
        I could type your arcane iptables commands for a couple hundred an hour. That stuff is easy compared to some software development tasks. I have sometimes struggled, but I've always found a solution after a few hours max.
  - esseph 7mo ago
    I would probably just build the infra in crossplane which standardizes a lot of features across the board and gives developers a set of APIs to use / dashboard against. Different deployments and orgs have different needs and desire different features though.
- SamuelAdams 7mo ago
  IAM comes to mind, with fine grained control over everything. S3 has excellent legal and auditory settings for data, as well as automatic data retention policies. KMS is a very secure and well done service. I dare you to find an equivalent on-prem solution that offers as much security. And then there's the whole DR idea. Failing over to another AWS region is largely trivial if you set it up correctly - on prem is typically custom to each organization, so you need to train new staff with your organizations workflows. Whereas in AWS, Route53 fail-over routing (for example) is the same across every organization. This reduces cost in training and hiring.
  - esseph 7mo ago
    I've worked at many enterprises that have done and do these very things. Some for fixed workloads at scale, some for data creation/use locality issues, some for performance. I think there is about a 15 year knowledge gap in on-prem competence and what the newest shiniest is on prem for some people. Yes, some of the vendors and gear are VERY bad, but not all, and there's always eBPF :)
- __turbobrew__ 7mo ago
  BGP based routing is a major pain in the ass to do on-prem. If you want true HA in the datacenter you are going to need to utilize BGP.
  - esseph 7mo ago
    I mean, BGP EVPN is the datacenter standard. (Linux infra / k8s / networking guy)
    - __turbobrew__ 7mo ago
      There are standards but actually designing a sane network architecture, buying all of the correct network hardware, and configuring all of the software to properly use that hardware is hard. At my company we have a team of about 20 people whose job it is to just design, install, and run the network.
      - esseph 7mo ago
        > There are standards but actually designing a sane network architecture, buying all of the correct network hardware, and configuring all of the software to properly use that hardware is hard. At my company we have a team of about 20 people whose job it is to just design, install, and run the network. Network engineers do network engineering :)
ragall 7mo ago
> I've estimated our IT workload would roughly double if we were to add physically racking machines, replacing failed disks, monitoring backups/SMART errors etc. That's why nowadays one would use a managed collocation service, not hosting a rack in the office basement.

esseph 7mo ago

> Also, cloud networking offers a bunch of really nice primitives which I'm not clear how I'd replicate on-prem. Like what?

danielheath 7mo ago
The biggest one for me is the way AWS security groups & IAM work. In AWS, it's straightforward to say e.g. "permit traffic on port X from instances holding IAM role Y". You can easily e.g. get the firewall rules for all your ec2 instances in a structured format. I really would not look forward to building something even 1/10th as functional as that.
- tempaccount5050 7mo ago
  And you think just anyone can set that up? No sys admin/infra guy needed? Seems pretty risky.
  - vel0city 7mo ago
    I mean not just anyone, but its far less complicated than dealing with arcane iptables commands. And yet far more powerful, being able to just say "instances like this can talk to instances like this in these particular ways, reject everything else". Don't need subnet rules or whatever, its all about identity of the actual things. Meanwhile lots of enterprise firewalls barely even have a concept of "zones". Its practically not even close to comparing for most deployments. Maybe with extremely fancy firewall stacks with $ $MAX_INT service contracts one can do something similar. But I guess with on-prem stuff things are often less ephemeral, so there's slightly less need.
    - esseph 7mo ago
      > I guess with on-prem stuff things are often less ephemeral, so there's slightly less need Kubernetes is running on bare metal quite a lot of places.
    - ahartmetz 7mo ago
      I could type your arcane iptables commands for a couple hundred an hour. That stuff is easy compared to some software development tasks. I have sometimes struggled, but I've always found a solution after a few hours max.
- esseph 7mo ago
  I would probably just build the infra in crossplane which standardizes a lot of features across the board and gives developers a set of APIs to use / dashboard against. Different deployments and orgs have different needs and desire different features though.
SamuelAdams 7mo ago
IAM comes to mind, with fine grained control over everything. S3 has excellent legal and auditory settings for data, as well as automatic data retention policies. KMS is a very secure and well done service. I dare you to find an equivalent on-prem solution that offers as much security. And then there's the whole DR idea. Failing over to another AWS region is largely trivial if you set it up correctly - on prem is typically custom to each organization, so you need to train new staff with your organizations workflows. Whereas in AWS, Route53 fail-over routing (for example) is the same across every organization. This reduces cost in training and hiring.
- esseph 7mo ago
  I've worked at many enterprises that have done and do these very things. Some for fixed workloads at scale, some for data creation/use locality issues, some for performance. I think there is about a 15 year knowledge gap in on-prem competence and what the newest shiniest is on prem for some people. Yes, some of the vendors and gear are VERY bad, but not all, and there's always eBPF :)
__turbobrew__ 7mo ago
BGP based routing is a major pain in the ass to do on-prem. If you want true HA in the datacenter you are going to need to utilize BGP.
- esseph 7mo ago
  I mean, BGP EVPN is the datacenter standard. (Linux infra / k8s / networking guy)
  - __turbobrew__ 7mo ago
    There are standards but actually designing a sane network architecture, buying all of the correct network hardware, and configuring all of the software to properly use that hardware is hard. At my company we have a team of about 20 people whose job it is to just design, install, and run the network.
    - esseph 7mo ago
      > There are standards but actually designing a sane network architecture, buying all of the correct network hardware, and configuring all of the software to properly use that hardware is hard. At my company we have a team of about 20 people whose job it is to just design, install, and run the network. Network engineers do network engineering :)

ragall 7mo ago

> I've estimated our IT workload would roughly double if we were to add physically racking machines, replacing failed disks, monitoring backups/SMART errors etc. That's why nowadays one would use a managed collocation service, not hosting a rack in the office basement.