7 ms·

As someone who split his time 50/50 between AWS and vmware based instances, i cant really think of anything we run in AWS that is not eventually supposed to go

As someone who split his time 50/50 between AWS and vmware based instances, i cant really think of anything we run in AWS that is not eventually supposed to go back on prem when we have stabilized the specification and lifecycle expectations. And who could not have gone in on prem if we had been willing to spend capex upfront(which would likely had led to lower lifecycle costs). The last time this debate came along ms was able to sweep in with an solution that did not require cloud integration with an price/support package that was by all estimate at bellow cost, and i would be surprised if the state in question have anything in the cloud that MS did not use to offer as on-prem solutions, and im pretty sure the current wintel/mso solution is based around those on-prem implementation of AD and SharePoint that no longer exists and viable alternatives to opensource based document storage repositories.

nine_k 3y ago
> i cant really think of anything we run in AWS that is not eventually supposed to go back on prem Hod do you handle multi-DC, multi-AZ resilience? What are you using instead of IAM policies that cover every resource? (Asking unironically, would just like to know. I worked at cloud-centric companies for last 15 years or so.)
- Stranger43 3y ago
  For AWS we rely almost entirely on backup/restore and database level log shipping on prem this is is in some cases argumented by storage level(netapp and previously hpe 3par) replication but this is going away for cost reasons. Given that my experience with AWS is to use the same application level cross region resilience techniques im used to on prem(i have worked with high end unix boxen most of my carear) im genuinely baffled when people start talking about cloud resilience as something magical, and nearly all our traffic happens inside of an private network(MPLS/VPN) that stretches across the different sites. I really haven't seen any magic multi-az resiliiance in aws that dont have an onprem counter part. None of this is in house whitebox hardware but relatively standard solutions from established vendors(VMware recently started abusing their near monopoly so everyone is looking for/at alternatives like proxmox, xen and nutanix but arent ready to move just yet).
  - nine_k 3y ago
    Thanks! Certainly multi-AZ setups are not magical. I just wanted to know how it's now done on-prem, what are the tools, solutions, approaches. Doing this in a cost-effective way for 1000 servers likely takes different tools than doing this at a major cloud provider scale. One thing that's more "magical" in AWS is S3. Their architecture is impressive, see e.g. https://www.allthingsdistributed.com/2021/04/s3-strong-consistency.html https://www.allthingsdistributed.com/2021/04/s3-strong-consi...
- kaliszad 3y ago
  I guess the same as people have always done. You have more server rooms on ideally different networks and grids. That can be a rented room/ rack in a bigger DC, your own if you have a big factory with subsidiaries etc. The IAM etc. will probably done by a combination of technical on organizational measures. You will have certain people doing certain things at least before the solution is ready for IaC. People will create roles, accounts, accesses and such. With networking gear that can be still tricky to implement, with virtualization solutions that is easier today. For databases etc. you can create accounts in them too. Of course, K8s and similar make these things more formalized/ transferable. However there is a lot of stuff before you can deploy that. People forget however that even if you have hundreds of servers you are tiny compared to the cloud providers. You don't have to have the same breadth and depth of offering. So while you need more baby sitting of hardware, probably will not get nearly as good deals on hardware as the big providers do, you will save their considerable margins. Also, they actually have some of the same expenses too - if a harddrive goes bad they will still swap them basically the same as you do. Big cloud providers will not get substantially different energy pricing than what e.g. a steel foundry would get. By hosting things on premise or in a nearby datacenter(s) you can shave off a lot of latency too. Some machinery likes to store a lot of data and shaving off latency will decrease your need for thick router buffers because you will not have such a big Bandwidth Delay Product and will achieve the same speeds with much smaller buffers. Building stuff on premise just for you makes some things easier too. Even if you loose some credentials usually you can just hard-reset the equipment as the last resort. There will be no credit card blocking that would affect the operations. If you are less strict with security it will usually matter much less - you are not sharing the hardware with unknown parties and all people that touch it have a contract with the company. So usually everybody want the company to succeed to get the paycheck. You build a deeper know how and can do some optimizations the cloud providers cannot do because you don't have to be general.
  - nine_k 3y ago
    Thanks, this makes sense! I mostly thought about the software infrastructure side. With thousands and even mere hundreds of servers over several locations you already want some uniformity. Would you run k8s? Nomad + Consul? MinIO or Ceph? MySQL + Galera? Would OwnCloud scale to many hundreds of users? How would you unify or integrate access control to all that? Nothing unsurmountable here, just interesting how it's done in real big on-prem installations.
- SoftTalker 3y ago
  Many users don't need multi-DC, multi-AZ resilience. Having a good (and tested) DR plan might be enough.
  - nine_k 3y ago
    This is correct. The original comment was about having basically anything AWS has to offer run on-prem, assuming that these things are actually needed.

nine_k 3y ago

> i cant really think of anything we run in AWS that is not eventually supposed to go back on prem Hod do you handle multi-DC, multi-AZ resilience? What are you using instead of IAM policies that cover every resource? (Asking unironically, would just like to know. I worked at cloud-centric companies for last 15 years or so.)

Stranger43 3y ago
For AWS we rely almost entirely on backup/restore and database level log shipping on prem this is is in some cases argumented by storage level(netapp and previously hpe 3par) replication but this is going away for cost reasons. Given that my experience with AWS is to use the same application level cross region resilience techniques im used to on prem(i have worked with high end unix boxen most of my carear) im genuinely baffled when people start talking about cloud resilience as something magical, and nearly all our traffic happens inside of an private network(MPLS/VPN) that stretches across the different sites. I really haven't seen any magic multi-az resiliiance in aws that dont have an onprem counter part. None of this is in house whitebox hardware but relatively standard solutions from established vendors(VMware recently started abusing their near monopoly so everyone is looking for/at alternatives like proxmox, xen and nutanix but arent ready to move just yet).
- nine_k 3y ago
  Thanks! Certainly multi-AZ setups are not magical. I just wanted to know how it's now done on-prem, what are the tools, solutions, approaches. Doing this in a cost-effective way for 1000 servers likely takes different tools than doing this at a major cloud provider scale. One thing that's more "magical" in AWS is S3. Their architecture is impressive, see e.g. https://www.allthingsdistributed.com/2021/04/s3-strong-consistency.html https://www.allthingsdistributed.com/2021/04/s3-strong-consi...
kaliszad 3y ago
I guess the same as people have always done. You have more server rooms on ideally different networks and grids. That can be a rented room/ rack in a bigger DC, your own if you have a big factory with subsidiaries etc. The IAM etc. will probably done by a combination of technical on organizational measures. You will have certain people doing certain things at least before the solution is ready for IaC. People will create roles, accounts, accesses and such. With networking gear that can be still tricky to implement, with virtualization solutions that is easier today. For databases etc. you can create accounts in them too. Of course, K8s and similar make these things more formalized/ transferable. However there is a lot of stuff before you can deploy that. People forget however that even if you have hundreds of servers you are tiny compared to the cloud providers. You don't have to have the same breadth and depth of offering. So while you need more baby sitting of hardware, probably will not get nearly as good deals on hardware as the big providers do, you will save their considerable margins. Also, they actually have some of the same expenses too - if a harddrive goes bad they will still swap them basically the same as you do. Big cloud providers will not get substantially different energy pricing than what e.g. a steel foundry would get. By hosting things on premise or in a nearby datacenter(s) you can shave off a lot of latency too. Some machinery likes to store a lot of data and shaving off latency will decrease your need for thick router buffers because you will not have such a big Bandwidth Delay Product and will achieve the same speeds with much smaller buffers. Building stuff on premise just for you makes some things easier too. Even if you loose some credentials usually you can just hard-reset the equipment as the last resort. There will be no credit card blocking that would affect the operations. If you are less strict with security it will usually matter much less - you are not sharing the hardware with unknown parties and all people that touch it have a contract with the company. So usually everybody want the company to succeed to get the paycheck. You build a deeper know how and can do some optimizations the cloud providers cannot do because you don't have to be general.
- nine_k 3y ago
  Thanks, this makes sense! I mostly thought about the software infrastructure side. With thousands and even mere hundreds of servers over several locations you already want some uniformity. Would you run k8s? Nomad + Consul? MinIO or Ceph? MySQL + Galera? Would OwnCloud scale to many hundreds of users? How would you unify or integrate access control to all that? Nothing unsurmountable here, just interesting how it's done in real big on-prem installations.
SoftTalker 3y ago
Many users don't need multi-DC, multi-AZ resilience. Having a good (and tested) DR plan might be enough.
- nine_k 3y ago
  This is correct. The original comment was about having basically anything AWS has to offer run on-prem, assuming that these things are actually needed.