The problem is not running kubernetes on "bare-metal". That's the somewhat easy and cheap part. The problem is the supply chain to spec, purchase, rack, validat

The problem is not running kubernetes on "bare-metal". That's the somewhat easy and cheap part. The problem is the supply chain to spec, purchase, rack, validate, provision and keep up to date your servers, their drivers and firmware. It's your capacity planning. It's keeping up with the rate of hardware evolution. What happens when you have to go from 10G to 25G networking ? Oh and you also need a solid scalable L4 loadbalancing solution (facebook has a nice opensource dataplane based on ebpf) You can build / assemble from open-source bits most of the classic IaaS bits (compute/storage/networkLB). It's doable, the open-source solutions are pretty good. But unless you have 10k compute nodes how can you bear the cost of having the expert knowledge necessary to debug this ? Hardware sucks and is always on fire. Once you take this into account, the economic equation changes dramatically.