This shocks me too. When I first had to design a system around spot instances I assumed it caps out at the on-demand price. But in practice that is very much no

This shocks me too. When I first had to design a system around spot instances I assumed it caps out at the on-demand price. But in practice that is very much not the case. The spot price can routinely go above on-demand, with all the downsides of spot still being applied to the instance. It is because many systems are only built to support spot. When running a workload, they might not want to give up availability so they just pay the higher price. To be fair, this is a game that AWS engineers. They essentially want to kick people off the infrastructure to free it up for other uses. So this is a way to signal how badly you don't want to be kicked off. A lot of times after a price goes above on-demand pricing, it will dip down below fairly quickly as workloads all re-adjust. Some companies are willing to play that gamble, that a short period above on-demand is still cheaper in the long run when you average out the invoice at the end of the month.