tl;dr: In 2020, I would expect most programs to be ultimately CPU bound (meaning the cost of the cpu is the dominant cost – you can upgrade other 3 resources mo

tl;dr: In 2020, I would expect most programs to be ultimately CPU bound (meaning the cost of the cpu is the dominant cost – you can upgrade other 3 resources more cheaply until cpu becomes the bottleneck). Hence cpu optimizations matter today more than it did in 2008. A program's execution time is a function of cpu performance, memory performance, disk performance or network performance. Performance here is a complex phenomenon – usually we measure performance in terms of 'latency' and 'throughput' of the operations we are executing – and its characteristics vary by attributes like – sequential vs random, block sizes, simple vs complex instructions, instruction offloads, queue depths, scheduling queues, context/mode switching etc. Imagine a naive execution of a program where all operations (cpu, memory, disk, network) are sequentially scheduled and executed. We can make two observations about such a program: 1. This program's execution time can be sped up by upgrading to a faster cpu or memory or disk or network – which of these we should upgrade first depends on what operation the program spends most of its execution time vs what costs less to upgrade. 2. While the program is executing on one of the four resources, the other resources are idling. In other words resource utilization will be less. #1 happens every few months/years as the hardware becomes faster for the same price. #2 is addressed by modern cpus, compilers and operating system schedulers through mechanisms like pipelining, parallelizing, prefetching, offloading etc. – to increase the overall utilization of all the resources while the program is executing. These techniques turn this naive program execution into a complex program execution. The automated optimization of the naive program in this way is not perfect/complete. A programmer will have to adjust the program to utilize the idling resources on a computer. This is the performance optimization work. Even after doing this, the program execution will be constrained by one of the 4 resources. In this situation we can say its execution time is bound by that resource's performance. Theoretically, if that resource performance were to become faster (through hardware upgrade), then the program will no longer by bound by that particular resource and it will now be bound by another resource. By definition, while the program execution time is bound by one resource, the other resources are under-utilized. In designing hardware+software systems, a purist objective is to ensure least resource underutilization occurs while ensuring the program's performance objective is met. Since resources cost differently, the under-utilization has to be weighed by its cost. This resource optimization at a datacenter level takes on a different dimension – a common mistake I have seen is to provision $5000 servers (where majority cost component is cpu and memory) and skimp on the network bandwidth between the servers. To build a full-clos non-blocking inter-server network at a reasonable enterprise scale, it would cost less than $300 per 10GE port. I have seen people save $150 per port and deploy an over-subscribed network that results in indeterministic network performance and cost much more wastage in the utilization of those $5000 servers. Another common mistake I have seen is to provision too much RAM (expensive in purchase cost as well as running cost - power/cooling) while not provisioning enough high-speed SSDs. In 2020, I would expect most programs to be ultimately CPU bound (meaning the cost of the cpu is the dominant cost – you can upgrade other 3 resources more cheaply until cpu becomes the bottleneck). Hence cpu optimizations matter today more than it did in 2008.