The OS is in charge of telling the CPU what to run. Literally the context switching code sets saves its old registers and reads in the registers of the program

The OS is in charge of telling the CPU what to run. Literally the context switching code sets saves its old registers and reads in the registers of the program that's about to run (including the program counter that says what address to start at). So at that time it can save information about how the current time and long the previous code ran. Low-level hardware block usage generally isn't accounted for that way, but CPUs can keep track of performance counter events tracking how many of each operation are performed and how many cycles each subunit is active. The OS can periodically read that data, but it's only used for low level metrics about how efficient the code is rather than for calculating overall CPU utilization. There are separate measurements about GPU and hard disk utilization, which are either determined by when driver queues work on the device and receives a notification that the device is done, or by the device keeping track of what work it's switching between and reporting it periodically.