My understanding is that you never directly read memory to which you shouldn't have access. You arrange for the processor to read a specific address you control

My understanding is that you never directly read memory to which you shouldn't have access. You arrange for the processor to read a specific address you control based on the value of the illegal read. By timing subsequent accesses to that user address, you can infer whether or not the processor brought it into the cache, and thus infer the value of the illegal read.