Linux package managers are great I haven't had to build a program from source in years. But how do I know the source code I check is the binary my machine runs

Linux package managers are great I haven't had to build a program from source in years. But how do I know the source code I check is the binary my machine runs? Even if I build from source I could have a malicious gcc that takes clean source and outputs a malicious binary. Unless you are running jit or something you can't really know what your computer is running even if you use open source.