I use my passport to verify my right to work in the US during the first week of my job, and then I don't bring it any other time. So unless you're changing jobs

I use my passport to verify my right to work in the US during the first week of my job, and then I don't bring it any other time. So unless you're changing jobs very frequently it shouldn't be necessary to carry all the time. A bigger issue would be that many people do not have passports.