I see where your confusion stems from. In the case of asbestos, it is the contractors using it to build the house who tend to experience adverse health effects,

I see where your confusion stems from. In the case of asbestos, it is the contractors using it to build the house who tend to experience adverse health effects, and not always the people who live there. Asbestos is harmless when it's inert in the wall, paint, etc. The danger is when you start doing construction / renovations and the dust is free floating. So OP's point is correct. The two situations are not exactly analogous.