Handyman here. You're largely correct. The siding is absolutely no big deal because it doesn't release fibers unless it breaks. Because it's solid and brittle,

Handyman here. You're largely correct. The siding is absolutely no big deal because it doesn't release fibers unless it breaks. Because it's solid and brittle, when it _does_ break, hardly any goes into the air. Plus you'd be outside, likely protected by at least a gentle breeze. Asbestos was used in floor tiles (generally on top of concrete), these are perfectly safe if kept waxed or with a clearcoat applied on top. But you'd want to wear a respirator if removing them. There was asbestos insulation in attics and often between walls, or wrapped around pipes. That's the worst stuff and you should use a respirator just being around it. (I'd wear a respirator in any attic, to be honest.)