I went on this trip too, and while yes asbestos is bad, white asbestos (by far the most common type, mined and used) is way less likely to cause the type of dam

I went on this trip too, and while yes asbestos is bad, white asbestos (by far the most common type, mined and used) is way less likely to cause the type of damage you're describing, brown and blue asbestos are more inline with what you describe. But even then, people who lived in Wittenoom, WA are still alive today.