There are some clever ways of detecting such things. The density of gas and dust can be determined by measuring the effect on starlight passing through it. Stat

There are some clever ways of detecting such things. The density of gas and dust can be determined by measuring the effect on starlight passing through it. Statistically, larger objects should occasionally pass in front of background stars making them "blink off" briefly or in the case of black holes, lens the light from the background star, making it briefly brighter. Surveys have been done looking for such things. We can put upper limits on the amount of "traditional" matter, and there simply isn't enough of it out there to explain our observations.