We would see something like that against the cosmic microwave background. Instead, we see that about 5/6ths of the total matter of the universe interacts with

We would see something like that against the cosmic microwave background. Instead, we see that about 5/6ths of the total matter of the universe interacts with the CMB only through gravity. Also, more hidden (somehow) baryonic matter would change our current theories surrounding Big Bang nucleosynthesis in ways that aren't supported by other observations, like the percentages of (visible) matter like helium and lithium.