Black Holes of reasonable can be detected by gravitational lensing. If Black Holes / Brown Dwarfs / etc. were primarily responsible for the dark matter locally,

Black Holes of reasonable can be detected by gravitational lensing. If Black Holes / Brown Dwarfs / etc. were primarily responsible for the dark matter locally, lensing searches would have detected more. See e.g. https://arxiv.org/abs/astro-ph/0002058 https://arxiv.org/abs/astro-ph/0002058 for an overview.