Note that mankind cannot make Von Neumann probes. We're still having troubles landing serious equipment on the moon (see recent crash), let alone create a probe

Note that mankind cannot make Von Neumann probes. We're still having troubles landing serious equipment on the moon (see recent crash), let alone create a probe that would be able to harvest raw materials for making a duplicate probe, refining those to the ingredients needed, converting those to probe building blocks, assemble the new probe, provide it with propellant, program it, and launch it. So I'm not sure why you'd think that a civilisation comparable to humans would be able to. Also: if you can build Von Neumann probes, you can construct a Dyson swarm. You just instruct the probes to create the components of the swarm from the raw materials in your star system. But that would massively boost a civilisation's energy budget. Therefore, it's quite likely that any civilisation capable of building Von Neumann probes will have built a Dyson swarm. So, secondly, it's about as paradoxical as the lack of detected Dyson swarms.