Right, but the galaxy is 13 billion years old and only 100,000 light years wide. We could theoretically detect any operating Dyson swarm as long as it was in op

Right, but the galaxy is 13 billion years old and only 100,000 light years wide. We could theoretically detect any operating Dyson swarm as long as it was in operation 100,000 years ago or longer, and there's a 10 billion years of "longer" for someone to have built them. We don't have a view of what exactly is happening right now across the other side of the galaxy but (on the timescale of the galaxy) our view isn't that out of date. If civilizations are common it would be weird if none of them had more than a hundred thousand year head-start on us. That sort of timing would be freakishly coincidental on a billion-year timescale, like everyone finishing a marathon a tenth of a second apart despite having started half an hour earlier or later.