You seem to be missing a key point. Brahmins are bound by rules, like: 1. They must sustain themselves by a livelihood that did little or no harm to creatures

You seem to be missing a key point. Brahmins are bound by rules, like: 1. They must sustain themselves by a livelihood that did little or no harm to creatures. 2. They could only accumulate minimum wealth for their sustenance, and that too only through the activities specific to them. 3. Punishments were harsher for Brahmins. For example, in instances of Theft, i. Shudra had to pay 8 times the money stolen ii. Vaishya had to pay 16 times the money stolen iii. Kshatriya had to pay 32 times the money stolen iv. Brahmins had to pay 64 to 128 times the money stolen 4. Brahmins were forbidden to make money by selling meat 5. They could not charge interest on lending 6. Could not touch weapons 7. Had to avoid seeking praise 8. Was not allowed to expect gifts from Shudras except in special circumstances. (Source: Snakes in the Ganga - Breaking India 2.0) It's their duty to perform rituals for other varnas, and if they get some money by performing those rituals, they can only use it for basic needs like food. Brahmins were ascetics who had to follow the most difficult of rules. That and in addition to the fact that they must be knowledgeable about Vedas and other scriptures is why a Brahmin's life is considered as the hardest out of all varnas. With such rules in play, it would be impossible for Brahmins to acquire wealth or military power to subdue Buddhists.