I feel like you answered much of your own questions. Beyond that many of the departments that this data is being extracted from have rules about who can access

I feel like you answered much of your own questions. Beyond that many of the departments that this data is being extracted from have rules about who can access (no not everyone in the IRS has free reign) and what they can do with it. For good reason, IRS's job is to focus on what the law says they should do, not say punish political enemies and so on. But transfer it to DODGE, ICE, Palantir, there are no laws at all regarding what they can do with that data.