On the other hand, they are predicting the Fed's actions, that's always a significant part of the models, and the Fed and the government etc aren't random actor

On the other hand, they are predicting the Fed's actions, that's always a significant part of the models, and the Fed and the government etc aren't random actors, they are somewhat predictable (and stability is their goal which helps in that regard). But they aren't really the market makers by themselves, so even if you get those predictions right, you might still be far away from being correct.