In my mind, the whole point of using Signal is that I don't have to trust the server. Why would it matter who hosts the server if we can trust that the clients'

In my mind, the whole point of using Signal is that I don't have to trust the server. Why would it matter who hosts the server if we can trust that the clients' communications are E2E encrypted?