The problem is that you wouldn't have to 'hack the server', at least for Matrix, given anyone can run a server - so all it would take is for one server in the r

The problem is that you wouldn't have to 'hack the server', at least for Matrix, given anyone can run a server - so all it would take is for one server in the room to be vulnerable to a social or technical attack (e.g. a nosey sysadmin). If an escrow public key has been mixed into the e2ee encryption, then all it takes is for the private key to be leaked (e.g. for a price on a dark market) and the nosey sysadmin can go and break the encryption of all its users. The same goes for a centralised service too (c.f. the twitter hack) - while it might be less likely given the smaller attack envelope, it's a much bigger prize.