You can't control the server operator though. In practice I have no way of preventing Signal from turning off SGX any time they feel like it because they aren'

You can't control the server operator though. In practice I have no way of preventing Signal from turning off SGX any time they feel like it because they aren't open and federated. They could push a software update and I'd be none the wiser until a security researcher noticed and pointed it out! I guess it would be nice if Matrix integrated some sort of remote attestation support into the identity server protocol. Maybe they should have done so from the start, maybe not. At this point I don't see their offering as being less secure than Signal's though, just slightly different. It's also worth noting that SGX (and AMD's competitive offering) has been broken an embarrassing number of times at this point. From my perspective, secure storage by the server is more or less orthogonal to any given spec or implementation. You either provide data to a server as cleartext or ciphertext. The server does with it what it will - you as the user have effectively no control over that.