One issue is that other Mastodon servers would need to decrypt the messages if any of their users is a member of a private group. Furthermore, users can switch

One issue is that other Mastodon servers would need to decrypt the messages if any of their users is a member of a private group. Furthermore, users can switch servers, which effectively means that any server they can move to could get access. If you want end-to-end encryption then the key management needs to be on the client and the client software needs to be securely distributed, which makes it more like a mobile or desktop app than a web app.