Your setup with Let's encrypt certificates is exactly what I was thinking about. I was thinking however about a one step further. Put the server on a smartphon

Your setup with Let's encrypt certificates is exactly what I was thinking about. I was thinking however about a one step further. Put the server on a smartphone. This way you're pretty much covered most of the time. Also one could be notified as soon as email is incoming. I thought also about a home server that would be used as a secondary. It would be best to give ability to create random subdomains, so a user can have a separate addresses for everyone. Then it would be easy to filter out spam on the proxy level. Which is crucial for a server on a mobile phone.