8 ms·

Ask HN: Self hosted email, Cloud storage and dovecot

If I am self hosting my email for friends and family, but have a volume attached where dovecot stores the email, can I dynamically add another volume to the VM and tell dovecot to balance mail storage between 2 directories?

lathiat 4y ago
I would do this with LVM. You can grow LVM to expand onto a second volume and expand the file system with it. Or ZFS. https://wiki.archlinux.org/title/LVM#Add_physical_volume_to_a_volume_group https://wiki.archlinux.org/title/LVM#Add_physical_volume_to_... However note self hosting mail used to be super easy but these days most report you have a real hard time getting people to accept your mail. When your IP has no prior history and small volume you’ll get false flagged as spam a lot. Likely to be your larger challenge.
- a-user-you-like 4y ago
  What if I am doing this at a cheap VPS provider? Can I just add two of their storage volumes and somehow combine them into 1 bigger volume?
  - ThePowerOfFuet 4y ago
    That makes it worse; cheap VPS providers tend to have lots of abuse issues, so their IP blocks tend to have poor reputation.
    - a-user-you-like 4y ago
      But besides that I can mount multiple volumes and use LVM to make it look like one?
      - lathiat 4y ago
        Yes
pengaru 4y ago
That's best not done at the application (dovecot) level. LVM, md, or filesystems like btrfs or zfs can spread data across devices in myriad ways. Kudos for doing this for friends and family.
- a-user-you-like 4y ago
  Ok so I’ll just add 2 storage volumes, and use ZFS on Linux to combine them into 1 volume
  - rektide 4y ago
    It's so a reliability concern. If any volume dies you'll probably lose all the data. Your nomenclature ("volume") sounds like you are using some kind of cloud system. Just expand your volume? It's unclear why you are asking for your ask.
    - a-user-you-like 4y ago
      There are volume size limits on my cloud provider, but it sounds like combining multiple ones together might not be a great idea.
      - necovek 4y ago
        It's totally a good idea. There is a different question of whether that provides enough reliability, but that depends on if your cloud provider already offers safety in case of hardware failure for your volumes. Note that even today, with a single volume, if you lose it, you lose all your email. Most cloud providers would provide some resiliency for storage volumes though.
        a-user-you-like 4y ago
        Would you prefer to use ZFS or LVM to handle something like the storage of mail for files which might need to add a volume from time to time?
ThePowerOfFuet 4y ago
I recommend something like Migadu instead of rolling it yourself.
asne11 4y ago
Not addressing your question, but FWIW: Mail is the one thing I eventually caved on and offload to Microsoft. It's an uphill battle keeping your mail out of Spam folders. You setup DKIM, SPF, etc. according to the latest spec, and at the end of the day, the MS and Google still drop your mail in spam. They simply don't care. Second, the forces and money keeping MS above legally bound 99.9% uptime tower in comparison to any one person's own resources. Is your daily driver even operational 99% of the time you use it? We hackers can't help but to tinker, so I suppose not. MS has teams of engineers paid specifically to keep the thing running and NOT tinker. At only ~$10 / month that's a no brainer.
- mpldr 4y ago
  They should pay their engineers to fix their MTA-STS validation. Haven't done so in ages. If you think you're not up to selfhosting (which in my experience is quite feasible and comfortable): please choose a provider that is not one of the surveillance capitalists. Recommendations are Migadu and mailbox.org
- mpalfrey 4y ago
  Exactly the same here. I had email hosted with TSOHost here a few years ago and had no end of problems with bounce backs. In the end I moved it all to GSuite for a few bucks a month. It's simply worth it to avoid the issues with sender rep.
zxcvbn4038 4y ago
Sadly self hosted e-mail is a 24/7 job where you will be fighting against big players for the luxury of sending e-mail from your own system. Despite the dozen or so different standards to attest you are a legitimate sender it has basically boiled down to a handful of big e-mail providers that trust each other and everyone else is eyed with utter suspicion and contempt - and it has nothing to do with the fact every member of the “in” crowd makes significant money selling access to their infrastructure - honest (sarcasm).
- citrin_ru 4y ago
  I self host my email spending a couple hours a month at most. But I have email related experience and if this is a new for you area you will need to learn it spending more time. Risk of being blocked by big players (gmail/O365) is real but IMHO very low if you follow BCP. The main problem is that big companies make it very hard or impossible to contact them to resolve the issue.
  - mpldr 4y ago
    > Risk of being blocked by big players (gmail/O365) is real but IMHO very low if you follow BCP Can confirm, only issue I have is with outlook.com because their MTA-STS validator is… for lack of a better word: wrong. I decided that replying to mails from outlook.com is not part of my mailserver's offerings and if it's required others channels have to be sought out. I'm not going to violate established standards just so Microsoft can do things their way.
  - boisgerault 4y ago
    Could you elaborate what BCP is? (I am not familiar with the term)
    - citrin_ru 4y ago
      BCP is Best Current Practice, which for email includes at least having Forward-confirmed reverse DNS for mailserver hostname (with the same FQDN in EHLO hostname), SPF, DKIM, DMARC. I'm sure most people here know about DMARC but there are still a lot of mail senders which either have no DMARC record, or send some messages in a way that either DKIM or SPF or even both fail, or have a syntax error in SPD/DKIM/DMARC DNS records. On top of this it is good to ensure that your servers doesn't generate email backsatter when possible. Send rate limits is a simple but rarely enforced by small servers way to limit amount of spam which can be send if user credentials or web-app hosted on the same server are compromised.
martinkivi 4y ago
Your question doesn't directly indicate which problem you are trying to solve. It would be helpful if you would mention the hosting provider as their service offerings do differ. If you are trying to solve storage reliability for single mail server then creating a file system level failover would be way to go as mentioned previously (LVM, soft RAID over multiple volumes, etc). If you are trying to solve mail service reliability/failover then adding second mail server would be helpful. In that case you can use dsync (does master/master sync) to keep emails on both servers in sync.
- a-user-you-like 4y ago
  Nice I didn’t know about dsync. I imagine it works on large amounts of email, in case I want to host it for a nonprofit?
rmdes 4y ago
One battleground tested self-hosted solution : Cloudron.io
- bachmeier 4y ago
  I don't understand their service. It's $180/year or $30/month but you have still have to run your own server. What are you paying for that would be worth that amount of money?

lathiat 4y ago

I would do this with LVM. You can grow LVM to expand onto a second volume and expand the file system with it. Or ZFS. https://wiki.archlinux.org/title/LVM#Add_physical_volume_to_a_volume_group https://wiki.archlinux.org/title/LVM#Add_physical_volume_to_... However note self hosting mail used to be super easy but these days most report you have a real hard time getting people to accept your mail. When your IP has no prior history and small volume you’ll get false flagged as spam a lot. Likely to be your larger challenge.

a-user-you-like 4y ago
What if I am doing this at a cheap VPS provider? Can I just add two of their storage volumes and somehow combine them into 1 bigger volume?
- ThePowerOfFuet 4y ago
  That makes it worse; cheap VPS providers tend to have lots of abuse issues, so their IP blocks tend to have poor reputation.
  - a-user-you-like 4y ago
    But besides that I can mount multiple volumes and use LVM to make it look like one?
    - lathiat 4y ago
      Yes

pengaru 4y ago

That's best not done at the application (dovecot) level. LVM, md, or filesystems like btrfs or zfs can spread data across devices in myriad ways. Kudos for doing this for friends and family.

a-user-you-like 4y ago
Ok so I’ll just add 2 storage volumes, and use ZFS on Linux to combine them into 1 volume
- rektide 4y ago
  It's so a reliability concern. If any volume dies you'll probably lose all the data. Your nomenclature ("volume") sounds like you are using some kind of cloud system. Just expand your volume? It's unclear why you are asking for your ask.
  - a-user-you-like 4y ago
    There are volume size limits on my cloud provider, but it sounds like combining multiple ones together might not be a great idea.
    - necovek 4y ago
      It's totally a good idea. There is a different question of whether that provides enough reliability, but that depends on if your cloud provider already offers safety in case of hardware failure for your volumes. Note that even today, with a single volume, if you lose it, you lose all your email. Most cloud providers would provide some resiliency for storage volumes though.
      - a-user-you-like 4y ago
        Would you prefer to use ZFS or LVM to handle something like the storage of mail for files which might need to add a volume from time to time?

ThePowerOfFuet 4y ago

I recommend something like Migadu instead of rolling it yourself.

asne11 4y ago

Not addressing your question, but FWIW: Mail is the one thing I eventually caved on and offload to Microsoft. It's an uphill battle keeping your mail out of Spam folders. You setup DKIM, SPF, etc. according to the latest spec, and at the end of the day, the MS and Google still drop your mail in spam. They simply don't care. Second, the forces and money keeping MS above legally bound 99.9% uptime tower in comparison to any one person's own resources. Is your daily driver even operational 99% of the time you use it? We hackers can't help but to tinker, so I suppose not. MS has teams of engineers paid specifically to keep the thing running and NOT tinker. At only ~$10 / month that's a no brainer.

mpldr 4y ago
They should pay their engineers to fix their MTA-STS validation. Haven't done so in ages. If you think you're not up to selfhosting (which in my experience is quite feasible and comfortable): please choose a provider that is not one of the surveillance capitalists. Recommendations are Migadu and mailbox.org
mpalfrey 4y ago
Exactly the same here. I had email hosted with TSOHost here a few years ago and had no end of problems with bounce backs. In the end I moved it all to GSuite for a few bucks a month. It's simply worth it to avoid the issues with sender rep.

zxcvbn4038 4y ago

Sadly self hosted e-mail is a 24/7 job where you will be fighting against big players for the luxury of sending e-mail from your own system. Despite the dozen or so different standards to attest you are a legitimate sender it has basically boiled down to a handful of big e-mail providers that trust each other and everyone else is eyed with utter suspicion and contempt - and it has nothing to do with the fact every member of the “in” crowd makes significant money selling access to their infrastructure - honest (sarcasm).

citrin_ru 4y ago
I self host my email spending a couple hours a month at most. But I have email related experience and if this is a new for you area you will need to learn it spending more time. Risk of being blocked by big players (gmail/O365) is real but IMHO very low if you follow BCP. The main problem is that big companies make it very hard or impossible to contact them to resolve the issue.
- mpldr 4y ago
  > Risk of being blocked by big players (gmail/O365) is real but IMHO very low if you follow BCP Can confirm, only issue I have is with outlook.com because their MTA-STS validator is… for lack of a better word: wrong. I decided that replying to mails from outlook.com is not part of my mailserver's offerings and if it's required others channels have to be sought out. I'm not going to violate established standards just so Microsoft can do things their way.
- boisgerault 4y ago
  Could you elaborate what BCP is? (I am not familiar with the term)
  - citrin_ru 4y ago
    BCP is Best Current Practice, which for email includes at least having Forward-confirmed reverse DNS for mailserver hostname (with the same FQDN in EHLO hostname), SPF, DKIM, DMARC. I'm sure most people here know about DMARC but there are still a lot of mail senders which either have no DMARC record, or send some messages in a way that either DKIM or SPF or even both fail, or have a syntax error in SPD/DKIM/DMARC DNS records. On top of this it is good to ensure that your servers doesn't generate email backsatter when possible. Send rate limits is a simple but rarely enforced by small servers way to limit amount of spam which can be send if user credentials or web-app hosted on the same server are compromised.

martinkivi 4y ago

Your question doesn't directly indicate which problem you are trying to solve. It would be helpful if you would mention the hosting provider as their service offerings do differ. If you are trying to solve storage reliability for single mail server then creating a file system level failover would be way to go as mentioned previously (LVM, soft RAID over multiple volumes, etc). If you are trying to solve mail service reliability/failover then adding second mail server would be helpful. In that case you can use dsync (does master/master sync) to keep emails on both servers in sync.

a-user-you-like 4y ago
Nice I didn’t know about dsync. I imagine it works on large amounts of email, in case I want to host it for a nonprofit?

rmdes 4y ago

One battleground tested self-hosted solution : Cloudron.io

bachmeier 4y ago
I don't understand their service. It's $180/year or $30/month but you have still have to run your own server. What are you paying for that would be worth that amount of money?