6 ms·

I setup an enterprise-grade Mattermost server in a weekend. The hardest part wasn't even Mattermost itself, but getting SMTP working reliably in the age of DKIM

by 22c 7y ago

I setup an enterprise-grade Mattermost server in a weekend. The hardest part wasn't even Mattermost itself, but getting SMTP working reliably in the age of DKIM/SPF/DMARC policies. This could be sidestepped by using any of the well-established email providers. No reason to use Slack when such options exist, but also no reason to write your own chat app, either.

raz32dust 7y ago
If the company relies on Mattermost, I would assume they'd need at least one maintainer per ~100 users, just to make sure that the SMTP keeps working reliably, there are backups etc. One person to maintain Mattermost would cost a company at least 150000 USD in the bay area. That's $100+ per month per user. Slack is way cheaper than that.
- dx034 7y ago
  Really, 50 maintainers for a 5000 user system? I'd think it scales much better, probably 2-3 people for the whole company. And in that case, it can quickly be cheaper than slack
  - linuxdude314 7y ago
    This is a more realistic scenario. I own/operate shared services for thousands of users on a team of 3. If it takes you that many engineers you are either using the wrong software or using it incorrectly.
  - amerine 7y ago
    Exactly. No way does it take 50 people to scales mattermost to 5k because of smtp. This op made that all up.
  - raz32dust 7y ago
    I think you are really underestimating the bureaucratic overheads in a large company. It is not the same as a 3-person startup serving 5000 customers. Of course I pulled that number out of the air, but the point is that it can make sense. I think I am right around the ballpark of needing a 5 people team to serve 500 users - you'd need a minimum of 3 people just to make sure there is enough redundancy if one person is sick or goes on vacation. You'd need 24x7 on-calls. The scaling factor probably changes after that - may be it is 10 people for 5000 users instead of 50. Adding to other large company problems, these people will need management layers, the team will need a charter for growth. Who wants to be the maintainer of the Mattermost servers in a large company? Add all of this up and then the slack deal starts to look reasonable. Edit: Just to add some real numbers, slack costs $12.50 per user per month [1] - that is ~$750k per year for 5000 users = 5 engineers. (More like 3 really in the Bay area) [1] https://slack.com/pricing https://slack.com/pricing
    - sciurus 7y ago
      Or you have your existing corporate IT team manage it the same way they're managing other internal apps. There's no way it suddenly needs a new team of dedicated full time resources.
    - candu 7y ago
      Your general point is solid, but one quibble with the numbers - even the pricing page has an "enterprise" tier "for very large businesses". There's no way a company with 5000 Slack users is paying the sticker price per head - they'll call Slack's enterprise sales team and negotiate some volume discount with whatever shiny enterprise-grade features, SLAs, support guarantees, etc. they need.
  - catwell 7y ago
    At Uber scale, you probably already have your own SMTP servers used by all your employees, so that's not really a problem. Moreover, email is hard and does require a dedicated team, but the order of magnitude is wrong here. 5 people is plenty.
- kortilla 7y ago
  Lol, where did you get the idea that smtp servers need more maintenance as users increase? What if I told you I’ve witnessed a team of 3 manage a mail system of a university of over 100k active email accounts?
- jholloway7 7y ago
  You're making a good underlying point but contriving way too many details in your argument for it to be taken seriously.
- Slartie 7y ago
  We are a company with around 100-150 employees, using Mattermost in a real Enterprise setting for years now (with SSO, CI and mail integration and whatnot) and that server is being maintained by our admin team basically "on the side" - nobody was hired specifically for this job, as it does take less than 20% of one person's time according to what the team says. These Bay Area tech people must be either incredibly inefficient (especially when comparing to what they get paid), or your estimate must be waaaaay off.
dvfjsdhgfv 7y ago
> The hardest part wasn't even Mattermost itself, but getting SMTP working reliably in the age of DKIM/SPF/DMARC policies. This could be sidestepped by using any of the well-established email providers. This is valid not just for Mattermost but many other apps that support sending e-mails. And that's why it's important that instead of giving up and outsourcing it, we all try to set it up and have first-hand experience in challenges we have to face in the process. Because if everybody just uses Mailgun & SES, in a decade or two it won't be be possible to set up your e-mail server anymore (of course you will be able to, but your e-mails won't reach most customers).
- 22c 7y ago
  What's interesting is setting up a handler for DMARC aggregated feedback reports. I determined that it's not worth doing yourself, even Microsoft do not handle their own DMARC reports. I ended up outsourcing DMARC report processing, if I wanted to do it in-house I'd have to setup and maintain an Elastic stack[1] which seems like huge overkill. [1] https://domainaware.github.io/parsedmarc/ https://domainaware.github.io/parsedmarc/
- 22c 7y ago
  > of course you will be able to, but your e-mails won't reach most customers Unfortunately, I found this to already be somewhat of a reality. Even with all the best practices I went to, many MTAs simply don't trust my IP address. Most "sane" MTAs seem to greylist me rather than blacklist outright, so it just adds a bit of delay to users receiving their emails. Thankfully it seems to be gradually getting better, but gone are the days of simply pointing your apps at sendmail letting them go.
jholloway7 7y ago
Let's not pretend that hosting an OSS chat server is a walk in the park for every enterprise just because you personally got one up and running in a weekend.
- Piskvorrr 7y ago
  Let's not pretend that building one from scratch isn't at least an order of magnitude harder, either.
  - mucholove 7y ago
    Looking at this from Spain, you did do this over the weekend. I also work 6 days a week—but my hope is that this didn’t take the place of building a family, having a relationship, finding a relationship going for a hike in the mountains, citizen science, or making art. Your weekend is valuable!
    - a_t48 7y ago
      You're totally right that the weekend is valuable, but some people enjoy doing this sort of thing with their free time. It's not anyone's place to decide what anyone else does for entertainment or self improvement. That is to say - what is not valuable to you might be valuable to someone else.
      - bravura 7y ago
        Actually if your leisure activities and your work activities are similar, and you don’t have many interests outside engineering (or finance or law or whatever), then it is the prerogative of others to criticize your lack of well roundedness and insularity.
        danjac 7y ago
        Why? If you're not a friend or family member, it's none of your business.
IanPBann 7y ago
Uber have talked about their internal chat app, uChat. When OP says they built their own internal chat app, that's not entirely accurate - uChat is built upon Mattermost. https://eng.uber.com/uchat/ https://eng.uber.com/uchat/
- webXL 7y ago
  That kinda sucks the air out of this whole thread. "that's not entirely accurate" is a bit of an understatement. But, don't let facts get in the way of a good rant! Mattermost has been rock solid for the last couple of years for my team BTW.

by 22c 7y ago

raz32dust 7y ago

If the company relies on Mattermost, I would assume they'd need at least one maintainer per ~100 users, just to make sure that the SMTP keeps working reliably, there are backups etc. One person to maintain Mattermost would cost a company at least 150000 USD in the bay area. That's $100+ per month per user. Slack is way cheaper than that.

dx034 7y ago
Really, 50 maintainers for a 5000 user system? I'd think it scales much better, probably 2-3 people for the whole company. And in that case, it can quickly be cheaper than slack
- linuxdude314 7y ago
  This is a more realistic scenario. I own/operate shared services for thousands of users on a team of 3. If it takes you that many engineers you are either using the wrong software or using it incorrectly.
- amerine 7y ago
  Exactly. No way does it take 50 people to scales mattermost to 5k because of smtp. This op made that all up.
- raz32dust 7y ago
  I think you are really underestimating the bureaucratic overheads in a large company. It is not the same as a 3-person startup serving 5000 customers. Of course I pulled that number out of the air, but the point is that it can make sense. I think I am right around the ballpark of needing a 5 people team to serve 500 users - you'd need a minimum of 3 people just to make sure there is enough redundancy if one person is sick or goes on vacation. You'd need 24x7 on-calls. The scaling factor probably changes after that - may be it is 10 people for 5000 users instead of 50. Adding to other large company problems, these people will need management layers, the team will need a charter for growth. Who wants to be the maintainer of the Mattermost servers in a large company? Add all of this up and then the slack deal starts to look reasonable. Edit: Just to add some real numbers, slack costs $12.50 per user per month [1] - that is ~$750k per year for 5000 users = 5 engineers. (More like 3 really in the Bay area) [1] https://slack.com/pricing https://slack.com/pricing
  - sciurus 7y ago
    Or you have your existing corporate IT team manage it the same way they're managing other internal apps. There's no way it suddenly needs a new team of dedicated full time resources.
  - candu 7y ago
    Your general point is solid, but one quibble with the numbers - even the pricing page has an "enterprise" tier "for very large businesses". There's no way a company with 5000 Slack users is paying the sticker price per head - they'll call Slack's enterprise sales team and negotiate some volume discount with whatever shiny enterprise-grade features, SLAs, support guarantees, etc. they need.
- catwell 7y ago
  At Uber scale, you probably already have your own SMTP servers used by all your employees, so that's not really a problem. Moreover, email is hard and does require a dedicated team, but the order of magnitude is wrong here. 5 people is plenty.
kortilla 7y ago
Lol, where did you get the idea that smtp servers need more maintenance as users increase? What if I told you I’ve witnessed a team of 3 manage a mail system of a university of over 100k active email accounts?
jholloway7 7y ago
You're making a good underlying point but contriving way too many details in your argument for it to be taken seriously.
Slartie 7y ago
We are a company with around 100-150 employees, using Mattermost in a real Enterprise setting for years now (with SSO, CI and mail integration and whatnot) and that server is being maintained by our admin team basically "on the side" - nobody was hired specifically for this job, as it does take less than 20% of one person's time according to what the team says. These Bay Area tech people must be either incredibly inefficient (especially when comparing to what they get paid), or your estimate must be waaaaay off.

dvfjsdhgfv 7y ago

> The hardest part wasn't even Mattermost itself, but getting SMTP working reliably in the age of DKIM/SPF/DMARC policies. This could be sidestepped by using any of the well-established email providers. This is valid not just for Mattermost but many other apps that support sending e-mails. And that's why it's important that instead of giving up and outsourcing it, we all try to set it up and have first-hand experience in challenges we have to face in the process. Because if everybody just uses Mailgun & SES, in a decade or two it won't be be possible to set up your e-mail server anymore (of course you will be able to, but your e-mails won't reach most customers).

22c 7y ago
What's interesting is setting up a handler for DMARC aggregated feedback reports. I determined that it's not worth doing yourself, even Microsoft do not handle their own DMARC reports. I ended up outsourcing DMARC report processing, if I wanted to do it in-house I'd have to setup and maintain an Elastic stack[1] which seems like huge overkill. [1] https://domainaware.github.io/parsedmarc/ https://domainaware.github.io/parsedmarc/
22c 7y ago
> of course you will be able to, but your e-mails won't reach most customers Unfortunately, I found this to already be somewhat of a reality. Even with all the best practices I went to, many MTAs simply don't trust my IP address. Most "sane" MTAs seem to greylist me rather than blacklist outright, so it just adds a bit of delay to users receiving their emails. Thankfully it seems to be gradually getting better, but gone are the days of simply pointing your apps at sendmail letting them go.

jholloway7 7y ago

Let's not pretend that hosting an OSS chat server is a walk in the park for every enterprise just because you personally got one up and running in a weekend.

Piskvorrr 7y ago
Let's not pretend that building one from scratch isn't at least an order of magnitude harder, either.
- mucholove 7y ago
  Looking at this from Spain, you did do this over the weekend. I also work 6 days a week—but my hope is that this didn’t take the place of building a family, having a relationship, finding a relationship going for a hike in the mountains, citizen science, or making art. Your weekend is valuable!
  - a_t48 7y ago
    You're totally right that the weekend is valuable, but some people enjoy doing this sort of thing with their free time. It's not anyone's place to decide what anyone else does for entertainment or self improvement. That is to say - what is not valuable to you might be valuable to someone else.
    - bravura 7y ago
      Actually if your leisure activities and your work activities are similar, and you don’t have many interests outside engineering (or finance or law or whatever), then it is the prerogative of others to criticize your lack of well roundedness and insularity.
      - danjac 7y ago
        Why? If you're not a friend or family member, it's none of your business.

IanPBann 7y ago

Uber have talked about their internal chat app, uChat. When OP says they built their own internal chat app, that's not entirely accurate - uChat is built upon Mattermost. https://eng.uber.com/uchat/ https://eng.uber.com/uchat/

webXL 7y ago
That kinda sucks the air out of this whole thread. "that's not entirely accurate" is a bit of an understatement. But, don't let facts get in the way of a good rant! Mattermost has been rock solid for the last couple of years for my team BTW.