7 ms·

I tried to adopt matrix for our small business and here is summary of my experience. - Synapse is very resource hungry even for a small server - Synapse creat

by kuon 5y ago

I tried to adopt matrix for our small business and here is summary of my experience. - Synapse is very resource hungry even for a small server - Synapse creates a gazillion of TCP connection and keep them open, it was a real problem for my ISP router (SOHO router for 1gbit/s) and it took a very long time to debug, be sure to limit the number of TCP connection on the server - Synapse is hard to get a good idea of the health of the server, the federation can break without warning - If the federation broke for some reason, when submitting messages, they will show as delivered, this was one of the major problem for us - Encryption is very cumbersome to get right for non technical users, there should be a server wide option to make it "easier but less secure". Even for me, it was hard to get rid of a warning popup about some device error. - Group VOIP is hard to setup, it should be as easy as discord to be viable. - Webhook is really hard to setup too, I want to be able to right click a channel and "copy webhook URL" to be able to post to it with CURL (discord has that) - Globally it is very slow when joining federated rooms, it can take days to sync 500+ people room In the end we just use discord, it's free and do the job. Of course they can read our messages and this is a problem for me as I am for privacy and encryption, but it works for everybody, it is friendly and easy to use. I will keep a close eye on matrix and try to adopt it again, but right now it is not usable for us.

Arathorn 5y ago
When was this? I suspect most of the problems here are due to users joining lots of big federated rooms, which will use loads of resources irrespective of how many local users you have. This is also what Fast Joins are trying to improve (at least for join speed) from the original post. Likewise the matrix-hookshot webhook bridge.
- kuon 5y ago
  I discontinued the server 2 months ago, and I operated the server for about 2 years.
- Jnr 5y ago
  Tried Matrix (and Synapse) the 3rd time a couple of months ago and gave up after a few days. Joining any larger federated room took days or did not complete at all. And then the messages were also delayed for quite a while. That must be the stupidest design I have seen. If it was not so messed up, Matrix probably would have become huge by now. I agree with others in comments, devs should have probably improved the federation and written the official server in some compiled language and provided it as a single binary.
  - foxfluff 5y ago
    I kind of disagree about the language. Not because I wouldn't want an implementation in a fast language, but because I think a well designed chat system must be performant even if your language is an order of magnitude (or two) slower than it has to be. If you need to start throwing brute force at it, something's up with the design. Get the design right first and demonstrate it is right by showing that even a slow system has no trouble with it.
    - Jnr 5y ago
      Sorry, it was not clear that I do not hate the language (I write in python myself) and I fully understand that it is the design choice that is the problem. I simply think that having a single small binary can be advantageous for adoption.
  - Arathorn 5y ago
    It might be worth reading the actual post to see how we're fixing slow joins. In terms of the language: it's not Python's fault at all. Instead, Synapse baked in a bunch of naive assumptions (being originally a prototype) which we have been steadily fixing... while also adding features, which pulls in the opposite direction. Given a choice between a featureful server that's resource-heavy (which we can then make go faster later) versus one which lacks the features to compete with centralised alternatives, we chose the former.
    - Jnr 5y ago
      Sorry, it was not clear from my comment. I do not blame Python. Of course it was design/code issue. I just think that having a single binary would improve adoption. I like projects that provide a single binary. Without the need for docker, or setting up complex environments.
IceWreck 5y ago
Your problems are with synapse. I had a similar experience with it in 2019. Dropped matrix for a while and started using Dendrite in 2020. It still lacks features compared to synapse but I'm happy with the performance. Notifications are sorely missing but I think the PR to add them is finally ready to merge. Give it a year or two and you'll have a full featured matrix home server that can run on a raspberry pi.
- blitzar 5y ago
  I believe that is when the Year of the Linux Desktop is scheduled as well.
- italwayslags 5y ago
3np 5y ago
> federation issues > gazillion TCP connections We’ve been running a Synapse server with quite a bit of federated rooms, even larger ones like you mention. Never noticed a single issue like this that I recall. I wonder if the reason is that we’re doing whitelisting, instead of federation with everyone by default. The whitelisting is quite liberal; I’ll add a server as soon as I’m in a conversation thread with someone from that server, or just want to see their avatar. Sometimes when I’m waiting for something else I’ll just scroll through a room and add a handful of servers. Essentially this is my proxy for only federation with “legit” servers and avoid being DDoSed. > there should be a server wide option to make it "easier but less secure" Disagree. Sure, UX can be improved, but compromising here defeats the purpose of why we’re doing this in the first place. Another one of mine: - E2EE seems complex to implement and very few of the clients implement it fully and properly. This is the one point I think needs to be resolved before Matrix can be ready for wider adoption. It’s kind of pointless without E2EE, or if Element is the only client that can work for all rooms and messages.
- kuon 5y ago
  I didn't whitelist anything, maybe that was a source for issue. > Disagree. Sure, UX can be improved, but compromising here defeats the purpose of why we’re doing this in the first place. The thing is, as I ran my own server in my office, SSL (with no E2E) is already a good privacy. And even with that setup, the server users got "harassed" by the UI asking them to sync devices keys. For many users, a single prompt about security is enough to discourage them.
  - CameronNemo 5y ago
    So you are asking for a client setting (in Element I presume) to stop nudging users to set up e2ee? Did you file an issue for that?
    - kuon 5y ago
      If I try again to "convert" non technical people around me, I'll try to document properly what is wrong in the UX and experience. (but maybe it will all be fixed)
teitoklien 5y ago
Synapse is bloated and mostly good for very large organisations. Use conduit.rs or dendrite. I use conduit and it takes a couple of MB ram even while I’m in large rooms
- xvilka 5y ago
  Official server should have never been in Python anyway. Not a proper language for the network server. They should have adopted Go or Rust servers as the official ones, and pour resources there instead.
  - mst 5y ago
    Synapse was about proving the concept.
    - xvilka 5y ago
      Well, they proved it, time for it to be dropped, and focus on the production-level implementation. But it's still[1] the recommended one. [1] https://matrix.org/docs/guides/introduction#how-can-i-get-my-own-server https://matrix.org/docs/guides/introduction#how-can-i-get-my...
      - mst 5y ago
        I sympathise with your aggravation but they're balancing doing that while not trying to migrate the existing user base too early and keeping supporting people with production workloads. Fundamentally they've 'lucked' into a 'nice problem to have' type situation here, and I get how annoying it is - I was freenode staff during a period where the irc<->matrix bridging was unexpectedly popular enough to be a 'nice problem to have' for everybody involved and the author of the blog post we're discussing probably remembers some sharp words on my part during the process - but frankly on balance I'm mostly impressed.
- kuon 5y ago
  Last time I checked critical features were not supported, like read receipts.
nisa 5y ago
I'm running a small (at the beginning it was completely public, now I'm keeping an eye on registrations) homeserver with synapse for a few years - I really like Matrix on a conceptual level and I'm happy that it exists even through it's not perfect it works pretty well mostly. I don't want to be too harsh because it's the best decentralized alternative for chat at the moment and it's open-source and there is a lot of progress but damn' it was a pain in the ass to run that server... we solved our problems mostly with throwing resources and memory on the problem... it's a nice tour de force if you want to learn about debugging distributed systems but it's nothing to install somewhere and just keep running. synapse is quite a mess and there are a lot of hidden footguns if you just run a server - this is just the tip of iceberg to keep the postgres database somewhat sane: https://levans.fr/shrink-synapse-database.html https://levans.fr/shrink-synapse-database.html - lot's of other issues in the issue-tracker where you can just scratch your head. bridges are all subtly broken - the xmpp bridge is horrible and broke in so much interesting ways that I'm not going to touch it ever again - telegram works okay most of the time, irc-bridge also have some warts - but it's easy to criticize from my chair and probably unfair to talk so negative about it here but it's often buggy and broken for edge-cases - it works most of the time pretty okay but it's quite a mess to get a mental model for the code and so it's difficult to debug things. moderation/spam/etc.pp is all hackable but not there out of the box - it looks and feels like mostly quickly hacked up nodejs code that at least for us exploded in all kinds of interesting ways. https://github.com/matrix-org/mjolnir https://github.com/matrix-org/mjolnir writing 3tb of logs in a few day and eating memory like crazy for instance. You have to babysit it and there is no simple ui for anything. So it's powerful but requires quite a bit of dedication and patience to get right. It's a full blown distributed system and often state is all over the place and once you make a mistake it's difficult to impossible to get that thing do work correct again without starting over. But there are so much promising projects that I'm confident that these issues will be resolved and it will only get better but in my experience it will break badly on all kinds of edge-cases - the mentioned xmpp-bridge created usernames that can't be deleted via the http api for instance. someone bridged 1000 channels via our telegram-bridge and there is no code to remove those channels - you have to code something up in python for yourself. irc bridge kicks you after 30 days idle because they can't handle the connections - freenode (before the takeover) said it's not them - maybe single threaded nodejs is not such a good idea for that. Could I do it any better and delivering? Probably not. But except some adventure and if you want to deploy it for an org carefully test any assumptions you take for granted. It's cool but it's also kind of quick'n'dirty in a lot of ways. Still better than anything else I'd use it over any megacorp messenger anytime but maybe don't switch your family yet. But for using it you don't have to care - and there are great projects like https://github.com/spantaleev/matrix-docker-ansible-deploy https://github.com/spantaleev/matrix-docker-ansible-deploy that solve most of the problems out of the box and mobile clients and web clients and E2E crypto also works really well.
janikvonrotz 5y ago
mayli 5y ago
Ack, pretty all of those points are true, which is partially due to python.

by kuon 5y ago

Arathorn 5y ago

When was this? I suspect most of the problems here are due to users joining lots of big federated rooms, which will use loads of resources irrespective of how many local users you have. This is also what Fast Joins are trying to improve (at least for join speed) from the original post. Likewise the matrix-hookshot webhook bridge.

kuon 5y ago
I discontinued the server 2 months ago, and I operated the server for about 2 years.
Jnr 5y ago
Tried Matrix (and Synapse) the 3rd time a couple of months ago and gave up after a few days. Joining any larger federated room took days or did not complete at all. And then the messages were also delayed for quite a while. That must be the stupidest design I have seen. If it was not so messed up, Matrix probably would have become huge by now. I agree with others in comments, devs should have probably improved the federation and written the official server in some compiled language and provided it as a single binary.
- foxfluff 5y ago
  I kind of disagree about the language. Not because I wouldn't want an implementation in a fast language, but because I think a well designed chat system must be performant even if your language is an order of magnitude (or two) slower than it has to be. If you need to start throwing brute force at it, something's up with the design. Get the design right first and demonstrate it is right by showing that even a slow system has no trouble with it.
  - Jnr 5y ago
    Sorry, it was not clear that I do not hate the language (I write in python myself) and I fully understand that it is the design choice that is the problem. I simply think that having a single small binary can be advantageous for adoption.
- Arathorn 5y ago
  It might be worth reading the actual post to see how we're fixing slow joins. In terms of the language: it's not Python's fault at all. Instead, Synapse baked in a bunch of naive assumptions (being originally a prototype) which we have been steadily fixing... while also adding features, which pulls in the opposite direction. Given a choice between a featureful server that's resource-heavy (which we can then make go faster later) versus one which lacks the features to compete with centralised alternatives, we chose the former.
  - Jnr 5y ago
    Sorry, it was not clear from my comment. I do not blame Python. Of course it was design/code issue. I just think that having a single binary would improve adoption. I like projects that provide a single binary. Without the need for docker, or setting up complex environments.

IceWreck 5y ago

Your problems are with synapse. I had a similar experience with it in 2019. Dropped matrix for a while and started using Dendrite in 2020. It still lacks features compared to synapse but I'm happy with the performance. Notifications are sorely missing but I think the PR to add them is finally ready to merge. Give it a year or two and you'll have a full featured matrix home server that can run on a raspberry pi.

blitzar 5y ago
I believe that is when the Year of the Linux Desktop is scheduled as well.
italwayslags 5y ago

3np 5y ago

> federation issues > gazillion TCP connections We’ve been running a Synapse server with quite a bit of federated rooms, even larger ones like you mention. Never noticed a single issue like this that I recall. I wonder if the reason is that we’re doing whitelisting, instead of federation with everyone by default. The whitelisting is quite liberal; I’ll add a server as soon as I’m in a conversation thread with someone from that server, or just want to see their avatar. Sometimes when I’m waiting for something else I’ll just scroll through a room and add a handful of servers. Essentially this is my proxy for only federation with “legit” servers and avoid being DDoSed. > there should be a server wide option to make it "easier but less secure" Disagree. Sure, UX can be improved, but compromising here defeats the purpose of why we’re doing this in the first place. Another one of mine: - E2EE seems complex to implement and very few of the clients implement it fully and properly. This is the one point I think needs to be resolved before Matrix can be ready for wider adoption. It’s kind of pointless without E2EE, or if Element is the only client that can work for all rooms and messages.

kuon 5y ago
I didn't whitelist anything, maybe that was a source for issue. > Disagree. Sure, UX can be improved, but compromising here defeats the purpose of why we’re doing this in the first place. The thing is, as I ran my own server in my office, SSL (with no E2E) is already a good privacy. And even with that setup, the server users got "harassed" by the UI asking them to sync devices keys. For many users, a single prompt about security is enough to discourage them.
- CameronNemo 5y ago
  So you are asking for a client setting (in Element I presume) to stop nudging users to set up e2ee? Did you file an issue for that?
  - kuon 5y ago
    If I try again to "convert" non technical people around me, I'll try to document properly what is wrong in the UX and experience. (but maybe it will all be fixed)

teitoklien 5y ago

Synapse is bloated and mostly good for very large organisations. Use conduit.rs or dendrite. I use conduit and it takes a couple of MB ram even while I’m in large rooms

xvilka 5y ago
Official server should have never been in Python anyway. Not a proper language for the network server. They should have adopted Go or Rust servers as the official ones, and pour resources there instead.
- mst 5y ago
  Synapse was about proving the concept.
  - xvilka 5y ago
    Well, they proved it, time for it to be dropped, and focus on the production-level implementation. But it's still[1] the recommended one. [1] https://matrix.org/docs/guides/introduction#how-can-i-get-my-own-server https://matrix.org/docs/guides/introduction#how-can-i-get-my...
    - mst 5y ago
      I sympathise with your aggravation but they're balancing doing that while not trying to migrate the existing user base too early and keeping supporting people with production workloads. Fundamentally they've 'lucked' into a 'nice problem to have' type situation here, and I get how annoying it is - I was freenode staff during a period where the irc<->matrix bridging was unexpectedly popular enough to be a 'nice problem to have' for everybody involved and the author of the blog post we're discussing probably remembers some sharp words on my part during the process - but frankly on balance I'm mostly impressed.
kuon 5y ago
Last time I checked critical features were not supported, like read receipts.

nisa 5y ago

I'm running a small (at the beginning it was completely public, now I'm keeping an eye on registrations) homeserver with synapse for a few years - I really like Matrix on a conceptual level and I'm happy that it exists even through it's not perfect it works pretty well mostly. I don't want to be too harsh because it's the best decentralized alternative for chat at the moment and it's open-source and there is a lot of progress but damn' it was a pain in the ass to run that server... we solved our problems mostly with throwing resources and memory on the problem... it's a nice tour de force if you want to learn about debugging distributed systems but it's nothing to install somewhere and just keep running. synapse is quite a mess and there are a lot of hidden footguns if you just run a server - this is just the tip of iceberg to keep the postgres database somewhat sane: https://levans.fr/shrink-synapse-database.html https://levans.fr/shrink-synapse-database.html - lot's of other issues in the issue-tracker where you can just scratch your head. bridges are all subtly broken - the xmpp bridge is horrible and broke in so much interesting ways that I'm not going to touch it ever again - telegram works okay most of the time, irc-bridge also have some warts - but it's easy to criticize from my chair and probably unfair to talk so negative about it here but it's often buggy and broken for edge-cases - it works most of the time pretty okay but it's quite a mess to get a mental model for the code and so it's difficult to debug things. moderation/spam/etc.pp is all hackable but not there out of the box - it looks and feels like mostly quickly hacked up nodejs code that at least for us exploded in all kinds of interesting ways. https://github.com/matrix-org/mjolnir https://github.com/matrix-org/mjolnir writing 3tb of logs in a few day and eating memory like crazy for instance. You have to babysit it and there is no simple ui for anything. So it's powerful but requires quite a bit of dedication and patience to get right. It's a full blown distributed system and often state is all over the place and once you make a mistake it's difficult to impossible to get that thing do work correct again without starting over. But there are so much promising projects that I'm confident that these issues will be resolved and it will only get better but in my experience it will break badly on all kinds of edge-cases - the mentioned xmpp-bridge created usernames that can't be deleted via the http api for instance. someone bridged 1000 channels via our telegram-bridge and there is no code to remove those channels - you have to code something up in python for yourself. irc bridge kicks you after 30 days idle because they can't handle the connections - freenode (before the takeover) said it's not them - maybe single threaded nodejs is not such a good idea for that. Could I do it any better and delivering? Probably not. But except some adventure and if you want to deploy it for an org carefully test any assumptions you take for granted. It's cool but it's also kind of quick'n'dirty in a lot of ways. Still better than anything else I'd use it over any megacorp messenger anytime but maybe don't switch your family yet. But for using it you don't have to care - and there are great projects like https://github.com/spantaleev/matrix-docker-ansible-deploy https://github.com/spantaleev/matrix-docker-ansible-deploy that solve most of the problems out of the box and mobile clients and web clients and E2E crypto also works really well.

janikvonrotz 5y ago

mayli 5y ago

Ack, pretty all of those points are true, which is partially due to python.