5 ms·

The matrix.org home server is spectacularly overloaded right now, causing messages to take several seconds to be delivered at a minimum. It isn't terribly surpr

by subway 9y ago

The matrix.org home server is spectacularly overloaded right now, causing messages to take several seconds to be delivered at a minimum. It isn't terribly surprising considering my own federated home server with under 20 users sits around 4gb of memory utilization with a single core (the server is single process/single threaded by default) pegged. There's some work that can be done to proxy some of the load off into separate workers, but Synapse is pretty darn awkward to work with.

16 9y ago
Have you tried leaving any giant rooms you may be in? Like Riot/Matrix HQ? I use my personal homeserver for my friends/some IRC channels and my load is nonexistent. May not be the answer you're looking for, but those huge (10,000+ users) rooms are a huge strain on homeservers right now. Look for performance updates in the next version of synapse.
- subway 9y ago
  Yup -- that's exactly what the end solution was. Unfortunately reaching out to those 20 users and saying "hey, could you be judicious about which federated rooms you join? otherwise you're gonna hose us all" kinda sucks.
derefr 9y ago
I've been considering writing a Synapse-equivalent Matrix server in Elixir, if there was enough interest.
- dest 9y ago
  FYI there have been efforts to rewrite an equivalent in Golang: https://github.com/matrix-org/bullettime https://github.com/matrix-org/bullettime
  - voxadam 9y ago
    Dendrite seems to be a much more active Matrix server written in Go. https://github.com/matrix-org/dendrite https://github.com/matrix-org/dendrite
  - deleted 9y ago
    [deleted]
- stevenicr 9y ago
  from the bit I learned at a phoenix/elixer meetup - it seemed that it may be the most well suited for multiple chat streams. I am really interested in this.
tomc1985 9y ago
> 20 users sits around 4gb dafuq? Are you serious? What could possibly need 4gb?
- Arathorn 9y ago
  The problem isn't the number of connected or registered users on the server - the problem is the complexity of the rooms they are participating in. So if those 20 users join hundreds of rooms with tens of thousands of users participating, spread across thousands of servers, with large numbers of joins/parts or messages sent a second... you shouldn't be surprised that it takes a few GB of RAM. HOWEVER, we have been working incredibly hard at optimisation over the last few months, and we have order-of-magnitude algorithmic performance improvements on the horizon over the coming weeks. https://twitter.com/matrixdotorg/status/979049079568240641 https://twitter.com/matrixdotorg/status/979049079568240641 etc. There's also a port to python3 happening which should improve RAM by 2-3x given python3 stores strings in utf8 rather than python2's utf32.
  - subway 9y ago
    I certainly don't mean to throw any shade on the project -- if anything, the RAM utilization was less of a surprise for me than being bound to single-core performance by default. Sure, it's possible to split Synapse up into workers, but it definitely feels like there's some voodoo involved (my first attempt at setting just a Synchroton worker failed miserably this week)
  - tomc1985 9y ago
    Is it really necessary to maintain the state of the entire network on all federated nodes? (Since I'm guessing that's what you've described)
    - Arathorn 9y ago
      It isn't necessary, and we don't do it. What I've described is what happens if you type '/join #matrix:matrix.org' and your server starts participating in a large room which spans tens of thousands of users and hundreds of servers. Rooms are only replicated over the servers which participate in them. So if you don't join massive rooms, your server can be have fine. (And in the near future massive rooms should be usable too).
tacoman 9y ago
I have a single non-federated synapse server with 10 users and 25 client devices, and the python process is using 135MB (RES in top). Load average is 0.00. There's 2 main rooms and probably around 10 chats. The homeserver.db is 26MB.

by subway 9y ago

16 9y ago

Have you tried leaving any giant rooms you may be in? Like Riot/Matrix HQ? I use my personal homeserver for my friends/some IRC channels and my load is nonexistent. May not be the answer you're looking for, but those huge (10,000+ users) rooms are a huge strain on homeservers right now. Look for performance updates in the next version of synapse.

subway 9y ago
Yup -- that's exactly what the end solution was. Unfortunately reaching out to those 20 users and saying "hey, could you be judicious about which federated rooms you join? otherwise you're gonna hose us all" kinda sucks.

derefr 9y ago

I've been considering writing a Synapse-equivalent Matrix server in Elixir, if there was enough interest.

dest 9y ago
FYI there have been efforts to rewrite an equivalent in Golang: https://github.com/matrix-org/bullettime https://github.com/matrix-org/bullettime
- voxadam 9y ago
  Dendrite seems to be a much more active Matrix server written in Go. https://github.com/matrix-org/dendrite https://github.com/matrix-org/dendrite
- deleted 9y ago
  [deleted]
stevenicr 9y ago
from the bit I learned at a phoenix/elixer meetup - it seemed that it may be the most well suited for multiple chat streams. I am really interested in this.

tomc1985 9y ago

> 20 users sits around 4gb dafuq? Are you serious? What could possibly need 4gb?

Arathorn 9y ago
The problem isn't the number of connected or registered users on the server - the problem is the complexity of the rooms they are participating in. So if those 20 users join hundreds of rooms with tens of thousands of users participating, spread across thousands of servers, with large numbers of joins/parts or messages sent a second... you shouldn't be surprised that it takes a few GB of RAM. HOWEVER, we have been working incredibly hard at optimisation over the last few months, and we have order-of-magnitude algorithmic performance improvements on the horizon over the coming weeks. https://twitter.com/matrixdotorg/status/979049079568240641 https://twitter.com/matrixdotorg/status/979049079568240641 etc. There's also a port to python3 happening which should improve RAM by 2-3x given python3 stores strings in utf8 rather than python2's utf32.
- subway 9y ago
  I certainly don't mean to throw any shade on the project -- if anything, the RAM utilization was less of a surprise for me than being bound to single-core performance by default. Sure, it's possible to split Synapse up into workers, but it definitely feels like there's some voodoo involved (my first attempt at setting just a Synchroton worker failed miserably this week)
- tomc1985 9y ago
  Is it really necessary to maintain the state of the entire network on all federated nodes? (Since I'm guessing that's what you've described)
  - Arathorn 9y ago
    It isn't necessary, and we don't do it. What I've described is what happens if you type '/join #matrix:matrix.org' and your server starts participating in a large room which spans tens of thousands of users and hundreds of servers. Rooms are only replicated over the servers which participate in them. So if you don't join massive rooms, your server can be have fine. (And in the near future massive rooms should be usable too).

tacoman 9y ago

I have a single non-federated synapse server with 10 users and 25 client devices, and the python process is using 135MB (RES in top). Load average is 0.00. There's 2 main rooms and probably around 10 chats. The homeserver.db is 26MB.