4 ms·

One thing I didn't like about Matrix: I had a single server with 3 or 4 other users. In a few months, the database ballooned to, like, 80GB. User-uploaded conte

One thing I didn't like about Matrix: I had a single server with 3 or 4 other users. In a few months, the database ballooned to, like, 80GB. User-uploaded content, including text, was in the ballpark of 2 or 3GB. Is there any way to clear out the cache of data received from old servers, say, anything more than 2 weeks old, and re-retrieve it if needed?

COGlory 3y ago
Yes, it's room/concersation states taking up all the space, and you can compress them. https://levans.fr/shrink-synapse-database.html https://levans.fr/shrink-synapse-database.html https://github.com/matrix-org/rust-synapse-compress-state https://github.com/matrix-org/rust-synapse-compress-state
- lightedman 3y ago
  That it generates that much cruft for something so simple is disappointing. How long does the compression bandaid work for?
  - COGlory 3y ago
    From the first link: >I shrunk my homeserver database from 100GB to a little under 8GB, during a long maintenance cleanup. I run the compressor on a cron job. I've been running for 5 years with ~ a dozen people on my server, all federated, in multiple large rooms, didn't run into space issues until this year. On Linode $10 and then Hetzner 4 CPU. One thing to watch out for: disable registration. Otherwise spambots will register accounts, join the largest matrix.org rooms to spam, and then you'll have to store all that crap for all those rooms.
    - jeltz 3y ago
      I feel registration is something which should come disabled by default and need to be explicitly enabled.
      - anoa_ 3y ago
        [dead]
      - nani8ot 3y ago
        At least on Conduit and Dendrite that is the case iirc, but it's easy to forget to disable it once you register an account.
      - COGlory 3y ago
        It may be. I setup my server half a decade ago and don't remember exactly how I configured it initially. I may have enabled it not realizing the implications.
toastal 3y ago
This is one of those things that make me appreciate XMPP. There’s no expectation to duplicate the entire history of chatrooms. Many MUCs default to the last 20 messages, which is usually enough to let you catch & jump in on the conversation. Costs of running my server have been quite cheap.
wjbolles 3y ago
Yeah, there are admin APIs that can clear old images, local and remote ones. For your specific scenario you probably want this one: https://matrix-org.github.io/synapse/v1.85/admin_api/media_admin_api.html#purge-remote-media-api https://matrix-org.github.io/synapse/v1.85/admin_api/media_a... My server is not federated but fairly active, and we treat it as ephemeral. We've configured it so anything older than a week or two gets reclaimed automatically, text and media.
- CameronNemo 3y ago
  Are you using synapse?

COGlory 3y ago

Yes, it's room/concersation states taking up all the space, and you can compress them. https://levans.fr/shrink-synapse-database.html https://levans.fr/shrink-synapse-database.html https://github.com/matrix-org/rust-synapse-compress-state https://github.com/matrix-org/rust-synapse-compress-state

lightedman 3y ago
That it generates that much cruft for something so simple is disappointing. How long does the compression bandaid work for?
- COGlory 3y ago
  From the first link: >I shrunk my homeserver database from 100GB to a little under 8GB, during a long maintenance cleanup. I run the compressor on a cron job. I've been running for 5 years with ~ a dozen people on my server, all federated, in multiple large rooms, didn't run into space issues until this year. On Linode $10 and then Hetzner 4 CPU. One thing to watch out for: disable registration. Otherwise spambots will register accounts, join the largest matrix.org rooms to spam, and then you'll have to store all that crap for all those rooms.
  - jeltz 3y ago
    I feel registration is something which should come disabled by default and need to be explicitly enabled.
    - anoa_ 3y ago
      [dead]
    - nani8ot 3y ago
      At least on Conduit and Dendrite that is the case iirc, but it's easy to forget to disable it once you register an account.
    - COGlory 3y ago
      It may be. I setup my server half a decade ago and don't remember exactly how I configured it initially. I may have enabled it not realizing the implications.

toastal 3y ago

This is one of those things that make me appreciate XMPP. There’s no expectation to duplicate the entire history of chatrooms. Many MUCs default to the last 20 messages, which is usually enough to let you catch & jump in on the conversation. Costs of running my server have been quite cheap.

wjbolles 3y ago

Yeah, there are admin APIs that can clear old images, local and remote ones. For your specific scenario you probably want this one: https://matrix-org.github.io/synapse/v1.85/admin_api/media_admin_api.html#purge-remote-media-api https://matrix-org.github.io/synapse/v1.85/admin_api/media_a... My server is not federated but fairly active, and we treat it as ephemeral. We've configured it so anything older than a week or two gets reclaimed automatically, text and media.

CameronNemo 3y ago
Are you using synapse?