3 ms·
Most scaling issues are due to the software, what’s newsworthy about this? Gitea, Gogs, GitLab, name a scm, have all had their fair share of growing pains. If y
by wutwutwat 3y ago
Most scaling issues are due to the software, what’s newsworthy about this? Gitea, Gogs, GitLab, name a scm, have all had their fair share of growing pains. If you run a SaaS, scaling issues are par for the course. They are also a good thing. Failing endeavors don’t have scaling issues.
You can likely squeeze some extra performance out of your database by:
slimming down large tables with old data. Move it to an archive table to reduce the index sizes
looking for missing indexes on hot tables, or inefficient ones. Combining multiple indexes into one can help too
Running vacuum full
checking and possibly increasing work_mem, and shared buffers. Enable pg_stat log temp files and look at how often you’re writing to disk due to worker memory exhaustion on complex queries and joins. You can set the logger to log above a certain size and adjust work mem to cover most hot queries. Don’t try to get them all, that’s a losing game and the gains are not worth it for seldom used queries. Just let them hit disk.
Take a look at your max connections and see if you’re at the limit, and increase or look into a connection pooler if so
Think about splitting reads and writes. This can be done in the code, or, using a pooler (pg bouncer I think or pg pooler). Send your writes to a primary and reads to n number of replicas following the primary. Throw replicas at it load balance reads until you optimize the queries.
First thing I would do though, throw beefier hardware at the db. If using managed service like aurora/rds, work mem, shared buffers, and max conns are proportional to instance size. More memory on a larger instance could get you stable again. I guarantee you, you’ve not reached as far as you can go on a single db server, you just have to pony up the dough
> Just adding: The performance problem is also worsened by the fact that we are hit with spam waves for several weeks now. Bots create thousands of issues on Codeberg continuously, and although we got more efficient with cleanign it up, it heavily stresses the database.
Sounds like you need to start blocking ip ranges instead of optimizing spam cleanup. If you can, put a WAF infront of the app servers. Throw captchas on forms that are spammed for suspicious ips. Implement rate limiting for issue creation and if it’s already there, make it more aggressive. People don’t make multiple issues on a repo in an entire day usually