3 ms·
Nope. The biggest bottleneck is in the async Sidekiq workers. I have 7 Sidekiq processes using a total of 5GB of RAM and as much CPU as they can get, but my dat
by kstrauser 4y ago
Nope. The biggest bottleneck is in the async Sidekiq workers. I have 7 Sidekiq processes using a total of 5GB of RAM and as much CPU as they can get, but my database server is at around 5% CPU and IO.
If the Rails part scaled as well as my DB, I believe I could trivially handle 10x the number of users I have today.
- vidarh 4y agoThat's a Mastodon and Rails architecture issue first and foremost, though, and far less of a Ruby issue. Mastodon is indeed a resource hog, so that part of the complaint I agree with. I've run queue processing on servers a fraction of a modern server CPU, less memory than that total, and with Ruby 1.8.x. It's not hard. It's hard to do with a naive Rails-based design based around Sidekick workers and ActivityRecord.
- kondro 4y agoYou might want to double-check your DB pooling numbers. Running out of DB connections can end up looking a lot like heavy CPU usage on Ruby. This bottleneck probably relates to the delivery of new messages to feeds. That's the busiest part of the backend and requires 5-7 DB requests and a bunch of Redis requests per recipient from a post on your server. I'm sure spending time just on making this part of the platform more efficient would have massive impacts on performance across on a Mastodon instance.
- kstrauser 4y agoThat part’s fine. I’m sitting being PgBouncer, and neither it nor PostgreSQL are anywhere near their connection limits. I sincerely wish that were the issue so I could configure my way around it.
- kondro 4y agoYes, but are the connection pools in Rails & Sidekiq configured for a very large number of connections. You can probably squeeze up to 100+ out of a single process.
- kstrauser 4y agoI’ve got a pool of 100 DB connections for each Sidekiq process, and haven’t gotten any errors related to lack of connections. For real, I’ve troubleshot this to hell and back. Mastodon’s Sidekiq processes eat servers for breakfast.