3 ms·

Great article! We have a Rails app that uses Puma with MySQL, but so far single threaded (the Puma config has `threads 1,1`), because we migrated from Unicorn

by pqdbr 6y ago

Great article! We have a Rails app that uses Puma with MySQL, but so far single threaded (the Puma config has `threads 1,1`), because we migrated from Unicorn and are still worrying about thread-safety. Currently our database.yml pool size is set to 25, because we have 25 puma workers on the server. Reading your article, this seems to be overkill, since that setting is _per worker_, and since each worker is running only with 1 thread, the default 5 should be more than enough? Also, I just checked and our MySQL is configured with 151 max connections; if every worker of ours can use up to 25 connections, that can lead up to 625 connections to the database, and that's without considering Sidekiq (concurrency 25).

sudhirj 6y ago
Well, even the default 5 would probably be overkill, unless you’re spawning threads on a request that each use their own DB connections. But yes, I think you can go with the the default. Once you’re confident about being threadsafe, the Heroku deployment guide for puma is pretty good for all the little knobs to turn.
sudhirj 6y ago
Re MySQL counts, based on your notes you’re probably using 50 conns at max whatever your settings are. 25 for sidekiq (assuming you do run a hot queue) and 25 at max for the servers. Irrespective of DB pool size, I think Rails does lazy connections, so you’re never going to go over 1.
- pqdbr 6y ago
  I changed database.yml to use 10 connections. I didn't change the Sidekiq configuration, which still uses concurrency: 25. Strangely, tonight I've seen the first exceptions like `ActiveRecord::ConnectionTimeoutError: could not obtain a connection from the pool within 5.000 seconds (waited 5.000 seconds); all pooled connections were in use` in many, many years. All of these exceptions came from within Sidekiq Workers, which is strange because, like I pointed, I didn't change the Sidekiq concurrency, only the database.yml pool, which shouldn't affect workers.
  - sudhirj 6y ago
    Isn’t sidekiq working off Redis? You might be adjusting the Redis concurrency instead. And if I remember correctly even with sidekiq you’re still using normal Rails and ActiveRecord, so a process that’s running 25 sidekiq jobs simultaneously is going to need 25 active record DB conns from the database.yml. I doubt the sidekiq config is overriding it in some way. But basically if your sidekiq workers and app servers are using the same database.yml values, you’ll need to set the greater of the two on both. I assumed you have different settings on app servers and workers.

by pqdbr 6y ago

sudhirj 6y ago

Well, even the default 5 would probably be overkill, unless you’re spawning threads on a request that each use their own DB connections. But yes, I think you can go with the the default. Once you’re confident about being threadsafe, the Heroku deployment guide for puma is pretty good for all the little knobs to turn.

sudhirj 6y ago

Re MySQL counts, based on your notes you’re probably using 50 conns at max whatever your settings are. 25 for sidekiq (assuming you do run a hot queue) and 25 at max for the servers. Irrespective of DB pool size, I think Rails does lazy connections, so you’re never going to go over 1.

pqdbr 6y ago
I changed database.yml to use 10 connections. I didn't change the Sidekiq configuration, which still uses concurrency: 25. Strangely, tonight I've seen the first exceptions like `ActiveRecord::ConnectionTimeoutError: could not obtain a connection from the pool within 5.000 seconds (waited 5.000 seconds); all pooled connections were in use` in many, many years. All of these exceptions came from within Sidekiq Workers, which is strange because, like I pointed, I didn't change the Sidekiq concurrency, only the database.yml pool, which shouldn't affect workers.
- sudhirj 6y ago
  Isn’t sidekiq working off Redis? You might be adjusting the Redis concurrency instead. And if I remember correctly even with sidekiq you’re still using normal Rails and ActiveRecord, so a process that’s running 25 sidekiq jobs simultaneously is going to need 25 active record DB conns from the database.yml. I doubt the sidekiq config is overriding it in some way. But basically if your sidekiq workers and app servers are using the same database.yml values, you’ll need to set the greater of the two on both. I assumed you have different settings on app servers and workers.