4 ms·

Totally. I've seen similar things. I've also seen thread contention (such as on a connection pool) that can look a lot like a slow database query/ an "IO bound"

Totally. I've seen similar things. I've also seen thread contention (such as on a connection pool) that can look a lot like a slow database query/ an "IO bound" workflow. I think profiling is just really hard and lots of code tends to be very inefficient at actually performing IO.

charrondev 6y ago
I spent my last 2 weeks optimizing a decade old SaaS to handle massive traffic spikes from one of our biggest customers. We had other customers serving similar amounts of traffic, but with smaller data sets. - Increasing the numbers of servers running the app. App connections were still stacking up though. This gave us more breathing room though for connections to start stacking and handle small spikes. - The database seemed very overloaded with so many concurrent connections. I began putting everything I could into memcached (we already had a lot of data in it, but I put more). - now we had a cache hotspot. Some digging found a age old bug in our cache driver where it didn’t actually keep things in process memory after fetching from memcached and we had a medium sized key getting fetch 100s of times per request. - Days and days of app optimization after profiling. Our average response time went improved by more than 50%. The site would still start collapsing under a little load. - While profiling in a single request all queries would complete very quickly (<50ms). Somehow the DB was still the bottleneck. We overprovisoned it significantly and it still would collapse. - I started collecting counts and timings for cumulative and maximum single cache/db, read/writes to our log stack. - the bottleneck was clearly still the DB. - at this point we were desperate. Thinking it might have been an issue in the underlying VM we live migrated the DB to a new VM. - the database was still the bottleneck. In the end the thing that fixed it? A simple OPTIMIZE TABLE. Somehow ANALYZE TABLE hadn’t detected anything but rebuilding the table still fixed the issue. If anyone is looking for a good load testing tool, Vegeta was invaluable. I highly recommend it.
unilynx 6y ago
These are hinted by high system% usage when your system is busy (ie higher than say 10%). If it looks cpu bound but spends a lot of time in the kernel thread switching or synchronization (eg mutexes) is happening too much.

charrondev 6y ago

I spent my last 2 weeks optimizing a decade old SaaS to handle massive traffic spikes from one of our biggest customers. We had other customers serving similar amounts of traffic, but with smaller data sets. - Increasing the numbers of servers running the app. App connections were still stacking up though. This gave us more breathing room though for connections to start stacking and handle small spikes. - The database seemed very overloaded with so many concurrent connections. I began putting everything I could into memcached (we already had a lot of data in it, but I put more). - now we had a cache hotspot. Some digging found a age old bug in our cache driver where it didn’t actually keep things in process memory after fetching from memcached and we had a medium sized key getting fetch 100s of times per request. - Days and days of app optimization after profiling. Our average response time went improved by more than 50%. The site would still start collapsing under a little load. - While profiling in a single request all queries would complete very quickly (<50ms). Somehow the DB was still the bottleneck. We overprovisoned it significantly and it still would collapse. - I started collecting counts and timings for cumulative and maximum single cache/db, read/writes to our log stack. - the bottleneck was clearly still the DB. - at this point we were desperate. Thinking it might have been an issue in the underlying VM we live migrated the DB to a new VM. - the database was still the bottleneck. In the end the thing that fixed it? A simple OPTIMIZE TABLE. Somehow ANALYZE TABLE hadn’t detected anything but rebuilding the table still fixed the issue. If anyone is looking for a good load testing tool, Vegeta was invaluable. I highly recommend it.

unilynx 6y ago

These are hinted by high system% usage when your system is busy (ie higher than say 10%). If it looks cpu bound but spends a lot of time in the kernel thread switching or synchronization (eg mutexes) is happening too much.