4 ms·
There is this one perfect illustration that I ended up using as a troubleshooting interview question in multiple places (sorry, it should now be considered burn
by randomswede 5y ago
There is this one perfect illustration that I ended up using as a troubleshooting interview question in multiple places (sorry, it should now be considered burned).
Devs build a reporting framework. They test it. All is well.
Devs deploy reporting framework to production. All reports suddenly time out.
Devs complain to the network team that the network is broken.
Network team spend weeks checking every single interface and link, in all involved network paths (8with multiple daily "we are not seeing any network issues, is the application still having problems?", so definitely not done in complete silence).
Network team eventually pulls out the packet sniffers. Network team then asks if they can see the relevant code snippet.
Root cause is that in the test environment, the RTT between app server and (test) DB was on the order of 0.1 ms. But, in production, the RTT between the app server and the DB server was ~20 ms. And the code generating the report did NOT pipeline any of the data gathering, effectively making the report generation taking ~200 times longer.
Unfortunately, the network team was unable to deploy a higher light speed limit for that network data path and instead suggested that the application fetch multiple data items in parallel.