At GitLab we've built and used strace-parser to help us get to quick and deep debugging, but running strace is EXPENSIVE: https://gitlab.com/gitlab-com/support/

At GitLab we've built and used strace-parser to help us get to quick and deep debugging, but running strace is EXPENSIVE: https://gitlab.com/gitlab-com/support/toolbox/strace-parser https://gitlab.com/gitlab-com/support/toolbox/strace-parser We've been eagerly awaiting some customers to adopt newer kernels so we can start leveraging eBPF because of the performance gains in these type of scenarios. Getting down the the kernel often can help find problems with disk access or network issues. One of our staff engineers is exploring it now for NFS stats: https://gitlab.com/wchandler/tracing-tools/-/blob/master/nfsqueue https://gitlab.com/wchandler/tracing-tools/-/blob/master/nfs... In Support Engineering we often straddle the line of 'SRE style stare at graphs and configuration as code' and 'log on to the box and look at syscalls'. We are very very excited about eBPF.