3 ms·

How did you generate those flamegraphs and what other tools did you use to measure performance? My motivation for asking comes from these findings in the pdf,

How did you generate those flamegraphs and what other tools did you use to measure performance? My motivation for asking comes from these findings in the pdf, Did the graph show the bottleneck contention on aio queue? Did the graph show that "a lot of time was spent accessing memory"? What made freebsd a better platform compared to Linux to begin tackling this problem? Thanks! Super interesting. Both a freebsd fan and I have workloads that I'd love to explore benchmarking to squeeze more performance.

drewg123 4y ago
> How did you generate those flamegraphs and what other tools did you use to measure performance? We have an internal shell script that takes hwpmc output and generates flamegraphs from the stacks. It also works with dtrace. I'm a huge fan of dtrace. I also make heavy use of lockstat, AMD uProf, and Intel Vtune. > Did the graph show the bottleneck contention on aio queue? Did the graph show that "a lot of time was spent accessing memory"? See the graph on page 32 or so of the presentation. It shows huge plateaus in lock_delay called out of the aio code. Its also obvious from lockstat stacks (run as lockstat -x aggsize=4m -s 10 sleep 10 > results.txt) See the graph on page 38 or so. The plateaus are mostly memory copy functions (memcpy, copyin, copyout). We already use FreeBSD on our CDN, so it just made sense to do the work in FreeBSD. The talk is on Youtube https://youtu.be/36qZYL5RlgY https://youtu.be/36qZYL5RlgY
smokel 4y ago
The flame graphs might be generated using Brendan Gregg's utility, see https://www.brendangregg.com/flamegraphs.html https://www.brendangregg.com/flamegraphs.html
- drewg123 4y ago
  They are generated by a local shell script that uses the same helpers (stackcollapse*.pl, difffolded.pl). Our revision control says the script was committed by somebody else though. It existed before I joined Netflix.

drewg123 4y ago

> How did you generate those flamegraphs and what other tools did you use to measure performance? We have an internal shell script that takes hwpmc output and generates flamegraphs from the stacks. It also works with dtrace. I'm a huge fan of dtrace. I also make heavy use of lockstat, AMD uProf, and Intel Vtune. > Did the graph show the bottleneck contention on aio queue? Did the graph show that "a lot of time was spent accessing memory"? See the graph on page 32 or so of the presentation. It shows huge plateaus in lock_delay called out of the aio code. Its also obvious from lockstat stacks (run as lockstat -x aggsize=4m -s 10 sleep 10 > results.txt) See the graph on page 38 or so. The plateaus are mostly memory copy functions (memcpy, copyin, copyout). We already use FreeBSD on our CDN, so it just made sense to do the work in FreeBSD. The talk is on Youtube https://youtu.be/36qZYL5RlgY https://youtu.be/36qZYL5RlgY

smokel 4y ago

The flame graphs might be generated using Brendan Gregg's utility, see https://www.brendangregg.com/flamegraphs.html https://www.brendangregg.com/flamegraphs.html

drewg123 4y ago
They are generated by a local shell script that uses the same helpers (stackcollapse*.pl, difffolded.pl). Our revision control says the script was committed by somebody else though. It existed before I joined Netflix.