brancz

9 ms·

31.
by brancz 3y ago
Right, but judging by the detail of the post, and not mentioning profiling makes it seem like it wasn’t even attempted.
32.
by brancz 3y ago
I’m incredibly confused why profiling wasn’t even mentioned in this post. I would highly recommend anyone troubleshooting something like this to do continuous profiling and the root cause of this type of problem becomes painfully obvious wi
33.
by brancz 3y ago
We’ll be hanging out in comments, let us know if you have any questions!
34.
Continuous Memory Profiling for Rust(polarsignals.com)
5 points by brancz 3y ago|1 comments
35.
by brancz 3y ago
We've "fixed"[1] all the tooling that we work on at Polar Signals, and even then I think this is the right thing to do. It enables so much more debuggability aside from profiling. [1] https://www.polarsignals.com&#
36.
by brancz 3y ago
With the combination of eBPF and DWARF-based it's not quite that simple, there's a pretty elaborate dance between user-space and kernel-space that needs to happen for this to work. With frame-pointers it's walking a linked li
37.
by brancz 3y ago
I would need to double check with the team on this detail, but if I recall correctly the architecture as it is is specifically designed to make the bpf verfier happy and we didn’t think it was going to be possible with existing formats. But
38.
by brancz 3y ago
A lot of profilers have, it’s literally what unwind information is for, but we did it in kernel therefore not the entire stack needs to be copied to user space so it’s way less overhead.
39.
by brancz 3y ago
Ubuntu hosts a debuginfod server that you can automatically discover debuginfo from any binary from using the binary’s build id. https://sourceware.org/elfutils/Debuginfod.html
40.
by brancz 3y ago
We’ve also figured out an alternative format to use from within eBPF to unwind stacks (we happen to only support dwarf at the moment but theoretically any source information could work): https://www.polarsignals.com/blog&#x2
41.
by brancz 3y ago
Correct and there are already known exceptions such as the python interpreter in which the “interpret function” function actually falls into this case and so for the foreseeable future python is going to continue to be compiled omitting fra
42.
by brancz 3y ago
Hyperscalers have long been doing infrastructure-wide profiling (or "Google-Wide Profiling" as the first whitepaper on the topic calls it [1]). This tech allows Google to reduce infra-resource usage by multiple percentage points p
43.
by brancz 3y ago
Profiling tools have already solved the ability to reliably unwind in the absence of frame pointers[1], but there are plenty of tools that this kind of investment is simply too much that it won't ever happen, like bpftrace or bcc-tools
44.
by brancz 3y ago
Founder of Polar Signals here, the profiling product that's mentioned in the blog post. We've already made it possible to unwind without frame pointers relatively cheaply (<1% overhead), and have written about this extensively
45.
by brancz 3y ago
I would argue that LTS releases are going to be deployed millions of times and stay around effectively forever with lots of very critical software being deployed on it. Having all processes/binaries be debuggable cheaply and easily in
46.
by brancz 3y ago
While profiling benefits immensely, imagine all the other ramifications. Cheap tracing of MySQL, postgres, or anything else using bpftrace or any other bcc tools. Getting stack traces of processes that are already core dumping... Profiling
47.
by brancz 3y ago
We wrote pretty extensively about what was needed to be able to profile things without frame pointers [1]. It's still possible at less than 1% overhead with the right set of technologies, but frame pointers unwinding is virtually free.
48.
by brancz 3y ago
You're not wrong, but isn't it exciting that it's gaining more traction? Ubuntu is also very widely deployed, so I'll celebrate every distro that makes this the default, be it Ubuntu in this case, or Debian, or Arch, or
49.
by brancz 3y ago
100% the way I see it! On 32-bit the performance benefits is very major on just about everything, but not so on 64-bit.
50.
by brancz 3y ago
Frame pointers are such a destructive micro-optimization to omit by default, I am beyond excited about this collaboration with the folks at Canonical to make Ubuntu debuggable by default!
51.
The Cost of Go's Interfaces and How to Fix It(polarsignals.com)
8 points by brancz 3y ago|0 comments
52.
by brancz 3y ago
Same with profilers. We had to go through a lot of trouble to get unwinding without frame pointers to work on the eBPF-based profiler we work on [1], and even crazier for interpreted languages [2]. [1] https://www.polarsignals.co
53.
by brancz 3y ago
I happen to work on a system-wide profiler[1] that is eBPF-based, and what we do is load our bpf programs on various kernels[2] using qemu. This is important as the jit and verifier change significantly across kernel versions and we want to
54.
by brancz 3y ago
I just spent ~1.5 years of my life on various aspects of debuginfos and that even just for profiling where I only need line/column information. So I feel this comment through and through.
55.
by brancz 3y ago
You're 100% correct, but for our users it sure seems like a bug.
56.
by brancz 3y ago
I happen to work on profiling tools where we rely on compilers to output metadata about the machine executable code correctly . It turns out this is surprisingly hard and goes wrong quite often, and sometimes it's even impossible to m
57.
by brancz 3y ago
DWARF is touring complete and not the accidental kind, it has instructions and all. GDB implements the whole set so you can basically make GDB do arbitrary code execution if you control the DWARF data.
58.
by brancz 3y ago
I was architect leading all things Observability at Red Hat until 3 years ago there was an absurd amount of support for this project internally I never understood it either. But there were huge amounts of customer support, sales and enginee
59.
by brancz 3y ago
Thanks for the shout out! The team has done a phenomenal job. If you enjoyed that post, I think you’ll also like the one we wrote about native unwinding without frame pointers: https://www.polarsignals.com/blog/posts&#x
60.
by brancz 3y ago
Yeah that’s basically how this system is built as well. We buffer a certain amount of data in the ingestion nodes and then offload it to object storage. In object storage it’s actually laid out in the parquet format so we could allow users

More ›