brendangregg

10 ms·

61.
by brendangregg 5y ago
Not added, but maybe I'll add something extra to make it more clear. I have a backlog of interesting case studies I'm trying to find time to share.
62.
by brendangregg 5y ago
Ah, thanks for the details, I wasn't referring to NUMA-induced lock contention, but rather lock contention in general. I've seen a workload hit 64 CPUs with 80% of CPU time in lock contention (and others in the 10-30%). Now, while
63.
by brendangregg 5y ago
Yes, this was in my predictions: Processor vendors may offer their own clouds (either built or acquired) to get the same level of workload access.
64.
by brendangregg 5y ago
I don't have a workload I'd prefer to see on 2x128-core: We're already microservices running across a pool of instances, and would prefer a bigger pool of faster instances than a smaller pool of slower ones at the same cost.
65.
by brendangregg 5y ago
Sure, and I have regular meetings with processor vendors who want to understand our workload to better serve it. But there's no easy or fast way for them to get low-level data, including processor trace (cycle logs), across many custom
66.
by brendangregg 5y ago
Thanks, I think it's the start of a new "cloud CPU" era. Cloud vendors have already split workloads into different instance types, so directly analyzing their workloads and developing CPUs for each instance type will lead to
67.
by brendangregg 5y ago
> Is there any examples of open source projects that died because people were making too many ports? Yes, Linux tracing. I entered in 2014 when there were 14 different tracers, none of them finished (ie, does everything and works on all
68.
by brendangregg 5y ago
Unfortunately that description is so old it's misleading. I described BPF in another comment here.
69.
by brendangregg 5y ago
Yes, BTF provides the kernel struct info in the vmlinux file. Like debuginfo, but far more lightweight. Of course, tracepoints are better, as they are stable and don't need BTF (although raw_tracepoints do), but there's often case
70.
by brendangregg 5y ago
Thanks; I also summarize the history of eBPF in my BPF Performance Tools book (which I had checked by Alexei & Dan) and list the main contributors in the Acknowledgements page, as well as include them in talks (e.g. https://w
71.
by brendangregg 5y ago
Sorry, I just added a link to https://ebpf.io . Although I suspect that will also annoy people as it doesn't explain the BPF acronym: The reason is that we're now treating BPF as a technology name (like LLVM) and not an
72.
by brendangregg 5y ago
I've seen blatant cases as well. There was a github repo where someone was sharing their notes on performance engineering, and there were tickets from people thanking them for their amazing work. Except these notes were literally copy-
73.
by brendangregg 5y ago
Disbelief and suspicion. And fear of my mental health I guess: What's wrong with this person? Maybe I just don't look or dress or sound like what one would expect. But there's context here too: At the time it's when thes
74.
by brendangregg 5y ago
The formal interface is tracepoints. So tracepoints in theory aren't brittle (they are best-effort stable) and don't need so much expert maintenance (which is mostly the case). In theory, someone could port tracepoint-based tools
75.
by brendangregg 5y ago
Nope, never heard back again. I'm not too surprised, Sun was a big company.
76.
by brendangregg 5y ago
No, it wasn't, I don't even remember the VIP's name. Met him that once and never heard of him again.
77.
by brendangregg 5y ago
Thanks. There was a time when many observability products were adding latency heat maps, and at one conference expo floor there were three companies with latency heat maps on their screen at the same time, pitching them as a flagship featur
78.
by brendangregg 5y ago
I appreciate the discussion as I want to do a follow on post about this in more detail. It doesn't do the BPF community any good to have people running old broken versions of my tools (just as back in 2005, Sun was selling DTrace by di
79.
by brendangregg 5y ago
In the article I included my guess about real cause for this: Sun's assumption that any good work had to be from a Sun employee. I'd guess the sequence of events was: - DTrace is the new hotness, we need it in our UI. - Ever
80.
by brendangregg 5y ago
Sorry, should have said "tech" employer, and this is referring to Sun and its ecosystem. A few years earlier I gave a talk to a local University about the job market and asked the comp sci students to have a showing of hands as to
81.
by brendangregg 5y ago
Just a couple of example scenarios: - Someone sells you BPF observability that's buggy and dissapointing, and you avoid BPF in the future. So in that case it's hurt the BPF community. - Funding goes to a project that gives nothing
82.
by brendangregg 5y ago
Yes, there can be good reasons to fork (especially after making a fair effort to have things fixed), and bad reasons. But also yes: There are two particular issues with bpf tooling forks. 1) They look deceptively simple, but those that are
83.
by brendangregg 5y ago
I never got a chance to see it again. I ended up spending more energy with another Sun project that wanted to switch the CDDL license to the GPL, only because they wanted to avoid going through the legal approval procedure to ship mixed-lic
84.
by brendangregg 5y ago
I was in shock, and a bit furious at being ripped off, but didn't feel there was a lot I could do about it. I edited this bit out (I was trimming the post): "You might wonder why I didn't talk about this first case publicly a
85.
by brendangregg 5y ago
In the SF Bay area there are a number of cricket leagues. We played in the GPL (Google Premiership League) but actually spent more time practicing. There's also cricbay, which has over 110 teams in the bay area. 110. Visit baseball dia
86.
by brendangregg 5y ago
Yes, moving to Byron for Australia is similar to Hawaii/Maui for California.
87.
by brendangregg 5y ago
It's true that house prices in Australia, especially Sydney, are so high right now they are on par with the Bay Area. Other financial details are harder to calculate: Your lifetime medical costs, for example. And also a lack of bonuses
88.
by brendangregg 5y ago
Sydney is currently deemed a bit lower, but not by huge amount. (So I'm not sharing something I don't know if I can, let me say that someone _else_ in a very similar situation got a 20% cut going to Sydney.) And I think that won&#
89.
by brendangregg 5y ago
Right, it is. It fixed my hot-path syscalls on x86 (via read/write functions, pthread_mutex functions, etc.). But if you have syscalls called via asm outside of libc (by who?) then they need frame pointers as well.
90.
by brendangregg 5y ago
Great work, thanks for sharing! Systems performance at its best. Nice to see the use of the custom palette.map (I forget to do that myself and I often end up hacking in highlights in the Perl code.) BTW, those disconnected kernel stacks can

More ›