5 ms·

Strobelight: A profiling service built on open source technology

177 points by birdculture 2y ago

saganus 2y ago
Ah, this is performance profiling. Seeing the title and the domain I thought this was user profiling and I was wondering why would Meta be publishing this.
- hunter2_ 2y ago
  > the domain Perhaps a contributing factor is how HN shows only the final non-eTLD [0] label of the domain. If it showed all labels, you'd have seen "engineering.fb.com" which, while not a dead giveaway, implies that the problem space is technical. It would be nice if this aggressive truncation were applied only above a certain threshold of length. [0] https://en.wikipedia.org/wiki/Public_Suffix_List https://en.wikipedia.org/wiki/Public_Suffix_List
  - teddyh 2y ago
    I suggested this 10 years ago. <https://news.ycombinator.com/item?id=8911044 https://news.ycombinator.com/item?id=8911044>
    - hunter2_ 2y ago
      We are actually saying different things, and your point highlights an error in mine (i.e., I assumed they show the eTLD from the PSL plus one extra label, but apparently they have their own shadow PSL which omits things like pp.se and therefore occasionally shows nothing but an eTLD?) but either way we agree that showing more would be better.
doctorhandshake 2y ago
I would assume the name is a reference to the use of strobes in examining high speed periodic motion, like that in motors or on production lines, eg: https://www.checkline.com/inspection_stroboscope https://www.checkline.com/inspection_stroboscope
arnath 2y ago
This is really cool! I've always thought that one thing preventing major competitors to AWS/Azure/GCP is the lack of easy-to-use tooling for machine level monitoring like this. When I was at Microsoft, we built a tool like this that used Windows Firewall filters to track all the network traffic between our services and it was incredibly useful for debugging. That said, as with anything from Meta, I approach this with a grain of salt and the fact that I can't tell what they stand to gain from this makes me suspicious.
- theptip 2y ago
  > the fact that I can't tell what they stand to gain from this makes me suspicious. Meta is one of the biggest contributors to FOSS in the world. (React, PyTorch, Llama, …). They stand to gain what every big company does, a community contributing to their infra. You’ll note that nobody is open sourcing their ad recommender, that is the one you should be skeptical about if you ever see. You don’t share your secret sauce.
  - schmorptron 2y ago
    ByteDance shared the TikTok content recommender, which I'd argue is somewhat close to an ad recommender :)
    - ipsum2 2y ago
      You mean the paper, not the source code?
  - paxys 2y ago
    Plus it helps them recruit engineers who are already familiar with their tech stack.
  - ipsum2 2y ago
    > You’ll note that nobody is open sourcing their ad recommender Actually... (2019) https://ai.meta.com/blog/dlrm-an-advanced-open-source-deep-learning-recommendation-model/ https://ai.meta.com/blog/dlrm-an-advanced-open-source-deep-l... Source code: https://github.com/facebookresearch/dlrm https://github.com/facebookresearch/dlrm Paper: https://arxiv.org/abs/1906.00091 https://arxiv.org/abs/1906.00091 Updated 2023 blog post, but solely for content recommendation, but ads recommendation is ~90% the same: https://engineering.fb.com/2023/08/09/ml-applications/scaling-instagram-explore-recommendations-system/ https://engineering.fb.com/2023/08/09/ml-applications/scalin... It's a little out of date, but the internal one is built with the same concepts, just more advanced modeling techniques and data.
Thaxll 2y ago
I recommand https://grafana.com/oss/pyroscope/ https://grafana.com/oss/pyroscope/ for continous profiling, I use it in Go and it works well. They have support for many languages https://grafana.com/docs/pyroscope/latest/configure-client/language-sdks/ https://grafana.com/docs/pyroscope/latest/configure-client/l... ( also based on eBPF ).
- outerspace 2y ago
  Good to know there's an OSS alternative.
  - hassleblad23 2y ago
    Strobelight is open source as well.
  - tdullien 2y ago
    The Otel profiling agent (formerly prodfiler, then Elastic profiler) is the underlying OSS.
varunneal 2y ago
Cool anecdote from inside article > A seasoned performance engineer was looking through Strobelight data and discovered that by filtering on a particular std::vector function call (using the symbolized file and line number) he could identify computationally expensive array copies that happen unintentionally with the ‘auto’ keyword in C++. > The engineer turned a few knobs, adjusted his Scuba query, and happened to notice one of these copies in a particularly hot call path in one of Meta’s largest ads services. He then cracked open his code editor to investigate whether this particular vector copy was intentional… it wasn’t. > It was a simple mistake that any engineer working in C++ has made a hundred times. > So, the engineer typed an “&” after the auto keyword to indicate we want a reference instead of a copy. It was a one-character commit, which, after it was shipped to production, equated to an estimated 15,000 servers in capacity savings per year!
- JoshTriplett 2y ago
  It's a cool anecdote. It's also a case study in heavyweight copies being something that shouldn't happen by default, and should require explicit annotation indicating that the engineer expects a heavyweight copy of the entire structure.
  - mhlakhani 2y ago
    I don’t know if that would have helped here, if memory serves me right: 1. The copy was needed initially 2. This structure wasn’t as heavy back then … over time the code evolved so it became heavy and the copy became unnecessary. That’s harder to find without profiling to guide things
  - ehsankia 2y ago
    If it's safety/correctness versus performance, I think the default should be the former. Copying, while inefficient is generally more correct and avoids hard-to-debug errors. It's the whole discussion about premature optimization. I'd rather make a copy than make sure the array is not mutated anywhere ever.
    - 2y ago
brancz 2y ago
We’re working hard to bring a lot of Strobelight to everyone through Parca[0] as OSS and Polar Signals[1] as the commercial version. Some parts already exists much to come this year! :) [0] https://www.parca.dev/ https://www.parca.dev/ [1] https://www.polarsignals.com/ https://www.polarsignals.com/ (Disclaimer: founder of polar signals)
- deleted 2y ago
  [deleted]
quelup 2y ago
Strobelight is a lifesaver. Especially with high qps services - makes it much easier to see where it's worth spending time trying to optimize.
samstave 2y ago
DOPE Fractal compute expense modeelling is hard. One may do well in applying fluid dynamics (such that we cannot maintain in head) into compute requirements, it will be funny once we realize that everything i mico (pico) fluid dynamics in general
ydjje 2y ago
[flagged]
Starlord2048 2y ago
Between LLVM's optimization passes, static analysis, and modern LLM-powered tools, couldn't we build systems that not only identify but automatically fix these performance issues? GitHub Copilot already suggests code - why not have "Copilot Performance" that refactors inefficient patterns? I'm curious if anyone is working on "self-healing" systems where the optimization feedback loop is closed automatically rather than requiring human engineers to parse complex profiling data.
maknee 2y ago
"All of this is made possible with the inclusion of frame pointers in all of Meta’s user space binaries, otherwise we couldn’t walk the stack to get all these addresses (or we’d have to do some other complicated/expensive thing which wouldn’t be as efficient)" This makes things so, so, so much easier. Otherwise, a lot of effort has to built into creating an unwinder in ebpf code, essentially porting .eh_frame cfa/ra/bp calculations. They claim to have event profilers for non-native languages (e.g. python). Does this mean that they use something similar to https://github.com/benfred/py-spy https://github.com/benfred/py-spy ? Otherwise, it's not obvious to me how they can read python state. Lastly, the github repo https://github.com/facebookincubator/strobelight https://github.com/facebookincubator/strobelight is pretty barebones. Wonder when they'll update it
- brancz 2y ago
  Already been done: 1) native unwinding: https://www.polarsignals.com/blog/posts/2022/11/29/dwarf-based-stack-walking-using-ebpf https://www.polarsignals.com/blog/posts/2022/11/29/dwarf-bas... 2) python: https://www.polarsignals.com/blog/posts/2023/10/04/profiling-python-and-ruby-with-ebpf https://www.polarsignals.com/blog/posts/2023/10/04/profiling... Both available as part of the Parca open source project. https://www.parca.dev/ https://www.parca.dev/ (Disclaimer I work on Parca and am the founder of Polar Signals)
  - maknee 2y ago
    Thanks! Those blogs are incredibly useful. Nice work on the profiler. :) I have multiple questions if you don’t mind answering them: Is there significant overhead to native unwinding and python in ebpf? EBPF needs to constantly read & copy from user space to read data structures. I ask this because unwinding with frame pointers can be done by reading without copying in userland. Python can be ran with different engines (cpython, pypy, etc) and versions (3.7, 3.8,…) and compilers can reorganize offsets. Reading from offsets in seems me to be handwavy. Does this work well in practice/when did it fail?
    - brancz 2y ago
flakiness 2y ago
And open sourcing: https://github.com/facebookincubator/strobelight https://github.com/facebookincubator/strobelight C++ from Meta/FB is much more pleasant to read than ones from ... other older big techs. I appreciate that.
- chriscbr 2y ago
  I only see around three .cpp files in the entire project?
  - flakiness 2y ago
    Look at other FB projects.
BigRedEye 2y ago
At Yandex we have a similar profiler that supports native languages seamlessly, with addition to Python/Java: https://github.com/yandex/perforator https://github.com/yandex/perforator. It's exciting to see new profilers from big players!
iampims 2y ago
I just wish Meta would open source Scuba.
suralind 2y ago
That's really cool. I only wish open source projects were this integrated. (Imagine if making a PR would estimate your AWS cost increase after running canary Kubernetes.) Also what's really cool to see is that Facebook's internal UI actually looks decent. Never work in a company of anywhere close to that size and the tooling always look like it was puked by a dog.

177 points by birdculture 2y ago

Ah, this is performance profiling. Seeing the title and the domain I thought this was user profiling and I was wondering why would Meta be publishing this.

hunter2_ 2y ago
> the domain Perhaps a contributing factor is how HN shows only the final non-eTLD [0] label of the domain. If it showed all labels, you'd have seen "engineering.fb.com" which, while not a dead giveaway, implies that the problem space is technical. It would be nice if this aggressive truncation were applied only above a certain threshold of length. [0] https://en.wikipedia.org/wiki/Public_Suffix_List https://en.wikipedia.org/wiki/Public_Suffix_List
- teddyh 2y ago
  I suggested this 10 years ago. <https://news.ycombinator.com/item?id=8911044 https://news.ycombinator.com/item?id=8911044>
  - hunter2_ 2y ago
    We are actually saying different things, and your point highlights an error in mine (i.e., I assumed they show the eTLD from the PSL plus one extra label, but apparently they have their own shadow PSL which omits things like pp.se and therefore occasionally shows nothing but an eTLD?) but either way we agree that showing more would be better.

doctorhandshake 2y ago

I would assume the name is a reference to the use of strobes in examining high speed periodic motion, like that in motors or on production lines, eg: https://www.checkline.com/inspection_stroboscope https://www.checkline.com/inspection_stroboscope

arnath 2y ago

This is really cool! I've always thought that one thing preventing major competitors to AWS/Azure/GCP is the lack of easy-to-use tooling for machine level monitoring like this. When I was at Microsoft, we built a tool like this that used Windows Firewall filters to track all the network traffic between our services and it was incredibly useful for debugging. That said, as with anything from Meta, I approach this with a grain of salt and the fact that I can't tell what they stand to gain from this makes me suspicious.

theptip 2y ago
> the fact that I can't tell what they stand to gain from this makes me suspicious. Meta is one of the biggest contributors to FOSS in the world. (React, PyTorch, Llama, …). They stand to gain what every big company does, a community contributing to their infra. You’ll note that nobody is open sourcing their ad recommender, that is the one you should be skeptical about if you ever see. You don’t share your secret sauce.
- schmorptron 2y ago
  ByteDance shared the TikTok content recommender, which I'd argue is somewhat close to an ad recommender :)
  - ipsum2 2y ago
    You mean the paper, not the source code?
- paxys 2y ago
  Plus it helps them recruit engineers who are already familiar with their tech stack.
- ipsum2 2y ago
  > You’ll note that nobody is open sourcing their ad recommender Actually... (2019) https://ai.meta.com/blog/dlrm-an-advanced-open-source-deep-learning-recommendation-model/ https://ai.meta.com/blog/dlrm-an-advanced-open-source-deep-l... Source code: https://github.com/facebookresearch/dlrm https://github.com/facebookresearch/dlrm Paper: https://arxiv.org/abs/1906.00091 https://arxiv.org/abs/1906.00091 Updated 2023 blog post, but solely for content recommendation, but ads recommendation is ~90% the same: https://engineering.fb.com/2023/08/09/ml-applications/scaling-instagram-explore-recommendations-system/ https://engineering.fb.com/2023/08/09/ml-applications/scalin... It's a little out of date, but the internal one is built with the same concepts, just more advanced modeling techniques and data.

Thaxll 2y ago

I recommand https://grafana.com/oss/pyroscope/ https://grafana.com/oss/pyroscope/ for continous profiling, I use it in Go and it works well. They have support for many languages https://grafana.com/docs/pyroscope/latest/configure-client/language-sdks/ https://grafana.com/docs/pyroscope/latest/configure-client/l... ( also based on eBPF ).

outerspace 2y ago
Good to know there's an OSS alternative.
- hassleblad23 2y ago
  Strobelight is open source as well.
- tdullien 2y ago
  The Otel profiling agent (formerly prodfiler, then Elastic profiler) is the underlying OSS.

varunneal 2y ago

Cool anecdote from inside article > A seasoned performance engineer was looking through Strobelight data and discovered that by filtering on a particular std::vector function call (using the symbolized file and line number) he could identify computationally expensive array copies that happen unintentionally with the ‘auto’ keyword in C++. > The engineer turned a few knobs, adjusted his Scuba query, and happened to notice one of these copies in a particularly hot call path in one of Meta’s largest ads services. He then cracked open his code editor to investigate whether this particular vector copy was intentional… it wasn’t. > It was a simple mistake that any engineer working in C++ has made a hundred times. > So, the engineer typed an “&” after the auto keyword to indicate we want a reference instead of a copy. It was a one-character commit, which, after it was shipped to production, equated to an estimated 15,000 servers in capacity savings per year!

JoshTriplett 2y ago
It's a cool anecdote. It's also a case study in heavyweight copies being something that shouldn't happen by default, and should require explicit annotation indicating that the engineer expects a heavyweight copy of the entire structure.
- mhlakhani 2y ago
  I don’t know if that would have helped here, if memory serves me right: 1. The copy was needed initially 2. This structure wasn’t as heavy back then … over time the code evolved so it became heavy and the copy became unnecessary. That’s harder to find without profiling to guide things
- ehsankia 2y ago
  If it's safety/correctness versus performance, I think the default should be the former. Copying, while inefficient is generally more correct and avoids hard-to-debug errors. It's the whole discussion about premature optimization. I'd rather make a copy than make sure the array is not mutated anywhere ever.
  - 2y ago

brancz 2y ago

We’re working hard to bring a lot of Strobelight to everyone through Parca[0] as OSS and Polar Signals[1] as the commercial version. Some parts already exists much to come this year! :) [0] https://www.parca.dev/ https://www.parca.dev/ [1] https://www.polarsignals.com/ https://www.polarsignals.com/ (Disclaimer: founder of polar signals)

deleted 2y ago
[deleted]

quelup 2y ago

Strobelight is a lifesaver. Especially with high qps services - makes it much easier to see where it's worth spending time trying to optimize.

samstave 2y ago

DOPE Fractal compute expense modeelling is hard. One may do well in applying fluid dynamics (such that we cannot maintain in head) into compute requirements, it will be funny once we realize that everything i mico (pico) fluid dynamics in general

ydjje 2y ago

[flagged]

Starlord2048 2y ago

Between LLVM's optimization passes, static analysis, and modern LLM-powered tools, couldn't we build systems that not only identify but automatically fix these performance issues? GitHub Copilot already suggests code - why not have "Copilot Performance" that refactors inefficient patterns? I'm curious if anyone is working on "self-healing" systems where the optimization feedback loop is closed automatically rather than requiring human engineers to parse complex profiling data.

maknee 2y ago

"All of this is made possible with the inclusion of frame pointers in all of Meta’s user space binaries, otherwise we couldn’t walk the stack to get all these addresses (or we’d have to do some other complicated/expensive thing which wouldn’t be as efficient)" This makes things so, so, so much easier. Otherwise, a lot of effort has to built into creating an unwinder in ebpf code, essentially porting .eh_frame cfa/ra/bp calculations. They claim to have event profilers for non-native languages (e.g. python). Does this mean that they use something similar to https://github.com/benfred/py-spy https://github.com/benfred/py-spy ? Otherwise, it's not obvious to me how they can read python state. Lastly, the github repo https://github.com/facebookincubator/strobelight https://github.com/facebookincubator/strobelight is pretty barebones. Wonder when they'll update it

brancz 2y ago
Already been done: 1) native unwinding: https://www.polarsignals.com/blog/posts/2022/11/29/dwarf-based-stack-walking-using-ebpf https://www.polarsignals.com/blog/posts/2022/11/29/dwarf-bas... 2) python: https://www.polarsignals.com/blog/posts/2023/10/04/profiling-python-and-ruby-with-ebpf https://www.polarsignals.com/blog/posts/2023/10/04/profiling... Both available as part of the Parca open source project. https://www.parca.dev/ https://www.parca.dev/ (Disclaimer I work on Parca and am the founder of Polar Signals)
- maknee 2y ago
  Thanks! Those blogs are incredibly useful. Nice work on the profiler. :) I have multiple questions if you don’t mind answering them: Is there significant overhead to native unwinding and python in ebpf? EBPF needs to constantly read & copy from user space to read data structures. I ask this because unwinding with frame pointers can be done by reading without copying in userland. Python can be ran with different engines (cpython, pypy, etc) and versions (3.7, 3.8,…) and compilers can reorganize offsets. Reading from offsets in seems me to be handwavy. Does this work well in practice/when did it fail?
  - brancz 2y ago

flakiness 2y ago

And open sourcing: https://github.com/facebookincubator/strobelight https://github.com/facebookincubator/strobelight C++ from Meta/FB is much more pleasant to read than ones from ... other older big techs. I appreciate that.

chriscbr 2y ago
I only see around three .cpp files in the entire project?
- flakiness 2y ago
  Look at other FB projects.

BigRedEye 2y ago

At Yandex we have a similar profiler that supports native languages seamlessly, with addition to Python/Java: https://github.com/yandex/perforator https://github.com/yandex/perforator. It's exciting to see new profilers from big players!

iampims 2y ago

I just wish Meta would open source Scuba.

suralind 2y ago

That's really cool. I only wish open source projects were this integrated. (Imagine if making a PR would estimate your AWS cost increase after running canary Kubernetes.) Also what's really cool to see is that Facebook's internal UI actually looks decent. Never work in a company of anywhere close to that size and the tooling always look like it was puked by a dog.