4 ms·

At my previous job, I wrote a simple bpftrace script [1] to track Python2 invocations. The output of this program was sent to a message bus and eventually store

At my previous job, I wrote a simple bpftrace script [1] to track Python2 invocations. The output of this program was sent to a message bus and eventually stored in a columnar database for exploration. While Python2 got removed from the base operating system image, we wanted to uncover weird edge cases of engineers running special builds. We found more than one :) I think BPF is underused for safe deprecations. It's super useful to prove that an executable is not loaded, or that a method is not called while being traced :) [1]: https://github.com/javierhonduco/bpf-playground/blob/master/bpftrace/python2_deprecation.bt https://github.com/javierhonduco/bpf-playground/blob/master/...

heybrendan 4y ago
Fascinating. Why was a message bus necessary and what message bus and database (presumably open source) solutions did you use? Thanks for sharing the use case. Super clever solution.
- javierhonduco 4y ago
  The script was running on a subset of the production machines, which still accounted for a rather large number. The message bus eventually transported bpftrace's output to the database. The roll out was done in phases to ensure that the overhead would not be prohibitive as well as to make sure that it wouldn't cause egregious priority inversions. Scribe [0] + LogDevice [1] + Scuba [2]. I believe Scribe was open sourced a very long time ago, and LogDevice more recently, but they are both archived now. - [0]: https://engineering.fb.com/2019/10/07/data-infrastructure/scribe/ https://engineering.fb.com/2019/10/07/data-infrastructure/sc... - [1]: https://logdevice.io/ https://logdevice.io/ - [2]: https://research.facebook.com/publications/scuba-diving-into-data-at-facebook/ https://research.facebook.com/publications/scuba-diving-into...
  - heybrendan 4y ago
    Very interesting indeed! Any idea why those projects (Scribe and LogDevice) were mothballed? Did Facebook simply lose interest in maintaining or did other OSS projects enter the mix that proved to be a better fit? Thanks again for sharing.
    - javierhonduco 4y ago
      I’m not sure there’s any public write-up on this, sorry

heybrendan 4y ago

Fascinating. Why was a message bus necessary and what message bus and database (presumably open source) solutions did you use? Thanks for sharing the use case. Super clever solution.

javierhonduco 4y ago
The script was running on a subset of the production machines, which still accounted for a rather large number. The message bus eventually transported bpftrace's output to the database. The roll out was done in phases to ensure that the overhead would not be prohibitive as well as to make sure that it wouldn't cause egregious priority inversions. Scribe [0] + LogDevice [1] + Scuba [2]. I believe Scribe was open sourced a very long time ago, and LogDevice more recently, but they are both archived now. - [0]: https://engineering.fb.com/2019/10/07/data-infrastructure/scribe/ https://engineering.fb.com/2019/10/07/data-infrastructure/sc... - [1]: https://logdevice.io/ https://logdevice.io/ - [2]: https://research.facebook.com/publications/scuba-diving-into-data-at-facebook/ https://research.facebook.com/publications/scuba-diving-into...
- heybrendan 4y ago
  Very interesting indeed! Any idea why those projects (Scribe and LogDevice) were mothballed? Did Facebook simply lose interest in maintaining or did other OSS projects enter the mix that proved to be a better fit? Thanks again for sharing.
  - javierhonduco 4y ago
    I’m not sure there’s any public write-up on this, sorry