6 ms·

Love the look of all the tools netflix have at their disposal

by elktea 5y ago

H8crilA 5y ago
A quick to use remote profiler does wonders to an org. By fast to use I mean it takes me 10 seconds to kick off a remote profiling scrape. Someone entrepreneurial please make a purchase-able service out of that, I'll be feeling lost once I quit my corp (not Netflix). Stuff is 100x more useful than remote debugging which ultimately almost nobody uses in practice.
- tdullien 5y ago
  Shameless plug: We just launched Prodfiler last week -- it's continuous in-production profiling based on eBPF, handles C/C++/Go/Rust/JVM/Python/Perl/Ruby (with NodeJS and .net in the pipeline), and it does not require symbols on the machine, nor recompiles with framepointer, nor special JVM flags. https://prodfiler.com https://prodfiler.com :-)
- bostonsre 5y ago
  Datadog has continuous profiling that is pretty useful. I'm pretty sure they follow Brendan religiously.. I hope they release something like flamescope next.
Cthulhu_ 5y ago
And that he described "Normally I'd SSH into a machine", but you shouldn't even be able to do something like that if you've got infrastructure-as-code; the machines are ephemeral, they probably don't have any of the tools that you need to do an analysis, they shouldn't even have SSH or open ports unless there's some tool interacting with it, but in practice, they should be disposable on a whim if there is an update of sorts. Had a colleague who asked for SSH access to production machines to debug an issue. Ops team asked what he wanted to do, guy just wanted to look at which env vars were set. Ops team told him how to do his job - log the configuration - instead of give him access, because they had a mandate to ensure five or seven nines uptime. Can't risk it. I had a lot more respect for the ops people there than I had for my fellow SWE's.
- raverbashing 5y ago
  I disagree. There are always bugs that will happen only on the prod machines. Sure, they are rare, but they exist. > just wanted to look at which env vars were set. Ops team told him how to do his job - log the configuration Well, that's not risk free neither. That needs a code change and you risk exposing secrets, or logging more than you should. (Though I agree with the general idea of you shouldn't be SSHing if you can do it another way)
- Dylan16807 5y ago
  > Ops team told him how to do his job - log the configuration - instead of give him access "do your job" is a really rude way to refer to debugging by logging instead of debugging interactively.
  - sokoloff 5y ago
    I suspect this usage is an awkward phrasing of “told him how to accomplish the result he wanted”; even though in common usage it implies a lack of overall competence, here I think it just implied a lack of a specific recipe/technique.
    - ubercow13 5y ago
      Doesn’t seem so, as GP specifically said this caused them to lose respect for their colleagues.
      - sokoloff 5y ago
        That’s a possible interpretation of their comment, but not the only interpretation and certainly not something they specifically said. (Another interpretation is that the author previously held Ops in higher regard than SWE and that this event did not change that.)
        robertlagrant 5y ago
        I imagine if overzealous rule enforcement is the cause of respect that the OP would've always had more respect for Ops :D

Love the look of all the tools netflix have at their disposal

by elktea 5y ago

Love the look of all the tools netflix have at their disposal

A quick to use remote profiler does wonders to an org. By fast to use I mean it takes me 10 seconds to kick off a remote profiling scrape. Someone entrepreneurial please make a purchase-able service out of that, I'll be feeling lost once I quit my corp (not Netflix). Stuff is 100x more useful than remote debugging which ultimately almost nobody uses in practice.

tdullien 5y ago
Shameless plug: We just launched Prodfiler last week -- it's continuous in-production profiling based on eBPF, handles C/C++/Go/Rust/JVM/Python/Perl/Ruby (with NodeJS and .net in the pipeline), and it does not require symbols on the machine, nor recompiles with framepointer, nor special JVM flags. https://prodfiler.com https://prodfiler.com :-)
bostonsre 5y ago
Datadog has continuous profiling that is pretty useful. I'm pretty sure they follow Brendan religiously.. I hope they release something like flamescope next.

Cthulhu_ 5y ago

And that he described "Normally I'd SSH into a machine", but you shouldn't even be able to do something like that if you've got infrastructure-as-code; the machines are ephemeral, they probably don't have any of the tools that you need to do an analysis, they shouldn't even have SSH or open ports unless there's some tool interacting with it, but in practice, they should be disposable on a whim if there is an update of sorts. Had a colleague who asked for SSH access to production machines to debug an issue. Ops team asked what he wanted to do, guy just wanted to look at which env vars were set. Ops team told him how to do his job - log the configuration - instead of give him access, because they had a mandate to ensure five or seven nines uptime. Can't risk it. I had a lot more respect for the ops people there than I had for my fellow SWE's.

raverbashing 5y ago
I disagree. There are always bugs that will happen only on the prod machines. Sure, they are rare, but they exist. > just wanted to look at which env vars were set. Ops team told him how to do his job - log the configuration Well, that's not risk free neither. That needs a code change and you risk exposing secrets, or logging more than you should. (Though I agree with the general idea of you shouldn't be SSHing if you can do it another way)
Dylan16807 5y ago
> Ops team told him how to do his job - log the configuration - instead of give him access "do your job" is a really rude way to refer to debugging by logging instead of debugging interactively.
- sokoloff 5y ago
  I suspect this usage is an awkward phrasing of “told him how to accomplish the result he wanted”; even though in common usage it implies a lack of overall competence, here I think it just implied a lack of a specific recipe/technique.
  - ubercow13 5y ago
    Doesn’t seem so, as GP specifically said this caused them to lose respect for their colleagues.
    - sokoloff 5y ago
      That’s a possible interpretation of their comment, but not the only interpretation and certainly not something they specifically said. (Another interpretation is that the author previously held Ops in higher regard than SWE and that this event did not change that.)
      - robertlagrant 5y ago
        I imagine if overzealous rule enforcement is the cause of respect that the OP would've always had more respect for Ops :D