3 ms·

Thanks, I hadn't read that far into the paper. But I have to say I had what I feel is a good reason to be skeptical before even reading a single word in the pap

Thanks, I hadn't read that far into the paper. But I have to say I had what I feel is a good reason to be skeptical before even reading a single word in the paper, honestly. Which is that Facebook never felt so... blazing fast, shall we say, to make me believe anyone even wanted to pay attention to tiny performance regressions, let alone the drive and tooling to do so.

vlovich123 2y ago
> blazing fast, shall we say, to make me believe anyone even wanted to pay attention to tiny performance regressions Important to distinguish frontend vs backend performance of course. This is about backend performance where they care about this stuff a lot because it multiplies at scale & starts costing them real money. Frontend performance has less of a direct impact on their numbers with the only data I know on that is the oft-cited Google stuff trying to claim that there's a direct correlation between lost revenue and latency (which I haven't seen anyone else bother to try to replicate & see if it holds up).
- dataflow 2y ago
  > Important to distinguish frontend vs backend performance of course. I'm not sure what you're calling frontend in this context (client side? or client-facing "front-end" servers?), but I'm talking about their server side specifically. I've looked at their API calls and understand when slowness is coming from the client or the server. The client side is even slower for sure, but the server side also never felt like it was optimized to the point where deviations this small mattered.
  - jacques_chester 2y ago
    I think the confusion arises because of the difference between optimization and control, which are superficially similar. Having control lets you see if things changed. Optimization is changing things. This team seems to be focused on control. I assume optimization is left to the service teams.
    - dataflow 2y ago
      I think by control you mean observability? I get that they're different, but the whole point is optimization here. They're not gathering performance metrics just to hang them up on a wall and marvel at the number of decimal points, right? They presumably invested all the effort into this infrastructure because they think this much precision has significant ROI on the optimization side.
      - jacques_chester 2y ago
        I'm using "control" in the statistical process control sense, where it means "we can tell if variation is ordinary or extraordinary". To me it seemed clear that the paper is about detecting regressions, which is control under my definition above. I still think of that as distinct from optimization.
      - joshuamorton 2y ago
        Right, but what is the point of optimizing? It often isn't to make things go faster for an individual user (often times the driving factor of latency is not computation, but inter-system rpc latency, etc.). The value is to bin-pack processing more requests into the same bucket of CPU. That can have latency wins, but it may not in a lot of contexts.

vlovich123 2y ago

> blazing fast, shall we say, to make me believe anyone even wanted to pay attention to tiny performance regressions Important to distinguish frontend vs backend performance of course. This is about backend performance where they care about this stuff a lot because it multiplies at scale & starts costing them real money. Frontend performance has less of a direct impact on their numbers with the only data I know on that is the oft-cited Google stuff trying to claim that there's a direct correlation between lost revenue and latency (which I haven't seen anyone else bother to try to replicate & see if it holds up).

dataflow 2y ago
> Important to distinguish frontend vs backend performance of course. I'm not sure what you're calling frontend in this context (client side? or client-facing "front-end" servers?), but I'm talking about their server side specifically. I've looked at their API calls and understand when slowness is coming from the client or the server. The client side is even slower for sure, but the server side also never felt like it was optimized to the point where deviations this small mattered.
- jacques_chester 2y ago
  I think the confusion arises because of the difference between optimization and control, which are superficially similar. Having control lets you see if things changed. Optimization is changing things. This team seems to be focused on control. I assume optimization is left to the service teams.
  - dataflow 2y ago
    I think by control you mean observability? I get that they're different, but the whole point is optimization here. They're not gathering performance metrics just to hang them up on a wall and marvel at the number of decimal points, right? They presumably invested all the effort into this infrastructure because they think this much precision has significant ROI on the optimization side.
    - jacques_chester 2y ago
      I'm using "control" in the statistical process control sense, where it means "we can tell if variation is ordinary or extraordinary". To me it seemed clear that the paper is about detecting regressions, which is control under my definition above. I still think of that as distinct from optimization.
    - joshuamorton 2y ago
      Right, but what is the point of optimizing? It often isn't to make things go faster for an individual user (often times the driving factor of latency is not computation, but inter-system rpc latency, etc.). The value is to bin-pack processing more requests into the same bucket of CPU. That can have latency wins, but it may not in a lot of contexts.