5 ms·

You probably weren't there when servers were running for many days at a time. By the time you joined and benchmarked these systems, the continuous rolling depl

You probably weren't there when servers were running for many days at a time. By the time you joined and benchmarked these systems, the continuous rolling deployment had taken over. If you're restarting the server every few hours, of course the memory fragmentation isn't much of an issue. > But you were unable to show an advantage at the system level when challenged on it, and that's what matters. You mean 5 years after I stopped working on the kernel and the underlying system had changed? I don't recall ever talking to you on the matter.

jcalvinowens 7mo ago
> By the time you joined and benchmarked these systems, the continuous rolling deployment had taken over Nope, I started in 2014. > I don't recall ever talking to you on the matter. I recall. You refused to believe the benchmark results and made me repeat the test, then stopped replying after I did :)
- nullpoint420 7mo ago
  This is why I love hacker news. I learn so much from these moments.
  - integricho 7mo ago
    I came here for the article, stayed for the drama.
  - danudey 7mo ago
    Like "never work at Meta unless you can out-toxic your coworkers".
    - __turbobrew__ 7mo ago
      Yea I knew meta was toxic, but publicly beefing over something over a decade ago is a whole other matter. I can’t even remember what I was working on 10 years ago, and even if I did I wouldn’t be bringing people down that much later.
      - baby 7mo ago
        The problem is a lot of very strong engineers are also very difficult to work with. I worked at Meta too and can tell you the other side of the coin is that people who were too toxic could get canned as well!
        __turbobrew__ 7mo ago
        Yes, I have worked with the strong but arrogant/snarky engineers. Luckily most of them got canned or forced out because the environment they create around themselves more than negates the positive impact they have. The strongest engineers I have worked with are all humble and kind. It is their loss, I cannot imagine letting a minor work quarrel live rent free in my head for over a decade. I feel bad enough when something is stuck in my mind for a week.
      - throwaway2037 7mo ago
        Yeah, I am loving the public mudslinging over shit from 10 years ago, like high school girls fighting. This is like the FAANG version of the TV show Suits. We can call it FAANGs and use Midjorney to create the cover art and give the actors vampire fangs. On a more serious note, it seems like any hyper competitive company eventually spirals into an awful, toxic working env.
    - CamperBob2 7mo ago
      Like "Hey, I wonder if Conway's Law works both ways. Huh. Wow. It looks like that is indeed the case."
    - lmm 7mo ago
      Funny, I was thinking what a relief it was to see people making their arguments frankly like on the HN of 10+ years ago.
    - hedayet 7mo ago
      Inside Meta, engineers are one of the kindest group of people. This thread would've been way more fun with a couple of middle managers and product managers in the mix ;-)
- adsharma 7mo ago
  The patches were written in 2011 and published in 2012. They did what they were supposed to at the time. For the peanut gallery: this is a manifestation of an internal eng culture at fb that I wasn't particularly fond of. Celebrating that "I killed X" and partying about it. You didn't reply to the main point: did you benchmark a server that was running several days at a time? Reasonable people can disagree about whether this a good deployment strategy or not. I tend to believe that there are many places which want to deploy servers and run for months if not days.
  - alexgartrell 7mo ago
    For the peanut gallery more: I worked with both of these guys at Meta on this. The "servers are only on for a few hours" thing was like never true so I have no idea where that claim is coming from. The web performance test took more than a few hours to run alone and we had way more aggressive soaks for other workloads. My recollection was that "write zeroes" just became a cheaper operation between '12 and '14. A fun fact to distract from the awkwardness: a lot of the kernel work done in the early days was exceedingly scrappy. The port mapping stuff for memcached UDP before SO_REUSEPORT for example. FB binaries couldn't even run on vanilla linux a lot of the time. Over the next several years we put a TON of effort in getting as close to mainline as possible and now Meta is one of the biggest drivers of Linux development.
    - adsharma 7mo ago
      [ Edit: "servers" in this context meant the HHVM server processes, not the physical server which of course had a longer uptime ] People got promoted for continuous deployment https://engineering.fb.com/2017/08/31/web/rapid-release-at-massive-scale/ https://engineering.fb.com/2017/08/31/web/rapid-release-at-m... I think it's fair to say the hardware changed, the deployment strategy changed and the patches were no longer relevant, so we stopped applying them. When I showed up, there were 100+ patches on top of a 2009 kernel tree. I reduced the size to about 10 or so critical patches, rebased them at a 6 months cadence over 2-3 years. Upstreamed a few. Didn't go around saying those old patches were bad ideas and I got rid of them. How you say it matters.

jcalvinowens 7mo ago

> By the time you joined and benchmarked these systems, the continuous rolling deployment had taken over Nope, I started in 2014. > I don't recall ever talking to you on the matter. I recall. You refused to believe the benchmark results and made me repeat the test, then stopped replying after I did :)

nullpoint420 7mo ago
This is why I love hacker news. I learn so much from these moments.
- integricho 7mo ago
  I came here for the article, stayed for the drama.
- danudey 7mo ago
  Like "never work at Meta unless you can out-toxic your coworkers".
  - __turbobrew__ 7mo ago
    Yea I knew meta was toxic, but publicly beefing over something over a decade ago is a whole other matter. I can’t even remember what I was working on 10 years ago, and even if I did I wouldn’t be bringing people down that much later.
    - baby 7mo ago
      The problem is a lot of very strong engineers are also very difficult to work with. I worked at Meta too and can tell you the other side of the coin is that people who were too toxic could get canned as well!
      - __turbobrew__ 7mo ago
        Yes, I have worked with the strong but arrogant/snarky engineers. Luckily most of them got canned or forced out because the environment they create around themselves more than negates the positive impact they have. The strongest engineers I have worked with are all humble and kind. It is their loss, I cannot imagine letting a minor work quarrel live rent free in my head for over a decade. I feel bad enough when something is stuck in my mind for a week.
    - throwaway2037 7mo ago
      Yeah, I am loving the public mudslinging over shit from 10 years ago, like high school girls fighting. This is like the FAANG version of the TV show Suits. We can call it FAANGs and use Midjorney to create the cover art and give the actors vampire fangs. On a more serious note, it seems like any hyper competitive company eventually spirals into an awful, toxic working env.
  - CamperBob2 7mo ago
    Like "Hey, I wonder if Conway's Law works both ways. Huh. Wow. It looks like that is indeed the case."
  - lmm 7mo ago
    Funny, I was thinking what a relief it was to see people making their arguments frankly like on the HN of 10+ years ago.
  - hedayet 7mo ago
    Inside Meta, engineers are one of the kindest group of people. This thread would've been way more fun with a couple of middle managers and product managers in the mix ;-)
adsharma 7mo ago
The patches were written in 2011 and published in 2012. They did what they were supposed to at the time. For the peanut gallery: this is a manifestation of an internal eng culture at fb that I wasn't particularly fond of. Celebrating that "I killed X" and partying about it. You didn't reply to the main point: did you benchmark a server that was running several days at a time? Reasonable people can disagree about whether this a good deployment strategy or not. I tend to believe that there are many places which want to deploy servers and run for months if not days.
- alexgartrell 7mo ago
  For the peanut gallery more: I worked with both of these guys at Meta on this. The "servers are only on for a few hours" thing was like never true so I have no idea where that claim is coming from. The web performance test took more than a few hours to run alone and we had way more aggressive soaks for other workloads. My recollection was that "write zeroes" just became a cheaper operation between '12 and '14. A fun fact to distract from the awkwardness: a lot of the kernel work done in the early days was exceedingly scrappy. The port mapping stuff for memcached UDP before SO_REUSEPORT for example. FB binaries couldn't even run on vanilla linux a lot of the time. Over the next several years we put a TON of effort in getting as close to mainline as possible and now Meta is one of the biggest drivers of Linux development.
  - adsharma 7mo ago
    [ Edit: "servers" in this context meant the HHVM server processes, not the physical server which of course had a longer uptime ] People got promoted for continuous deployment https://engineering.fb.com/2017/08/31/web/rapid-release-at-massive-scale/ https://engineering.fb.com/2017/08/31/web/rapid-release-at-m... I think it's fair to say the hardware changed, the deployment strategy changed and the patches were no longer relevant, so we stopped applying them. When I showed up, there were 100+ patches on top of a 2009 kernel tree. I reduced the size to about 10 or so critical patches, rebased them at a 6 months cadence over 2-3 years. Upstreamed a few. Didn't go around saying those old patches were bad ideas and I got rid of them. How you say it matters.