3 ms·

I worked as a Google partner for some specialised projects within AAA online gaming. I continue in a similar position today and thus my NDA is still in complet

by dijit 3y ago

I worked as a Google partner for some specialised projects within AAA online gaming. I continue in a similar position today and thus my NDA is still in complete effect which limits what I can say if there’s nothing public. I have not worked for Google, just very closely.

tptacek 3y ago
Oh. Then, unless a Googler jumps in here and says I'm wrong: no, ordinary applications at Google are not as a rule built on userspace networking. That's not my opinion (though: it was my prior, having done a bunch of userspace networking stuff), it's the result of asking Google people about it. Maybe it's all changed in the last year! But then: that makes all of this irrelevant to the thread, about FreeBSD vs. Linux network stack performance.
- dijit 3y ago
  Based on this I understand why you're talking like this: I think you have made an assumption/interpretation here and argued the assumption because nobody here (I believe) has claimed that Google only uses user-space networking; merely that google makes use of user-space networking where it's "appropriate" (IE; when FreeBSD would have had an advantage). Which is backed up by basically everything in this thread. Which is why I said you "probably read it wrong". Google is much happier to throw hardware at the problem in most cases, only when it really matters and they would have had to rearchitect the kernel to improve a situation any further do they break out the user-space networking. The point I was driving at was that it's more common than you think. Your base assertion that it's ubiquitous is very obviously false because Chromebooks are pretty common inside google offices and those are running stock chromeOS (except in the offices that are developing chromeOS)
  - tptacek 3y ago
    Let me make my point clearly: Google depends on the Linux kernel stack as much as the top-of-the-thread comment suggests that they did, and the things they're doing in user-mode, they would also be doing in user-mode on FreeBSD. That's all I'm here to say. As these things go, in the course of making your argument, you made a falsifiable and, I believe, flatly incorrect claim: Most of the larger tech companies these days are using userland networking and bypass the kernel almost completely for networking At least in Google's case, this isn't true. People doing custom network stack stuff totally do bypass the kernel stack (sometimes with usermode stacks, and sometimes with eBPF, and sometimes with offload). But the way you phrased this, you implied pretty directly that networking writ large was usermode at Google, and while I entertained the possibility that this might be true, when I investigated, it wasn't (unsurprisingly, given how annoying user mode networking is to interact directly with, as opposed to in middlebox applications).
    - dijit 3y ago
      Ok, this is a very hostile and could not possibly be considered a charitable interpretation of what I said, in fact I'd say it borders on trying to pick an argument where there isn't one. I did not "flat out lie" and I detest the insinuation. I expect better of you honestly. To answer: "Most of the larger tech companies these days are using userland networking and bypass the kernel almost completely for networking" You could read "almost completely" as in "almost across the whole company" which would be a weird way to read it, but you seem to have read it this way. I intended it to mean: when they bypass the kernel; it is a near complete bypass. Since there actually is still a network connection going through the kernel (the host itself will still be connected of course), which is of course the inverse of what you seem to have taken away; in that user-mode networking is used even less than entirely even on a single node. edit: In fact, I stated multiple times in my post: "Google is mostly happy to just throw hardware at this" which you seemed to just.. ignore? Google are absolutely happy to throw hardware at issues until they can't anymore or the gains are too enormous to avoid. I thought I was extremely clear about that. Your point about user-land networking in FreeBSD is just a nonsense one to make -- and not the point we were discussing anyway, like suggesting "if it did rain beer, would we all get drunk?" it's completely hypothetical and not based in any subjective reality or objective truth. You have absolutely no way of knowing if FreeBSD could do those things, the statement that the architecture permits is is shown somewhat in FreeBSD's use in Netflix, which has been commented elsewhere in this thread to achieve close to 0.8TiB of data transfer, but knowing what google would have done with freebsd would require seeing into alternative realities. I don't know where you work, but as far as I know: nobody has managed to perfect that technology yet.
      - tptacek 3y ago
        That is exactly how I took your comment. This is a weird cursed thread that started out with a pretty silly† claim about FreeBSD vs. Linux network stack performance (neither of us started it; it's a standard platform war argument). Someone made a comment that the hyperscalers all depend on the Linux kernel stack, to a far greater degree than they do on FreeBSD. That's a true statement; at this point, you and I have both agreed on it. When that point was pressed earlier, you and another comment brought up kernel bypass (usermode networking, specifically) as a way of dismissing hyperscaler Linux dependencies. But that's not really a valid argument. Hyperscalers do kernel bypass stuff! Of course they do! But they're doing it for the things that you'd formerly have bought dedicated network equipment for, and in no case are they doing it in a situation where deploying FreeBSD and using the FreeBSD stack would be a valid alternative. The disconnect between us is that I'm still talking about the subject that the thread was originally about --- whether Google using the Linux stack is a valid point backing up its fitness for purpose. I think it pretty clearly is a valid point. I think the usermode networking stuff is interesting, but is a sideshow. † "Silly" because these kinds of claims never, ever get resolved, and just bring out each side's cheering section, not because I have low opinions of FreeBSD's kernel stack --- I came up on that stack and find it easier to follow than Linux's, though I rather doubt the claim that it has a decisive performance advantage.

by dijit 3y ago

tptacek 3y ago

Oh. Then, unless a Googler jumps in here and says I'm wrong: no, ordinary applications at Google are not as a rule built on userspace networking. That's not my opinion (though: it was my prior, having done a bunch of userspace networking stuff), it's the result of asking Google people about it. Maybe it's all changed in the last year! But then: that makes all of this irrelevant to the thread, about FreeBSD vs. Linux network stack performance.

dijit 3y ago
Based on this I understand why you're talking like this: I think you have made an assumption/interpretation here and argued the assumption because nobody here (I believe) has claimed that Google only uses user-space networking; merely that google makes use of user-space networking where it's "appropriate" (IE; when FreeBSD would have had an advantage). Which is backed up by basically everything in this thread. Which is why I said you "probably read it wrong". Google is much happier to throw hardware at the problem in most cases, only when it really matters and they would have had to rearchitect the kernel to improve a situation any further do they break out the user-space networking. The point I was driving at was that it's more common than you think. Your base assertion that it's ubiquitous is very obviously false because Chromebooks are pretty common inside google offices and those are running stock chromeOS (except in the offices that are developing chromeOS)
- tptacek 3y ago
  Let me make my point clearly: Google depends on the Linux kernel stack as much as the top-of-the-thread comment suggests that they did, and the things they're doing in user-mode, they would also be doing in user-mode on FreeBSD. That's all I'm here to say. As these things go, in the course of making your argument, you made a falsifiable and, I believe, flatly incorrect claim: Most of the larger tech companies these days are using userland networking and bypass the kernel almost completely for networking At least in Google's case, this isn't true. People doing custom network stack stuff totally do bypass the kernel stack (sometimes with usermode stacks, and sometimes with eBPF, and sometimes with offload). But the way you phrased this, you implied pretty directly that networking writ large was usermode at Google, and while I entertained the possibility that this might be true, when I investigated, it wasn't (unsurprisingly, given how annoying user mode networking is to interact directly with, as opposed to in middlebox applications).
  - dijit 3y ago
    Ok, this is a very hostile and could not possibly be considered a charitable interpretation of what I said, in fact I'd say it borders on trying to pick an argument where there isn't one. I did not "flat out lie" and I detest the insinuation. I expect better of you honestly. To answer: "Most of the larger tech companies these days are using userland networking and bypass the kernel almost completely for networking" You could read "almost completely" as in "almost across the whole company" which would be a weird way to read it, but you seem to have read it this way. I intended it to mean: when they bypass the kernel; it is a near complete bypass. Since there actually is still a network connection going through the kernel (the host itself will still be connected of course), which is of course the inverse of what you seem to have taken away; in that user-mode networking is used even less than entirely even on a single node. edit: In fact, I stated multiple times in my post: "Google is mostly happy to just throw hardware at this" which you seemed to just.. ignore? Google are absolutely happy to throw hardware at issues until they can't anymore or the gains are too enormous to avoid. I thought I was extremely clear about that. Your point about user-land networking in FreeBSD is just a nonsense one to make -- and not the point we were discussing anyway, like suggesting "if it did rain beer, would we all get drunk?" it's completely hypothetical and not based in any subjective reality or objective truth. You have absolutely no way of knowing if FreeBSD could do those things, the statement that the architecture permits is is shown somewhat in FreeBSD's use in Netflix, which has been commented elsewhere in this thread to achieve close to 0.8TiB of data transfer, but knowing what google would have done with freebsd would require seeing into alternative realities. I don't know where you work, but as far as I know: nobody has managed to perfect that technology yet.
    - tptacek 3y ago
      That is exactly how I took your comment. This is a weird cursed thread that started out with a pretty silly† claim about FreeBSD vs. Linux network stack performance (neither of us started it; it's a standard platform war argument). Someone made a comment that the hyperscalers all depend on the Linux kernel stack, to a far greater degree than they do on FreeBSD. That's a true statement; at this point, you and I have both agreed on it. When that point was pressed earlier, you and another comment brought up kernel bypass (usermode networking, specifically) as a way of dismissing hyperscaler Linux dependencies. But that's not really a valid argument. Hyperscalers do kernel bypass stuff! Of course they do! But they're doing it for the things that you'd formerly have bought dedicated network equipment for, and in no case are they doing it in a situation where deploying FreeBSD and using the FreeBSD stack would be a valid alternative. The disconnect between us is that I'm still talking about the subject that the thread was originally about --- whether Google using the Linux stack is a valid point backing up its fitness for purpose. I think it pretty clearly is a valid point. I think the usermode networking stuff is interesting, but is a sideshow. † "Silly" because these kinds of claims never, ever get resolved, and just bring out each side's cheering section, not because I have low opinions of FreeBSD's kernel stack --- I came up on that stack and find it easier to follow than Linux's, though I rather doubt the claim that it has a decisive performance advantage.