4 ms·

You may have read it wrong.

by dijit 3y ago

tptacek 3y ago
Did I? How?
- dijit 3y ago
  For one by assuming the work that is done primarily for microkernels/appliances is the absolute limit of userspace networking at Google and that similar work would not go into a hypervisor (hypervisors which are universally treated as a vSwitch in almost all virtual environments the world over). And making that assumption when there are many public examples of Google doing this in other areas such as gVisor and Netstack?
  - tptacek 3y ago
    If you have information about other userspace networking projects at Google, I'd love to read it, but the Snap paper repeatedly suggests that the userspace networking characteristics of the design are distinctive. Certainly, most networking at Google isn't netstack. Have you done much with netstack? It is many things, but ultra-high-performance isn't one of them.
    - dijit 3y ago
      userspace networking will take different forms depending on the use-case. Which is one of the arguments of why to do it that way; instead of using general purpose networking. I haven't the time or inclination to find anything public on this, nor am I interested really in convincing you. Ask a former googler.
      - tptacek 3y ago
        OK. I did. They said "no, it's not the case that networking at Google is predominately user-mode". (They also said "it depends on what you mean by most"). Do you have more you want me to relay to them? Did you work on this stuff at Google? Per the Snap thread above: if you're building a router or a load balancer or some other bit of network infrastructure, it's not unlikely that there's userland IP involved. But if you're shipping a normal program on, like, Borg or whatever, it's kernel networking.
        dijit 3y ago
        I worked as a Google partner for some specialised projects within AAA online gaming. I continue in a similar position today and thus my NDA is still in complete effect which limits what I can say if there’s nothing public. I have not worked for Google, just very closely.
        tptacek 3y ago
        Oh. Then, unless a Googler jumps in here and says I'm wrong: no, ordinary applications at Google are not as a rule built on userspace networking. That's not my opinion (though: it was my prior, having done a bunch of userspace networking stuff), it's the result of asking Google people about it. Maybe it's all changed in the last year! But then: that makes all of this irrelevant to the thread, about FreeBSD vs. Linux network stack performance.
        dijit 3y ago
        Based on this I understand why you're talking like this: I think you have made an assumption/interpretation here and argued the assumption because nobody here (I believe) has claimed that Google only uses user-space networking; merely that google makes use of user-space networking where it's "appropriate" (IE; when FreeBSD would have had an advantage). Which is backed up by basically everything in this thread. Which is why I said you "probably read it wrong". Google is much happier to throw hardware at the problem in most cases, only when it really matters and they would have had to rearchitect the kernel to improve a situation any further do they break out the user-space networking. The point I was driving at was that it's more common than you think. Your base assertion that it's ubiquitous is very obviously false because Chromebooks are pretty common inside google offices and those are running stock chromeOS (except in the offices that are developing chromeOS)
        tptacek 3y ago
        Let me make my point clearly: Google depends on the Linux kernel stack as much as the top-of-the-thread comment suggests that they did, and the things they're doing in user-mode, they would also be doing in user-mode on FreeBSD. That's all I'm here to say. As these things go, in the course of making your argument, you made a falsifiable and, I believe, flatly incorrect claim: Most of the larger tech companies these days are using userland networking and bypass the kernel almost completely for networking At least in Google's case, this isn't true. People doing custom network stack stuff totally do bypass the kernel stack (sometimes with usermode stacks, and sometimes with eBPF, and sometimes with offload). But the way you phrased this, you implied pretty directly that networking writ large was usermode at Google, and while I entertained the possibility that this might be true, when I investigated, it wasn't (unsurprisingly, given how annoying user mode networking is to interact directly with, as opposed to in middlebox applications).

You may have read it wrong.

by dijit 3y ago

You may have read it wrong.

tptacek 3y ago

Did I? How?

dijit 3y ago
For one by assuming the work that is done primarily for microkernels/appliances is the absolute limit of userspace networking at Google and that similar work would not go into a hypervisor (hypervisors which are universally treated as a vSwitch in almost all virtual environments the world over). And making that assumption when there are many public examples of Google doing this in other areas such as gVisor and Netstack?
- tptacek 3y ago
  If you have information about other userspace networking projects at Google, I'd love to read it, but the Snap paper repeatedly suggests that the userspace networking characteristics of the design are distinctive. Certainly, most networking at Google isn't netstack. Have you done much with netstack? It is many things, but ultra-high-performance isn't one of them.
  - dijit 3y ago
    userspace networking will take different forms depending on the use-case. Which is one of the arguments of why to do it that way; instead of using general purpose networking. I haven't the time or inclination to find anything public on this, nor am I interested really in convincing you. Ask a former googler.
    - tptacek 3y ago
      OK. I did. They said "no, it's not the case that networking at Google is predominately user-mode". (They also said "it depends on what you mean by most"). Do you have more you want me to relay to them? Did you work on this stuff at Google? Per the Snap thread above: if you're building a router or a load balancer or some other bit of network infrastructure, it's not unlikely that there's userland IP involved. But if you're shipping a normal program on, like, Borg or whatever, it's kernel networking.
      - dijit 3y ago
        I worked as a Google partner for some specialised projects within AAA online gaming. I continue in a similar position today and thus my NDA is still in complete effect which limits what I can say if there’s nothing public. I have not worked for Google, just very closely.
        tptacek 3y ago
        Oh. Then, unless a Googler jumps in here and says I'm wrong: no, ordinary applications at Google are not as a rule built on userspace networking. That's not my opinion (though: it was my prior, having done a bunch of userspace networking stuff), it's the result of asking Google people about it. Maybe it's all changed in the last year! But then: that makes all of this irrelevant to the thread, about FreeBSD vs. Linux network stack performance.
        dijit 3y ago
        Based on this I understand why you're talking like this: I think you have made an assumption/interpretation here and argued the assumption because nobody here (I believe) has claimed that Google only uses user-space networking; merely that google makes use of user-space networking where it's "appropriate" (IE; when FreeBSD would have had an advantage). Which is backed up by basically everything in this thread. Which is why I said you "probably read it wrong". Google is much happier to throw hardware at the problem in most cases, only when it really matters and they would have had to rearchitect the kernel to improve a situation any further do they break out the user-space networking. The point I was driving at was that it's more common than you think. Your base assertion that it's ubiquitous is very obviously false because Chromebooks are pretty common inside google offices and those are running stock chromeOS (except in the offices that are developing chromeOS)
        tptacek 3y ago
        Let me make my point clearly: Google depends on the Linux kernel stack as much as the top-of-the-thread comment suggests that they did, and the things they're doing in user-mode, they would also be doing in user-mode on FreeBSD. That's all I'm here to say. As these things go, in the course of making your argument, you made a falsifiable and, I believe, flatly incorrect claim: Most of the larger tech companies these days are using userland networking and bypass the kernel almost completely for networking At least in Google's case, this isn't true. People doing custom network stack stuff totally do bypass the kernel stack (sometimes with usermode stacks, and sometimes with eBPF, and sometimes with offload). But the way you phrased this, you implied pretty directly that networking writ large was usermode at Google, and while I entertained the possibility that this might be true, when I investigated, it wasn't (unsurprisingly, given how annoying user mode networking is to interact directly with, as opposed to in middlebox applications).