9 ms·

Snap: A Microkernel Approach to Host Networking

144 points by sandGorgon 7y ago

sandGorgon 7y ago
"A change to the kernel stack takes 1-2 months to deploy; a new Snap release gets deployed on a weekly basis."
- persistent 7y ago
  At many companies rolling out a new kernel in a month would be an outright miracle.
nimrody 7y ago
"While early microkernel work saw significant performanceoverheads attributed to inter-process communication (IPC)and address space changes [15,16,20], such overheads areless significant today. Compared to the uniprocessor systemsof the 80s and 90s, today’s servers contain dozens of cores,which allows microkernel invocation to leverage inter-coreIPC while maintaining application cache locality. This ap-proach can evenimproveoverall performance when there islittle state to communicate across the IPC (common in zero-copy networking) and in avoiding ring switch costs of systemcalls. Moreover, recent security vulnerabilities such as Melt-down [43] force kernel/user address space isolation, even inmonolithic kernels [25]. Techniques like tagged-TLB supportin modern processors, streamlined ring switching hardwaremade necessary with the resurgence of virtualization, andIPC optimization techniques such as those explored in theL4 microkernel [29], in FlexSC [58], and in SkyBridge [44],further allow a modern microkernel to essentially close theperformance gap between direct system calls and indirectsystem calls through IPC."
- snvzz 7y ago
  You're right to quote some text that addresses the idea that "microkernels are slow", which keeps popping up. To that, I add this article[0], which is sufficient in destroying that idea. [0]https://blog.darknedgy.net/technology/2016/01/01/0/ https://blog.darknedgy.net/technology/2016/01/01/0/
  - AlphaSite 7y ago
    Another point is that macOS and iOS are moving towards a user space driver stack and away from Kernel space.
    - snvzz 7y ago
      Because, save few exceptions (e.g. timers, early debug serial...) running drivers in supervisor mode isn't very smart. Drivers tend to have high bug density, and supervisor mode means high potential for damage. Much can be mitigated by running drivers in userspace[0][1], especially with some IOMMU help. [0] https://www.cs.vu.nl/~herbertb/papers/minix3_dsn09.pdf https://www.cs.vu.nl/~herbertb/papers/minix3_dsn09.pdf [1] https://wiki.minix3.org/doku.php?id=www:documentation:reliability https://wiki.minix3.org/doku.php?id=www:documentation:reliab...
    - pjmlp 7y ago
      Android and Windows as well.
      - panpanna 7y ago
        Linux has been moving towards hybrid drivers for certain components for a while now. I think you can run large parts of the filesystem and network stack in user-land now.
        pjmlp 7y ago
        There is a big difference is that the other OSes are doing it no matter how many pitchforks come into their way, while with Linux you need to find a security conscious distribution.
        Matthias247 7y ago
        I think it’s hard to say that without knowing what’s going on in that companies and how many internal disagreements had been there. And even if Apple now talks about user-Mode drivers - do we even know what percentage they aim for?
beefhash 7y ago
I'm surprised that microkernel-like IPC still hasn't found its way into any *NIX system. The closest we've gotten is System V IPC, which has never really taken off. Is this just difficult to design well or are people genuinely okay with socket(AF_UNIX, SOCK_SEQPACKET, 0)?
- monocasa 7y ago
  It's difficult to tack on to an existing kernel syscall interface. You really want a capability based interface to keep most of the permission checks out of the data plane. And even then modern microkernel IPC goes to extreme lengths. For instance the L4s tend to do crazy stuff like not save all of the registers, but make sure to only clobber the registers that can't be arguments for the call itself. You might be able to tack that onto the front of the syscall interface sort of like how objc_msgsend works, but it'd be a huge pain.
- saagarjha 7y ago
  Mach?
- Matthias247 7y ago
  I don't think it's difficult to design an IPC mechanism. But I DO think it's difficult to design an IPC mechanism that everyone is happy with. Some people want only synchronous IPC, others also asynchronous (out of order responses), some want multicast, some want events without associated request, some want pub/sub, etc. If at the end people continue building their own IPC mechanisms on top of TCP/IP or unix domain sockets then the in-kernel mechanism will just be another thing to maintain. There had been some endaveurs to bring new IPC mechanisms into the Linux Kernel (AF_BUS, KDBUS, BUS1). I think those failed for similar reasons (although I'm not sure were BUS1 now is - the others are definitely discontinued).
  - jacquesm 7y ago
    KISS: QnX MsgSend, MsgReceive, MsgReply. That's really all it takes. http://www.qnx.com/developers/docs/6.5.0/index.jsp?topic=%2Fcom.qnx.doc.neutrino_lib_ref%2Fm%2Fmsgsend.html http://www.qnx.com/developers/docs/6.5.0/index.jsp?topic=%2F... You can go all the way from the lowest level kernel uses to very high level application constructs with that. Re-inventing existing wheels badly is something the software world excels at.
mkj 7y ago
Wonder if their MicroQuanta microsecond granularity Linux scheduler would be useful for other near realtime things like audio or electronics interfacing. I guess the low latency scheduler is a key part of making Snap work in userspace. https://lkml.org/lkml/2019/9/6/177 https://lkml.org/lkml/2019/9/6/177
Y_Y 7y ago
There are already so many software projects called Snap.
- jacquesm 7y ago
  For Google that's a non-issue, they will simply promote their own use of the term over the previous uses.
  - tartrate 7y ago
    Even Google has a limit though, they could've named their programming language "The".
    - jacquesm 7y ago
      I would say taking the '+' character and attempting to make it their own is proof positive that they don't have limits.
ZeroCool2u 7y ago
"Snap has been running in production for over three years, supporting the extensible communication needs of several large and critical systems." Are there many systems using microkernels in production environments? I mean at least at this kind, or a somewhat similar, scale? 3 years seems like a long time, while up until this moment microkernels seemed fairly niche to me and something reserved for more experimental systems. The only one I can think of off the top of my head is Fuchsia.
- mnem 7y ago
  QNX has been around for quite a while and used across many areas: https://blackberry.qnx.com/en https://blackberry.qnx.com/en Edit: The Wikipedia entry has a little less fluff than the official homepage: https://en.wikipedia.org/wiki/QNX https://en.wikipedia.org/wiki/QNX
- als0 7y ago
  Well Snap isn’t a microkernel. It’s a Linux user space with microkernel inspiration. So you can’t use it as a basis for comparison.
- als0 7y ago
  OKL4 is a microkernel that’s shipped in a billion mobile devices.
- eadan 7y ago
  The Secure Enclave processor on Apple devices runs a version of the L4 microkernel [0] https://support.apple.com/en-us/HT209632 https://support.apple.com/en-us/HT209632
skywhopper 7y ago
Oops, the TLS cert on this site expired just nine minutes ago.
- ratsimihah 7y ago
  Is it still safe?
alexnewman 7y ago
Projects need to stop calling themselves snap
- Gibbon1 7y ago
  ans to dead question. > Do they pay people to do this? Yes you fucking bet they do.
  - Gibbon1 7y ago
    > 0 points by Gibbon1 Makes my point
Gonzih 7y ago
looks like certificate for ai.google expired today, what a coincidence!
- njhartwell 7y ago
  And with HSTS, modern browsers will make sure you can't ignore it :) In the meantime https://storage.googleapis.com/pub-tools-public-publication-data/pdf/36f0f9b41e969a00d75da7693571e988996c9f4c.pdf https://storage.googleapis.com/pub-tools-public-publication-... works.
  - judge2020 7y ago
    Not sure about Firefox but in Chrome you can type "thisisunsafe" on the page to bypass even HSTS (you can test at https://subdomain.preloaded-hsts.badssl.com/ https://subdomain.preloaded-hsts.badssl.com/ )
persistent 7y ago
The actual paper, where their TLS certificate is not yet expired, is https://storage.googleapis.com/pub-tools-public-publication-data/pdf/36f0f9b41e969a00d75da7693571e988996c9f4c.pdf https://storage.googleapis.com/pub-tools-public-publication-...
vkaku 7y ago
Unpopular opinion ahead: The deal with user facing libraries like this is that I'd rather they generalize this and expose the existing network library drivers with a uring interface, and the user processes can take care of packet decap the way they want it. Of course, helpful to have the stub to map the PCI BARs to userspace, and hopefully without any message signalled interrupts. These two alone, may be hard to do, with all the existing network drivers out there. Engineering feats like these are good but not helpful to most people unless they are simple and generic enough to work on most devices. I hope the guys who wrote this take note and eventually layer out and open source this library.

144 points by sandGorgon 7y ago

sandGorgon 7y ago

"A change to the kernel stack takes 1-2 months to deploy; a new Snap release gets deployed on a weekly basis."

persistent 7y ago
At many companies rolling out a new kernel in a month would be an outright miracle.

nimrody 7y ago

"While early microkernel work saw significant performanceoverheads attributed to inter-process communication (IPC)and address space changes [15,16,20], such overheads areless significant today. Compared to the uniprocessor systemsof the 80s and 90s, today’s servers contain dozens of cores,which allows microkernel invocation to leverage inter-coreIPC while maintaining application cache locality. This ap-proach can evenimproveoverall performance when there islittle state to communicate across the IPC (common in zero-copy networking) and in avoiding ring switch costs of systemcalls. Moreover, recent security vulnerabilities such as Melt-down [43] force kernel/user address space isolation, even inmonolithic kernels [25]. Techniques like tagged-TLB supportin modern processors, streamlined ring switching hardwaremade necessary with the resurgence of virtualization, andIPC optimization techniques such as those explored in theL4 microkernel [29], in FlexSC [58], and in SkyBridge [44],further allow a modern microkernel to essentially close theperformance gap between direct system calls and indirectsystem calls through IPC."

snvzz 7y ago
You're right to quote some text that addresses the idea that "microkernels are slow", which keeps popping up. To that, I add this article[0], which is sufficient in destroying that idea. [0]https://blog.darknedgy.net/technology/2016/01/01/0/ https://blog.darknedgy.net/technology/2016/01/01/0/
- AlphaSite 7y ago
  Another point is that macOS and iOS are moving towards a user space driver stack and away from Kernel space.
  - snvzz 7y ago
    Because, save few exceptions (e.g. timers, early debug serial...) running drivers in supervisor mode isn't very smart. Drivers tend to have high bug density, and supervisor mode means high potential for damage. Much can be mitigated by running drivers in userspace[0][1], especially with some IOMMU help. [0] https://www.cs.vu.nl/~herbertb/papers/minix3_dsn09.pdf https://www.cs.vu.nl/~herbertb/papers/minix3_dsn09.pdf [1] https://wiki.minix3.org/doku.php?id=www:documentation:reliability https://wiki.minix3.org/doku.php?id=www:documentation:reliab...
  - pjmlp 7y ago
    Android and Windows as well.
    - panpanna 7y ago
      Linux has been moving towards hybrid drivers for certain components for a while now. I think you can run large parts of the filesystem and network stack in user-land now.
      - pjmlp 7y ago
        There is a big difference is that the other OSes are doing it no matter how many pitchforks come into their way, while with Linux you need to find a security conscious distribution.
        Matthias247 7y ago
        I think it’s hard to say that without knowing what’s going on in that companies and how many internal disagreements had been there. And even if Apple now talks about user-Mode drivers - do we even know what percentage they aim for?

beefhash 7y ago

I'm surprised that microkernel-like IPC still hasn't found its way into any *NIX system. The closest we've gotten is System V IPC, which has never really taken off. Is this just difficult to design well or are people genuinely okay with socket(AF_UNIX, SOCK_SEQPACKET, 0)?

monocasa 7y ago
It's difficult to tack on to an existing kernel syscall interface. You really want a capability based interface to keep most of the permission checks out of the data plane. And even then modern microkernel IPC goes to extreme lengths. For instance the L4s tend to do crazy stuff like not save all of the registers, but make sure to only clobber the registers that can't be arguments for the call itself. You might be able to tack that onto the front of the syscall interface sort of like how objc_msgsend works, but it'd be a huge pain.
saagarjha 7y ago
Mach?
Matthias247 7y ago
I don't think it's difficult to design an IPC mechanism. But I DO think it's difficult to design an IPC mechanism that everyone is happy with. Some people want only synchronous IPC, others also asynchronous (out of order responses), some want multicast, some want events without associated request, some want pub/sub, etc. If at the end people continue building their own IPC mechanisms on top of TCP/IP or unix domain sockets then the in-kernel mechanism will just be another thing to maintain. There had been some endaveurs to bring new IPC mechanisms into the Linux Kernel (AF_BUS, KDBUS, BUS1). I think those failed for similar reasons (although I'm not sure were BUS1 now is - the others are definitely discontinued).
- jacquesm 7y ago
  KISS: QnX MsgSend, MsgReceive, MsgReply. That's really all it takes. http://www.qnx.com/developers/docs/6.5.0/index.jsp?topic=%2Fcom.qnx.doc.neutrino_lib_ref%2Fm%2Fmsgsend.html http://www.qnx.com/developers/docs/6.5.0/index.jsp?topic=%2F... You can go all the way from the lowest level kernel uses to very high level application constructs with that. Re-inventing existing wheels badly is something the software world excels at.

mkj 7y ago

Wonder if their MicroQuanta microsecond granularity Linux scheduler would be useful for other near realtime things like audio or electronics interfacing. I guess the low latency scheduler is a key part of making Snap work in userspace. https://lkml.org/lkml/2019/9/6/177 https://lkml.org/lkml/2019/9/6/177

Y_Y 7y ago

There are already so many software projects called Snap.

jacquesm 7y ago
For Google that's a non-issue, they will simply promote their own use of the term over the previous uses.
- tartrate 7y ago
  Even Google has a limit though, they could've named their programming language "The".
  - jacquesm 7y ago
    I would say taking the '+' character and attempting to make it their own is proof positive that they don't have limits.

ZeroCool2u 7y ago

"Snap has been running in production for over three years, supporting the extensible communication needs of several large and critical systems." Are there many systems using microkernels in production environments? I mean at least at this kind, or a somewhat similar, scale? 3 years seems like a long time, while up until this moment microkernels seemed fairly niche to me and something reserved for more experimental systems. The only one I can think of off the top of my head is Fuchsia.

mnem 7y ago
QNX has been around for quite a while and used across many areas: https://blackberry.qnx.com/en https://blackberry.qnx.com/en Edit: The Wikipedia entry has a little less fluff than the official homepage: https://en.wikipedia.org/wiki/QNX https://en.wikipedia.org/wiki/QNX
als0 7y ago
Well Snap isn’t a microkernel. It’s a Linux user space with microkernel inspiration. So you can’t use it as a basis for comparison.
als0 7y ago
OKL4 is a microkernel that’s shipped in a billion mobile devices.
eadan 7y ago
The Secure Enclave processor on Apple devices runs a version of the L4 microkernel [0] https://support.apple.com/en-us/HT209632 https://support.apple.com/en-us/HT209632

skywhopper 7y ago

Oops, the TLS cert on this site expired just nine minutes ago.

ratsimihah 7y ago
Is it still safe?

alexnewman 7y ago

Projects need to stop calling themselves snap

Gibbon1 7y ago
ans to dead question. > Do they pay people to do this? Yes you fucking bet they do.
- Gibbon1 7y ago
  > 0 points by Gibbon1 Makes my point

Gonzih 7y ago

looks like certificate for ai.google expired today, what a coincidence!

njhartwell 7y ago
And with HSTS, modern browsers will make sure you can't ignore it :) In the meantime https://storage.googleapis.com/pub-tools-public-publication-data/pdf/36f0f9b41e969a00d75da7693571e988996c9f4c.pdf https://storage.googleapis.com/pub-tools-public-publication-... works.
- judge2020 7y ago
  Not sure about Firefox but in Chrome you can type "thisisunsafe" on the page to bypass even HSTS (you can test at https://subdomain.preloaded-hsts.badssl.com/ https://subdomain.preloaded-hsts.badssl.com/ )

persistent 7y ago

The actual paper, where their TLS certificate is not yet expired, is https://storage.googleapis.com/pub-tools-public-publication-data/pdf/36f0f9b41e969a00d75da7693571e988996c9f4c.pdf https://storage.googleapis.com/pub-tools-public-publication-...

vkaku 7y ago

Unpopular opinion ahead: The deal with user facing libraries like this is that I'd rather they generalize this and expose the existing network library drivers with a uring interface, and the user processes can take care of packet decap the way they want it. Of course, helpful to have the stub to map the PCI BARs to userspace, and hopefully without any message signalled interrupts. These two alone, may be hard to do, with all the existing network drivers out there. Engineering feats like these are good but not helpful to most people unless they are simple and generic enough to work on most devices. I hope the guys who wrote this take note and eventually layer out and open source this library.