5 ms·

This doesn't list the biggest gotcha of all, the fact that the cmsg API is incredibly sharp and there are at least 7 ways you can screw it up. Nearly every sing

by jmgao 6y ago

This doesn't list the biggest gotcha of all, the fact that the cmsg API is incredibly sharp and there are at least 7 ways you can screw it up. Nearly every single use of cmsg in Android's source tree was buggy in at least one of these ways: - not aligning the cmsg buffer - leaking fds if more fds are received than expected - blindly dereferencing CMSG_DATA without checking the header - using CMSG_SPACE(fd_count) instead of CMSG_SPACE(fd_count * sizeof(int)) - using CMSG_SPACE instead of CMSG_LEN for .cmsg_len - using CMSG_LEN instead of CMSG_SPACE for .msg_controllen - using a length specified in number of fds instead of bytes It's possible that Android is uniquely bad at this, but I'm skeptical.

the_mitsuhiko 6y ago
Yeah, it's pretty awful. The only remotely sane way to use them is also to use SOCK_SEQPACKET which is not supported everywhere. Another way that works for some cases on Linux at least is to pick up fds from /proc.
Joker_vD 6y ago
Compare this to WinAPI approach: its analogue of dup() (called DuplicateHandle) also accepts a descriptor of a process into which the file descriptor must be duplicated. You need to have the correct access right, of course. However, because in Windows TCP sockets are implemented in the user space, you can't quite use this routine to pass sockets: the part of its state in the user-space won't get duplicated (well, you can't send FILE* into another process in neither Linux nor Windows, right), so there is WSADuplicateSocket function which doesn't actually duplicates the socket, it gives you a struct full of its internal state which you have to send to another process somehow, and then the other process has to re-create the socket manually, by passing this struct into WSASocket call... and I've no idea how the inter-process sharing is then set up, but apparently it is set up, so those two sockets would actually have synchronized state from then on.
- earhart 6y ago
  A more analogous approach would be to use ALPC; its handle-passing support was inspired by unix domain sockets. Unfortunately, it looks like ALPC never got exposed via Win32. Windows TCP sockets are definitely their own thing, which is pretty unfortunate, but Windows exposes a number of other kernel abstractions as handles - processes, mutexes, shared memory sections, events, timers, &c; “Wait for either this particular process to exit or this timer to expire” is trivial. At the time I worked on it, it was one of the features I really preferred about Windows over Unix.
  - Joker_vD 6y ago
    The whole idea of having (awaitable) process descriptors in addition to PIDs is amazing. Suddenly, you don't need wait/waitpid/wait3/wait4/SIGCHLD or even PID 1. When the last process descriptor vanishes, the process itself can be purged from the process table. And Win8 added nested jobs, so now you can basically have inescapable process groups with "if the leader process dies, terminate all its children and grandchildren" semantics. On Linux, I had to just give up and use Docker containers for this: some software simply insists on daemonizing itself too much, to the point that on receiving SIGTERM, it re-sends SIGTERM to everyone in its process group (including you, the unsuspecting parent. That was very "funny" to debug). But of course, the downside of Windows is that the amount of legacy crap and weird quirks and incompatibilities is insane, even more than in x86. "The Windows low-level APIs wouldn't recognize "general" if it showed up in a uniform with stars on the shoulder." (c) njsmith
    - rwmj 6y ago
      It's my understanding that this hole will be filled with pidfd. It's now upstream but there are no glibc bindings yet. https://lwn.net/Articles/801319/ https://lwn.net/Articles/801319/
- muststopmyths 6y ago
  >However, because in Windows TCP sockets are implemented in the user space, That is not what's happening. TCP is entirely in the kernel. You are confusing Winsock with TCP. Even then, Winsock has it's own driver (AFD.sys, which definitely does not stand for Another Fucking Driver) as well. Most of the user-mode Winsock (ws2_32.dll, mswsock.dll, ws2tcpip.dll) implement the Winsock 2.0 spec-related bookkeeping, but Winsock calls essentially boil down to Ioctl()s to AFD.sys in the common case. The thing to realize is that Winsock 2.0 was spec-ed at a time when Windows NT (and 95) allowed vendors to install things called "layered service providers", which were providers that hooked into Winsock and allowed you to intercept/redirect/filter various Winsock calls. Lots of internet filters worked in this way in the early days. Now, given that these 3rd party providers were free to implement their protocol in user space completely, the socket handle you got was not guaranteed to be a kernel handle (Called an IFS handle all over Winsock documentation, for Installable FileSystem handle. ) Thus, WSADuplicateSocket() was a way to allow these providers to re-create a socket based on whatever provider-specific information was passed in through IPC. Going back to your original claim, you can actually take a native AFD socket handle to a TCP connection and duplicate it (with DuplicateHandle) into another processes. It will also probably work just fine, but no one at Microsoft is going to officially support this usage, for the above-mentioned compatibility reasons, even though Winsock layered providers are officially deprecated now. Edit: clarified what I meant by duplicate in last paragraph.
  - Joker_vD 6y ago
    Thanks for clarifying! And I guess one could still implement an LSP in the user mode while supporting DuplicateHandle directly, without all the WSADuplicateSocket stuff, by having a service with all the custom logic, and returning pipes into this service as socket handles. That's basically what a microkernel would do as well, right?
    - muststopmyths 6y ago
      I guess you technically could, because the handle your LSP returns is entirely up to it. Haven't thought about microkernels in a couple of decades :), but yes, I can see how what you are saying could work since the handle will technically be a duplicatable kernel handle. It might be better for the service (or subsystem in NT microkernel terminology) to have a driver so that the client can reference it with a proper, dedicated handle, but that's a bit beyond my wheelhouse as to which approach is better. Edit: Completely forgot about ws2ifsl.sys which is a driver that exists just to give Non-IFS providers a kernel handle with the WPUCreateSocketHandle() call !
huhtenberg 6y ago
I have no way of verifying your Android claim, but this API is really well-documented including examples of how to use it properly. Here's the OpenBSD version, which is quite typical - https://man.openbsd.org/CMSG_DATA.3 https://man.openbsd.org/CMSG_DATA.3. It's literally a no-brainer to code it all correctly on the first try. So the scary stuff on your list is exactly of the same nature as using printf() without reading the spec and then calling it "incredibly sharp".

by jmgao 6y ago

the_mitsuhiko 6y ago

Yeah, it's pretty awful. The only remotely sane way to use them is also to use SOCK_SEQPACKET which is not supported everywhere. Another way that works for some cases on Linux at least is to pick up fds from /proc.

Joker_vD 6y ago

Compare this to WinAPI approach: its analogue of dup() (called DuplicateHandle) also accepts a descriptor of a process into which the file descriptor must be duplicated. You need to have the correct access right, of course. However, because in Windows TCP sockets are implemented in the user space, you can't quite use this routine to pass sockets: the part of its state in the user-space won't get duplicated (well, you can't send FILE* into another process in neither Linux nor Windows, right), so there is WSADuplicateSocket function which doesn't actually duplicates the socket, it gives you a struct full of its internal state which you have to send to another process somehow, and then the other process has to re-create the socket manually, by passing this struct into WSASocket call... and I've no idea how the inter-process sharing is then set up, but apparently it is set up, so those two sockets would actually have synchronized state from then on.

earhart 6y ago
A more analogous approach would be to use ALPC; its handle-passing support was inspired by unix domain sockets. Unfortunately, it looks like ALPC never got exposed via Win32. Windows TCP sockets are definitely their own thing, which is pretty unfortunate, but Windows exposes a number of other kernel abstractions as handles - processes, mutexes, shared memory sections, events, timers, &c; “Wait for either this particular process to exit or this timer to expire” is trivial. At the time I worked on it, it was one of the features I really preferred about Windows over Unix.
- Joker_vD 6y ago
  The whole idea of having (awaitable) process descriptors in addition to PIDs is amazing. Suddenly, you don't need wait/waitpid/wait3/wait4/SIGCHLD or even PID 1. When the last process descriptor vanishes, the process itself can be purged from the process table. And Win8 added nested jobs, so now you can basically have inescapable process groups with "if the leader process dies, terminate all its children and grandchildren" semantics. On Linux, I had to just give up and use Docker containers for this: some software simply insists on daemonizing itself too much, to the point that on receiving SIGTERM, it re-sends SIGTERM to everyone in its process group (including you, the unsuspecting parent. That was very "funny" to debug). But of course, the downside of Windows is that the amount of legacy crap and weird quirks and incompatibilities is insane, even more than in x86. "The Windows low-level APIs wouldn't recognize "general" if it showed up in a uniform with stars on the shoulder." (c) njsmith
  - rwmj 6y ago
    It's my understanding that this hole will be filled with pidfd. It's now upstream but there are no glibc bindings yet. https://lwn.net/Articles/801319/ https://lwn.net/Articles/801319/
muststopmyths 6y ago
>However, because in Windows TCP sockets are implemented in the user space, That is not what's happening. TCP is entirely in the kernel. You are confusing Winsock with TCP. Even then, Winsock has it's own driver (AFD.sys, which definitely does not stand for Another Fucking Driver) as well. Most of the user-mode Winsock (ws2_32.dll, mswsock.dll, ws2tcpip.dll) implement the Winsock 2.0 spec-related bookkeeping, but Winsock calls essentially boil down to Ioctl()s to AFD.sys in the common case. The thing to realize is that Winsock 2.0 was spec-ed at a time when Windows NT (and 95) allowed vendors to install things called "layered service providers", which were providers that hooked into Winsock and allowed you to intercept/redirect/filter various Winsock calls. Lots of internet filters worked in this way in the early days. Now, given that these 3rd party providers were free to implement their protocol in user space completely, the socket handle you got was not guaranteed to be a kernel handle (Called an IFS handle all over Winsock documentation, for Installable FileSystem handle. ) Thus, WSADuplicateSocket() was a way to allow these providers to re-create a socket based on whatever provider-specific information was passed in through IPC. Going back to your original claim, you can actually take a native AFD socket handle to a TCP connection and duplicate it (with DuplicateHandle) into another processes. It will also probably work just fine, but no one at Microsoft is going to officially support this usage, for the above-mentioned compatibility reasons, even though Winsock layered providers are officially deprecated now. Edit: clarified what I meant by duplicate in last paragraph.
- Joker_vD 6y ago
  Thanks for clarifying! And I guess one could still implement an LSP in the user mode while supporting DuplicateHandle directly, without all the WSADuplicateSocket stuff, by having a service with all the custom logic, and returning pipes into this service as socket handles. That's basically what a microkernel would do as well, right?
  - muststopmyths 6y ago
    I guess you technically could, because the handle your LSP returns is entirely up to it. Haven't thought about microkernels in a couple of decades :), but yes, I can see how what you are saying could work since the handle will technically be a duplicatable kernel handle. It might be better for the service (or subsystem in NT microkernel terminology) to have a driver so that the client can reference it with a proper, dedicated handle, but that's a bit beyond my wheelhouse as to which approach is better. Edit: Completely forgot about ws2ifsl.sys which is a driver that exists just to give Non-IFS providers a kernel handle with the WPUCreateSocketHandle() call !

huhtenberg 6y ago

I have no way of verifying your Android claim, but this API is really well-documented including examples of how to use it properly. Here's the OpenBSD version, which is quite typical - https://man.openbsd.org/CMSG_DATA.3 https://man.openbsd.org/CMSG_DATA.3. It's literally a no-brainer to code it all correctly on the first try. So the scary stuff on your list is exactly of the same nature as using printf() without reading the spec and then calling it "incredibly sharp".