7 ms·

How Raw sockets behave differently in macOS and Linux

111 points by adityaathalye 2y ago

jiehong 2y ago
Speaking of sockets on Mac, anyone knows if there is a documentation about how the network stack on MacOs works? (Routing, firewall, resolving, kernel extensions filtering/offloading, etc).
- donaldihunter 2y ago
  This might be a good place to start: https://developer.apple.com/library/archive/documentation/NetworkingInternet/Conceptual/NetworkingConcepts/Introduction/Introduction.html https://developer.apple.com/library/archive/documentation/Ne...
egberts1 2y ago
Thr stark difference between Linux and macos is that ONLY macos can filter inbound Ether packets by its destination process ID. Becasue BSD raw socket rules. That's how you get Sniffer to be performing waaaaay more intuitively on macos than at all on Linux.
- codetrotter 2y ago
  Do you have a link to Sniffer?
- tankenmate 2y ago
  You can do the same on Linux, just not easily from the command line; ebpf however can do all this and more. Once you have built it you can just create a program that you can use from the command line. There are guides out there to show you how.[0] [0] https://www.datadoghq.com/blog/ebpf-guide/ https://www.datadoghq.com/blog/ebpf-guide/
  - carlmr 2y ago
    I've been meaning to try out eBPF. I read a tutorial on writing a simple perf application in Rust, but I kind of lack the intuition to know what it's good for. Do you know a good overview of what can be done with eBPF?
    - packetlost 2y ago
      Think of it like small, isolated programs that run in a VM that is managed by the kernel and has access to many kernel APIs and datastructures without needing to do syscalls. I believe they can both monitor and manipulate data as it flows through the OS primatives (sockets, file descriptors, etc.).
- Denvercoder9 2y ago
  nftables supports matching by cgroups, which arguably is a better idea than process IDs. https://wiki.nftables.org/wiki-nftables/index.php/Matching_packet_metainformation#Matching_by_security_selectors https://wiki.nftables.org/wiki-nftables/index.php/Matching_p...
deleted 2y ago
[deleted]
donaldihunter 2y ago
It's fascinating watching someone write code for macOS while only referring to Linux and FreeBSD man pages. On macOS, 'man ip' gives all the necessary info about raw sockets and IP_HDRINCL: > Outgoing packets automatically have an IP header prepended to them (based on the destination address and the protocol number the socket is created with), unless the IP_HDRINCL option has been set. > Unlike previous BSD releases, the program must set all the fields of the IP header > Note that the ip_off and ip_len fields are in host byte order.
- chrsig 2y ago
  In a lot of cases, the macos man pages leave a lot to be desired, or don't come up in a google search. I know I'm personally not in the habit of remembering apropros exists, so I can imagine that the first step to manpage discovery being a google search. And most of the time the docs are interchangable enough. Until they're not. At the raw socket layer, I'd still expect the macos docs to be consulted _eventually_.
torginus 2y ago
This is why i f**ing hate macOS. Developers wax poetic about 'oh its a UNIX', but NOTHING works like Linux once you dig into it. Zsh is different from bash, subtle API differences, different filesystem, esp with stuff like /dev or /proc or /etc, no systemd, some cli utils have different flags, subtle differences like this. At least Windows is completely different and there's no expectation of compatibility, but with the mac you can start off thinking things would work, but they don't.
- tikkabhuna 2y ago
  Its been a decade since I had a Mac, but it definitely gave me that uncanny valley feeling. So close that you start expecting certain behaviour. Windows and WSL feels better to me at least.
  - freedomben 2y ago
    Indeed, I've been Linux-only for a bit over 10 years now, and when I have to do some work on a Windows or Mac for work, Windows with WSL feels a lot better than modern macos. You'll encounter the occasional alien thing when Windows details leak through, but for the most part it just behaves like you expect. If I had to pick a mac or windows pc for daily driver, Windows would be an easy choice now (although as coming of age in the 90s when MS was evil, it's mind-blowing to hear me say this all!)
  - swagnikdutta 2y ago
    I'm genuinely interested in understanding this "uncanny valley feeling" you've experienced. Could you elaborate a bit on it?
- simonh 2y ago
  Having cut my teeth on Solaris (both flavours), AIX, HPUX and later Linux in the 90s, get off my lawn! ;)
- OskarS 2y ago
  Linux != UNIX
- 2y ago
eptcyka 2y ago
Why would anyone resort to reading FreeBSD documentation when you can read the source and manpages for macOS?
- swagnikdutta 2y ago
  I guess I was tunnel visioned and lost clarity (and also, common sense). While searching for reasons why the code (from Graham King’s blog) had worked on Linux and not on macOS, a stack overflow answer [1] from 2015, from a person who had also been on the same journey, directed me to that page [2] — that talked about raw socket peculiarities in FreeBSD. So I dived straight into FreeBSD docs. [1]- https://stackoverflow.com/a/32599757/3728336 https://stackoverflow.com/a/32599757/3728336 [2] - https://cseweb.ucsd.edu/~braghava/notes/freebsd-sockets.txt https://cseweb.ucsd.edu/~braghava/notes/freebsd-sockets.txt
  - FujiApple 2y ago
    OP, you may find this [1] “trick” useful. It allows you to dynamically determine the correct byte order for the various IPv4 headers for the platform and thus avoid the need to statically decide on the byte ordering for each platform you intend to target. You may also find this [2] table useful, it shows which platforms allow the combination of IPPROTO_ICMP + IP_HDRINCL so it may be used without elevated privileges. In general, my experience of raw sockets is that they are not very “raw” at all, the OS can and does still perform a variety of modifications and additions to what you send and receive, in highly platform specific and often poorly documented ways. In particular, TCP and raw sockets should generally be avoided. [1] https://github.com/fujiapple852/trippy/blob/master/crates/trippy-core/src/net/platform/byte_order.rs https://github.com/fujiapple852/trippy/blob/master/crates/tr... [2] https://github.com/fujiapple852/trippy/issues/101#issuecomment-1784009140 https://github.com/fujiapple852/trippy/issues/101#issuecomme...
ks2048 2y ago
Apple has a relatively new (I think introduced in 2018) "Network.framework" Swift API [1] for working with raw sockets. Maybe this post is dealing with the layer below - I'm out of my comfort zone with networking - but I recently built a custom protocol using Network and it's working well for me so far. [1] https://developer.apple.com/documentation/network https://developer.apple.com/documentation/network
- kuhsaft 2y ago
  ICMP is lower-level than what Network.framework provides. CFSocket would be the Swift/Obj-C API. https://developer.apple.com/documentation/corefoundation/cfsocket-rg7 https://developer.apple.com/documentation/corefoundation/cfs...
- mzs 2y ago
  There's no NWProtocolRAW class so no: https://developer.apple.com/documentation/network/building-a-custom-peer-to-peer-protocol https://developer.apple.com/documentation/network/building-a...
mzs 2y ago
This post links to some good documentation about sniffer related options: https://wiki.wireshark.org/CaptureSetup/Offloading#checksum-offload https://wiki.wireshark.org/CaptureSetup/Offloading#checksum-... The sysctls are net.link.generic.system.hwcksum_tx and net.link.generic.system.hwcksum_rx on macos.

111 points by adityaathalye 2y ago

Speaking of sockets on Mac, anyone knows if there is a documentation about how the network stack on MacOs works? (Routing, firewall, resolving, kernel extensions filtering/offloading, etc).

donaldihunter 2y ago
This might be a good place to start: https://developer.apple.com/library/archive/documentation/NetworkingInternet/Conceptual/NetworkingConcepts/Introduction/Introduction.html https://developer.apple.com/library/archive/documentation/Ne...

egberts1 2y ago

Thr stark difference between Linux and macos is that ONLY macos can filter inbound Ether packets by its destination process ID. Becasue BSD raw socket rules. That's how you get Sniffer to be performing waaaaay more intuitively on macos than at all on Linux.

codetrotter 2y ago
Do you have a link to Sniffer?
tankenmate 2y ago
You can do the same on Linux, just not easily from the command line; ebpf however can do all this and more. Once you have built it you can just create a program that you can use from the command line. There are guides out there to show you how.[0] [0] https://www.datadoghq.com/blog/ebpf-guide/ https://www.datadoghq.com/blog/ebpf-guide/
- carlmr 2y ago
  I've been meaning to try out eBPF. I read a tutorial on writing a simple perf application in Rust, but I kind of lack the intuition to know what it's good for. Do you know a good overview of what can be done with eBPF?
  - packetlost 2y ago
    Think of it like small, isolated programs that run in a VM that is managed by the kernel and has access to many kernel APIs and datastructures without needing to do syscalls. I believe they can both monitor and manipulate data as it flows through the OS primatives (sockets, file descriptors, etc.).
Denvercoder9 2y ago
nftables supports matching by cgroups, which arguably is a better idea than process IDs. https://wiki.nftables.org/wiki-nftables/index.php/Matching_packet_metainformation#Matching_by_security_selectors https://wiki.nftables.org/wiki-nftables/index.php/Matching_p...

deleted 2y ago

[deleted]

donaldihunter 2y ago

It's fascinating watching someone write code for macOS while only referring to Linux and FreeBSD man pages. On macOS, 'man ip' gives all the necessary info about raw sockets and IP_HDRINCL: > Outgoing packets automatically have an IP header prepended to them (based on the destination address and the protocol number the socket is created with), unless the IP_HDRINCL option has been set. > Unlike previous BSD releases, the program must set all the fields of the IP header > Note that the ip_off and ip_len fields are in host byte order.

chrsig 2y ago
In a lot of cases, the macos man pages leave a lot to be desired, or don't come up in a google search. I know I'm personally not in the habit of remembering apropros exists, so I can imagine that the first step to manpage discovery being a google search. And most of the time the docs are interchangable enough. Until they're not. At the raw socket layer, I'd still expect the macos docs to be consulted _eventually_.

torginus 2y ago

This is why i f**ing hate macOS. Developers wax poetic about 'oh its a UNIX', but NOTHING works like Linux once you dig into it. Zsh is different from bash, subtle API differences, different filesystem, esp with stuff like /dev or /proc or /etc, no systemd, some cli utils have different flags, subtle differences like this. At least Windows is completely different and there's no expectation of compatibility, but with the mac you can start off thinking things would work, but they don't.

tikkabhuna 2y ago
Its been a decade since I had a Mac, but it definitely gave me that uncanny valley feeling. So close that you start expecting certain behaviour. Windows and WSL feels better to me at least.
- freedomben 2y ago
  Indeed, I've been Linux-only for a bit over 10 years now, and when I have to do some work on a Windows or Mac for work, Windows with WSL feels a lot better than modern macos. You'll encounter the occasional alien thing when Windows details leak through, but for the most part it just behaves like you expect. If I had to pick a mac or windows pc for daily driver, Windows would be an easy choice now (although as coming of age in the 90s when MS was evil, it's mind-blowing to hear me say this all!)
- swagnikdutta 2y ago
  I'm genuinely interested in understanding this "uncanny valley feeling" you've experienced. Could you elaborate a bit on it?
simonh 2y ago
Having cut my teeth on Solaris (both flavours), AIX, HPUX and later Linux in the 90s, get off my lawn! ;)
OskarS 2y ago
Linux != UNIX
2y ago

eptcyka 2y ago

Why would anyone resort to reading FreeBSD documentation when you can read the source and manpages for macOS?

swagnikdutta 2y ago
I guess I was tunnel visioned and lost clarity (and also, common sense). While searching for reasons why the code (from Graham King’s blog) had worked on Linux and not on macOS, a stack overflow answer [1] from 2015, from a person who had also been on the same journey, directed me to that page [2] — that talked about raw socket peculiarities in FreeBSD. So I dived straight into FreeBSD docs. [1]- https://stackoverflow.com/a/32599757/3728336 https://stackoverflow.com/a/32599757/3728336 [2] - https://cseweb.ucsd.edu/~braghava/notes/freebsd-sockets.txt https://cseweb.ucsd.edu/~braghava/notes/freebsd-sockets.txt
- FujiApple 2y ago
  OP, you may find this [1] “trick” useful. It allows you to dynamically determine the correct byte order for the various IPv4 headers for the platform and thus avoid the need to statically decide on the byte ordering for each platform you intend to target. You may also find this [2] table useful, it shows which platforms allow the combination of IPPROTO_ICMP + IP_HDRINCL so it may be used without elevated privileges. In general, my experience of raw sockets is that they are not very “raw” at all, the OS can and does still perform a variety of modifications and additions to what you send and receive, in highly platform specific and often poorly documented ways. In particular, TCP and raw sockets should generally be avoided. [1] https://github.com/fujiapple852/trippy/blob/master/crates/trippy-core/src/net/platform/byte_order.rs https://github.com/fujiapple852/trippy/blob/master/crates/tr... [2] https://github.com/fujiapple852/trippy/issues/101#issuecomment-1784009140 https://github.com/fujiapple852/trippy/issues/101#issuecomme...

ks2048 2y ago

Apple has a relatively new (I think introduced in 2018) "Network.framework" Swift API [1] for working with raw sockets. Maybe this post is dealing with the layer below - I'm out of my comfort zone with networking - but I recently built a custom protocol using Network and it's working well for me so far. [1] https://developer.apple.com/documentation/network https://developer.apple.com/documentation/network

kuhsaft 2y ago
ICMP is lower-level than what Network.framework provides. CFSocket would be the Swift/Obj-C API. https://developer.apple.com/documentation/corefoundation/cfsocket-rg7 https://developer.apple.com/documentation/corefoundation/cfs...
mzs 2y ago
There's no NWProtocolRAW class so no: https://developer.apple.com/documentation/network/building-a-custom-peer-to-peer-protocol https://developer.apple.com/documentation/network/building-a...

mzs 2y ago

This post links to some good documentation about sniffer related options: https://wiki.wireshark.org/CaptureSetup/Offloading#checksum-offload https://wiki.wireshark.org/CaptureSetup/Offloading#checksum-... The sysctls are net.link.generic.system.hwcksum_tx and net.link.generic.system.hwcksum_rx on macos.