7 ms·

I think quite a few of the hyperbolic comments here haven't seen any of this video and are just reacting to the title. From what I can gather from watching thi

I think quite a few of the hyperbolic comments here haven't seen any of this video and are just reacting to the title. From what I can gather from watching this so far is, it just sounds like a layer you can optionally choose to use on-top of sockets that manages things like: TLS, multiple networks (4G/3G/Wi-Fi), DNS resolving, proxies, etc for you so you don't need to write it all yourself from scratch every time. Maybe I'm wrong about that, or maybe none of this is actually as hard as it sounds in practice. But this seems like it might be atleast some what useful.

rad_gruchalski 8y ago
Exactly as you say. I'm not into apple products from the development perspective at this moment in time. But I did watch the video. My first thought was: I like it a lot, can I have it in every language. The framework looks really good. They mention at the end of the video that some of the things might not be very well polished but I'm sure it's all going to get there in the end.
carlosrg 8y ago
> I think quite a few of the hyperbolic comments That's what you get from things like deprecating OpenGL and replacing it with a non-multiplatform alternative: bad reputation. They should ask Microsoft if it's easy to get rid of if once you have it.
- vvanders 8y ago
  Ironically Microsoft has a non-standard API for higher performance networking for quite a while, AKA I/O Completion Ports. It's just they built it right by making it work with existing socket handles rather than fragmenting the platform with a new API.
  - daeken 8y ago
    Microsoft really nailed it with IOCPs. They added them on pretty seamlessly to existing APIs, from file ops to sockets, and made them fit into existing workflows without developers having to waste mental cycles. They've screwed up a lot of things, but man, they got that right.
  - kluqle 8y ago
    IOCP is a pretty generic async IO API, not specific to networking - it is similar to epoll on linux and kqueue on BSDs and MacOS. Network.framework is an API which gives you more direct access to, and better support for TLS, TCP, and UDP.
    - josephg 8y ago
      > [IOCP] is similar to epoll on linux and kqueue on BSDs and MacOS Similar, but the differences matter. kqueue is readyness-oriented. It tells you that the driver is ready to queue more writes. But you still need to call fsync and friends to confirm that your data has actually been written out. There's no non-blocking fsync. Servers like Redis run an extra thread just to call fsync; and eat all the performance problems that entails. In contrast, IOCP is completion-oriented. There's no need for any blocking fsync calls in order to find out when data has been written. Its way more granular - the OS can reorder writes. And it doesn't suffer from fsync's awful non-local error handling problems. Honestly I'm surprised there's no equivalent for linux / BSD. (I mean, we have AIO but its really not good enough). IOCP is a fantastic API for high performance databases and servers. It enables faster and simpler server code. We should collectively get on it.
      - dwaite 8y ago
        Yeah, unix usually is based on reactor pattern async I/O, while IOCP is proactor pattern. You can build a proactor pattern on top of a reactor pattern, but that implementation needs to provide quite a bit - more coordination of pending memory/state and probably needing back pressure support (which I never really understood if Microsoft supplied or if you had to track that yourself). I know Microsoft used to have several IOCP-related patents; I believe I remember one for event scheduling for multi-threaded locality (preferring to dispatch a completion to the same thread that made the original request).
      - eps 8y ago
        Write flushing is not a good example. You still need FlushFileBuffers even with IOCP if files are opened in buffered mode (which is the default). While you will know when a write was completed, an explicit flush is still needed to make sure it’s written through. So it’s pretty much exactly the same case as with fsync(). Alternatively, you can use the write-through mode, but that kills performance like that, especially with a random write pattern. IOCP is basically a better/cleaner abstracted epoll() that also doubles as a manager of a worker thread pool.
- _wmd 8y ago
  I'm not a graphics programmer, but everything I've read about OpenGL tells me moving to Metal isn't some embrace/extend maneuver but more likely due to the fact OpenGL is steaming ancient garbage. Kind of like the types who once complained that MacOS wasn't built on X Window (chortle) BSD sockets isn't a bastion of extensible interface design, it basically amounts to one giant ioctl() in every operating system. Darwin has weird escape hatches built into it (AF_SYSTEM) much like Linux netlink, and protocol parameter documentation is strewn across 5+ man pages each documenting a different set of magic value combinations for non-portably controlling some part of the stack. TCP_USER_TIMEOUT? That's one of many, many Linux socket options, but it's part of our precious so-portable socket interface we must maintain forever! OS X chooses to bury its system call interface too, it's always been considered a private detail of the C library. I see no reason why socket() and friends couldn't similarly be forgotten. If some clean 21st century redesign of an ancient API has knock-on effects for the structure of code in open source projects across the ecosystem, that's probably a good thing in the long run. Progress never happens by hugging the past, especially when that past is ugly and filled with horrors Try not to look at this as "hey, MacOS is breaking precious UNIX", but more as "hey look, why can't Bonjour+multipath TCP be a one liner on Linux too?"
  - IshKebab 8y ago
    OpenGL is certainly old and warty but it is nothing like X11. I think people would be ok with Apple dropping OpenGL if they supported Vulkan, but their motivation and desire to lock people into Apple-only APIs is so obvious and transparent... Why would you support it? At least Microsoft works with third parties and acknowledges the existence of some standards. Apple are always like "Here is the Apple way. I mean, the way. Because there is only Apple. Apple is the whole world and nothing else exists."
    - anonfunction 8y ago
      Which makes sense from a business point of view. Obviously that doesn’t always align with a hacker mindset but there is a reason Apple is worth almost a trillion dollars.
mpweiher 8y ago
> on-top of sockets That would have been my guess as well, but looking at the slides it appears to actually have a separate "user space networking" path (p105-108 in the PDF). So not entirely sure it's built on top, at least not API-wise. Of course, there are sockets underneath in the OS.
- chadcmulligan 8y ago
  it seems they've written their own socket code, at least from the video. It's not like BSD was handed down from on high, it certainly could do with an update to make it easier.
- mrpippy 8y ago
  Correct. On iOS/tvOS, it looks like Network.framework (and NSURLSession) is using the transport stack in-process, not layered on top of BSD sockets. On macOS, I believe they're still using the traditional in-kernel transport stack for everything. They want to move to user-space for macOS though, this is why Network Kernel Extensions (NKEs) are deprecated (which started last year). I wouldn't be surprised if macOS uses the user-space stack starting next year and NKEs are fully gone (this also corresponds with 32-bit processes being gone)
  - eeks 8y ago
    It is limited to UDP for the moment (at least AFAICT). I would be very surprised if they ever offer user-space TCP. It's a much different beast than UDP to handle right in user-space. Also, the 30% less overhead is underwhelming. I would have expected much better improvement than that. That being said, if their measurement includes the encryption layer then the I/O benefit may be overshadowed by the encryption overhead. Still, pretty cool stuff to make publicly available.
    - doctorsher 8y ago
      According to their slides from WWDC last year, it also offers TCP in userspace.

rad_gruchalski 8y ago

Exactly as you say. I'm not into apple products from the development perspective at this moment in time. But I did watch the video. My first thought was: I like it a lot, can I have it in every language. The framework looks really good. They mention at the end of the video that some of the things might not be very well polished but I'm sure it's all going to get there in the end.

carlosrg 8y ago

> I think quite a few of the hyperbolic comments That's what you get from things like deprecating OpenGL and replacing it with a non-multiplatform alternative: bad reputation. They should ask Microsoft if it's easy to get rid of if once you have it.

vvanders 8y ago
Ironically Microsoft has a non-standard API for higher performance networking for quite a while, AKA I/O Completion Ports. It's just they built it right by making it work with existing socket handles rather than fragmenting the platform with a new API.
- daeken 8y ago
  Microsoft really nailed it with IOCPs. They added them on pretty seamlessly to existing APIs, from file ops to sockets, and made them fit into existing workflows without developers having to waste mental cycles. They've screwed up a lot of things, but man, they got that right.
- kluqle 8y ago
  IOCP is a pretty generic async IO API, not specific to networking - it is similar to epoll on linux and kqueue on BSDs and MacOS. Network.framework is an API which gives you more direct access to, and better support for TLS, TCP, and UDP.
  - josephg 8y ago
    > [IOCP] is similar to epoll on linux and kqueue on BSDs and MacOS Similar, but the differences matter. kqueue is readyness-oriented. It tells you that the driver is ready to queue more writes. But you still need to call fsync and friends to confirm that your data has actually been written out. There's no non-blocking fsync. Servers like Redis run an extra thread just to call fsync; and eat all the performance problems that entails. In contrast, IOCP is completion-oriented. There's no need for any blocking fsync calls in order to find out when data has been written. Its way more granular - the OS can reorder writes. And it doesn't suffer from fsync's awful non-local error handling problems. Honestly I'm surprised there's no equivalent for linux / BSD. (I mean, we have AIO but its really not good enough). IOCP is a fantastic API for high performance databases and servers. It enables faster and simpler server code. We should collectively get on it.
    - dwaite 8y ago
      Yeah, unix usually is based on reactor pattern async I/O, while IOCP is proactor pattern. You can build a proactor pattern on top of a reactor pattern, but that implementation needs to provide quite a bit - more coordination of pending memory/state and probably needing back pressure support (which I never really understood if Microsoft supplied or if you had to track that yourself). I know Microsoft used to have several IOCP-related patents; I believe I remember one for event scheduling for multi-threaded locality (preferring to dispatch a completion to the same thread that made the original request).
    - eps 8y ago
      Write flushing is not a good example. You still need FlushFileBuffers even with IOCP if files are opened in buffered mode (which is the default). While you will know when a write was completed, an explicit flush is still needed to make sure it’s written through. So it’s pretty much exactly the same case as with fsync(). Alternatively, you can use the write-through mode, but that kills performance like that, especially with a random write pattern. IOCP is basically a better/cleaner abstracted epoll() that also doubles as a manager of a worker thread pool.
_wmd 8y ago
I'm not a graphics programmer, but everything I've read about OpenGL tells me moving to Metal isn't some embrace/extend maneuver but more likely due to the fact OpenGL is steaming ancient garbage. Kind of like the types who once complained that MacOS wasn't built on X Window (chortle) BSD sockets isn't a bastion of extensible interface design, it basically amounts to one giant ioctl() in every operating system. Darwin has weird escape hatches built into it (AF_SYSTEM) much like Linux netlink, and protocol parameter documentation is strewn across 5+ man pages each documenting a different set of magic value combinations for non-portably controlling some part of the stack. TCP_USER_TIMEOUT? That's one of many, many Linux socket options, but it's part of our precious so-portable socket interface we must maintain forever! OS X chooses to bury its system call interface too, it's always been considered a private detail of the C library. I see no reason why socket() and friends couldn't similarly be forgotten. If some clean 21st century redesign of an ancient API has knock-on effects for the structure of code in open source projects across the ecosystem, that's probably a good thing in the long run. Progress never happens by hugging the past, especially when that past is ugly and filled with horrors Try not to look at this as "hey, MacOS is breaking precious UNIX", but more as "hey look, why can't Bonjour+multipath TCP be a one liner on Linux too?"
- IshKebab 8y ago
  OpenGL is certainly old and warty but it is nothing like X11. I think people would be ok with Apple dropping OpenGL if they supported Vulkan, but their motivation and desire to lock people into Apple-only APIs is so obvious and transparent... Why would you support it? At least Microsoft works with third parties and acknowledges the existence of some standards. Apple are always like "Here is the Apple way. I mean, the way. Because there is only Apple. Apple is the whole world and nothing else exists."
  - anonfunction 8y ago
    Which makes sense from a business point of view. Obviously that doesn’t always align with a hacker mindset but there is a reason Apple is worth almost a trillion dollars.

mpweiher 8y ago

> on-top of sockets That would have been my guess as well, but looking at the slides it appears to actually have a separate "user space networking" path (p105-108 in the PDF). So not entirely sure it's built on top, at least not API-wise. Of course, there are sockets underneath in the OS.

chadcmulligan 8y ago
it seems they've written their own socket code, at least from the video. It's not like BSD was handed down from on high, it certainly could do with an update to make it easier.
mrpippy 8y ago
Correct. On iOS/tvOS, it looks like Network.framework (and NSURLSession) is using the transport stack in-process, not layered on top of BSD sockets. On macOS, I believe they're still using the traditional in-kernel transport stack for everything. They want to move to user-space for macOS though, this is why Network Kernel Extensions (NKEs) are deprecated (which started last year). I wouldn't be surprised if macOS uses the user-space stack starting next year and NKEs are fully gone (this also corresponds with 32-bit processes being gone)
- eeks 8y ago
  It is limited to UDP for the moment (at least AFAICT). I would be very surprised if they ever offer user-space TCP. It's a much different beast than UDP to handle right in user-space. Also, the 30% less overhead is underwhelming. I would have expected much better improvement than that. That being said, if their measurement includes the encryption layer then the I/O benefit may be overshadowed by the encryption overhead. Still, pretty cool stuff to make publicly available.
  - doctorsher 8y ago
    According to their slides from WWDC last year, it also offers TCP in userspace.