11 ms·

BSD socket API revamp

119 points by rumcajz 10y ago

comex 10y ago
Very interesting idea overall. I've long thought the current segregation of network protocols between userland and the kernel is rather arbitrary and inflexible - why should reliability (TCP) be in the kernel but security (TLS) be in userland? I don't think the idea of requiring Go-style lightweight threading is viable. Lightweight threading has many nice properties, but also inherently requires more expensive operations to deal with split stacks, and tends to use more memory than a manual approach. It's also rather poorly supported in general by existing languages and OSes (other than Go), while any new OS-level socket API should be as universally accessible as the existing one. In particular, many scripting-ish languages either have no concept of threading at all or only support isolated threads whose communication is limited to relatively slow message passing. In theory the language implementations could write a C shim to expose a non-blocking interface around blocking underlying operations - this is what libuv does today for file I/O, for instance - but the result would be a lot of unnecessary overhead.
- tptacek 10y ago
  How would a design with TCP done entirely in userland work? A given socket address might in one instant belong to uid 10, and in the next, after uid=10's connection closes, to uid 20. (I'm not challenging you, just asking what you think the semantics should look like). Worth adding for everyone else, though I'm sure you know: high speed network implementations already pull a lot of this stuff into userland.
  - zrm 10y ago
    Basically the same as SOCK_DGRAM works except that send() and recv() expect/provide the TCP header before the transport payload. If a user process calls bind() on some address/port then it gets all TCP packets for that local address/port. If it calls connect() then it gets only the ones that match both the local and remote address/port.
    - 0x0 10y ago
      With packet retransmissions and out-of-order packet delivery, wouldn't there be a risk that the client software at uid=10 sees a clean shutdown and disconnects, but uid=20 instantly binds with the same ports and steals a bunch of delayed packets?
      - zrm 10y ago
        That happens with UDP already and is another reason everything ought to be encrypted with session keys. But the normal TCP solution of the OS holding released ports in TIME_WAIT for a couple of minutes still works as ever.
  - comex 10y ago
    There are a few different ways to answer that question. First, I believe that for most applications, it's not that TCP should be in userland; rather, TLS should be in the kernel, or otherwise system managed. See the wall of text I just replied to someone else with. Second, if you do want to customize the transport layer, which is of course fairly common today - multiplayer games often go for some custom "reliable UDP" protocol that drops ordering guarantees to improve latency, and then there's Chromium with QUIC, and uTorrent ‎µTP, and WebRTC... well, the reality of networking today is that that has to be done on top of UDP rather than IP directly. I doubt there's much point writing a new API for TCP itself that exposes all the bells and whistles, because the design of TCP is dated. Rather, there should be more standardized APIs for applications to more easily use replacements for TCP that run over UDP. Ideally I should be able to switch from TCP to QUIC just by changing a few lines of code. (Edit: By the way, if we could redo networking protocols from scratch, I think ports ought to exist at the IP layer rather than being replicated in TCP and UDP, and then UDP could be abolished entirely. But we can't.) Actually, that ties into high performance networking too. As you know, those alternate stacks require userland to talk directly or near-directly to hardware, meaning you just have to give up on different ports belonging to different applications. But beyond that, one thing holding those stacks back is that they require a lot of custom code and don't work with existing applications. With a standardized networking API that supported userspace plugins, it could be possible to add "DPDK+rumptcpip" as an option next to "kernel TCP" and "QUIC", and configure any random process to use it.
    - zrm 10y ago
      > I doubt there's much point writing a new API for TCP itself that exposes all the bells and whistles, because the design of TCP is dated. Rather, there should be more standardized APIs for applications to more easily use replacements for TCP that run over UDP. The advantage TCP has over UDP is that middleboxes know what TCP FIN is and so are willing to use much longer timeouts for TCP sessions. For example the default Linux connection tracking timeout for established TCP connections is five days but for UDP streams it's three minutes. So if you need a long-lived session to receive event-based messages your choices are to use UDP with a mapped port using NAT-PMP or PCP (ideal but not always available), use UDP with frequent keepalives (expensive), or use TCP. Being able to do TCP in userspace would be very useful for any VPN-like thing because you could get the long timeouts but still deliver packets immediately even if an earlier one was lost, and avoid the TCP-over-TCP performance degradation by deferring congestion control to the inner packets.
  - justincormack 10y ago
    The thing is, if addresses are scarce, you do need something (typically a kernel) to allocate them. Port numbers on a single IP are scarce. IPv6 addresses, if each machine has a /64 though are not, and it becomes much easier, each (64 bit!) pid can have its own IP with neither being reused, so routing can be dumb. (And/or encrypt everything, keys are not a scarce resource either).
  - rumcajz 10y ago
    Can be done for example by having a address allocation daemon running in user space. The socket library would create an IPC connection to the daemon and ask for an address/port. The daemon would reserve the address/port and pass it back. The daemon would have to health check the applications so that it can return the addresses back to the pool in case of failure.
- aftbit 10y ago
  The big reason security is in userspace is because it requires policy decisions. How do we determine which remote users to trust? WoT, cert chains, trust on first use? There is kernel-mode security (IPsec) but it turns out that userspace is better equipped to make those decisions.
  - comex 10y ago
    Actually, policy decisions are a big part of why I think TLS should be "in the kernel". It doesn't have to literally be in the kernel - it could be a userspace daemon instead - but that's an implementation detail. What matters is that just like with TCP, there should be one high-level API, provided by the system, which all applications are expected to use. Scenario: You want to temporarily ignore an invalid certificate. (Yes, I know this is usually a bad idea, but sometimes it’s necessary.) Today: Every application needs its own flag, environment variable, configuration option, and/or UI element to disable validation. If you’re unlucky, there won’t be one. If you’re really unlucky, there won’t be any way to enable validation; even in 2017 there are still lots of poorly written applications like that. Centralized: There should be a standard environment variable that works with everything, and/or maybe a “security center” applet where you can add a temporary system-wide exemption per-host, or something like that. Scenario: You want to sniff a secure connection for debugging purposes. Today: You have to use a MITM proxy, which is gross and needs setup and breaks things, and requires trusting invalid certificates, which doesn’t always work; see above and below. Centralized: The kernel could just hand decrypted packets to packet sniffing programs, allowing for purely passive observation. Scenario: You want to ensure that you’re using high-quality ciphersuites. Today: Up to the application, usually not customizable, so browsers will do the right thing and other applications will have some list the developer copy+pasted that seemed to work. Welp. Centralized: The system should be able to specify policy for all applications, with the option for the user to override it. After all, it’s almost never actually desirable for a cipher suite to be supported by one application and not another - like, it’s okay if my mail client connects to Google insecurely, but not if my instant messaging client does. Scenario: You want to add a custom trusted CA, or generally you want different applications to be consistent in their policy decisions. Today: Different TLS libraries have their own trust store formats and trust rules. If you’re lucky, your TLS library of choice will have some way to use the system trust store, on Windows and macOS where such a thing exists, although who knows if it’ll properly replicate the various rules browsers like to impose, e.g. requiring certificate transparency. If you’re unlucky, you end up with things like Java having its own trust store, or wget on macOS refusing to download any https URLs until you go install some .crt file and point wget to the right path. In both cases, you end up with a separate list of trusted certificates, that requires a separate process if you want to add a custom one, or upgrade to remove newly distrusted roots, etc. Centralized: Well, there’s already some centralization with system trust stores, but ideally not would those be supported universally, the entire process of “should this TLS connection be established” could be defined and maintained in one place. Scenario: On a server, you want to run a program that accepts TLS connections. Today: Each program needs its own copy of the private key and its own TLS configuration, which often doesn’t expose the full range of settings it’s desirable to customize. Centralized: The system holds the keys (reducing both configuration burden and the danger of leaks); applications can just say “listen on port X using TLS”, and send/receive plaintext on the resulting sockets, and encryption will be handled automatically. ... I admit there are also some downsides of a centralized design. One of them is updates: these days, on most systems, users update their browsers a lot more than they update the OS, so the browser is likely to be up to date while the OS is not. But "at least the browser is secure, even if other apps are not" is not exactly an ideal outcome, and this too seems like an artifact of history... there's no reason that OS components can't auto-update like browsers do.
- jstimpfle 10y ago
  I've come to think that the whole distinction "kernel vs userland" is inflexible. There was a thread here on HN a few days ago, and it showed some clear deficits of conventional OS, for example in processes. Speaking of lack of composability of clients and linked-in libraries, speaking of IPC and uncontrolled implicit environment inheritance (Docker... :<), speaking of the ad-hoc security model (Unix permissions, setuid hacks, and more recently extended attributes). There's just a single trust boundary in Unix. What if the system was instead just a graph of interacting entities, each managing control over some resource? The "userland process" would just be another entity that is in control of another (more ephemeral) resource. We wouldn't even separate address spaces! Security by careful propagation through fine-grained trust boundaries (instead of globally accessible resource managers that have a hard time deciding whether a requesting entity should be authorized). Think svn vs git. And my understanding is that that's much of what Lisp Machines were about...
  - morb 10y ago
    Could you please link to the mentioned thread?
    - jstimpfle 10y ago
      https://news.ycombinator.com/item?id=13621623 https://news.ycombinator.com/item?id=13621623 If you search for my name there, you will find at least discussions about file modes and also why I claim that processes are a failed abstraction (it was really another user who pointed that out).
  - nwmcsween 10y ago
    Define 'separate address spaces', you have linear address space (virtual) and kernel/user 'address spaces' which are address spaces but not in the way linear address spaces are. It would be easier to simply move almost everything not needed by the kernel into 'userspace' and have some sort of capability system to enforce access rights.
    - jstimpfle 10y ago
cafxx 10y ago
Am I missing something? It seems to propose to ditch all non-blocking APIs under the assumption that "runtimes" will provide lightweight concurrency. But AFAIK most (all?) runtimes that provide lightweight concurrency make extensive use of the non-blocking network APIs to work - including Golang, that is even cited as an example: During the decades since BSD sockets were first introduced the way they are used have changed significantly. While in the beginning the user was supposed to fork a new process for each connection and do all the work using simple blocking calls nowadays they are expected to keep a pool of connections, check them via functions like poll() or kqueue() and dispatch any work to be done to one of the worker threads in a thread pool. In other words, user is supposed to do both network and CPU scheduling. [...] To address this problem, this memo assumes that there already exists an efficient concurrency implementation where forking a new lightweight process takes at most hundreds of nanoseconds and context switch takes tens of nanoseconds. Note that there are already such concurrency systems deployed in the wild. One well-known example are Golang's goroutines but there are others available as well.
- tptacek 10y ago
  Apart from the layering, which really has nothing to do with the sockets interface and can be (and has been, repeatedly, as this author knows) built today on top of the sockets interface, I don't really see what's interesting at all about this interface; it seems like a simple restatement of the existing blocking socket API.
  - rumcajz 10y ago
    It tries to describe in detail what are the minimal semantic preconditions for protocol implementations to be composable (both vertically and horizontally) while at the same time being efficient (so that a stack with 20 microprotocols won't grind to a halt). The most interesting parts are handshakes, how they are conflated among multiple protocols as well as semantics of error handling. But the real proposition here is to have an ecosystem of microprotocols (e.g. a CRLF protocol that splits bytestream into messages using CR+LF as a delimiter) which, by the virtue of having common, well understood semantics, could by composed into ad-hoc networking stacks according to users' needs.
Animats 10y ago
Why is this being proposed to the IETF? It's not an issue visible at the wire protocol level. It doesn't affect interoperability. Sockets belong to the POSIX spec, and to some extent to language specs.
- luchs 10y ago
  There are already other RFCs regarding the socket API, such as RFC3493 (IPv6 socket API).
  - dfox 10y ago
    There even are RFCs (RFC 2783 comes to mind) that specify OS-level APIs that are only tangentialy related to networking.
shabbyrobe 10y ago
Didn't notice this on my first read, but this is by Martin Sustrik. Sustrik is a former ZeroMQ contributor and, more recently, author of the abandoned zmq alternative nanomsg [1], as well as the libmill [2] and libdill [3] lightweight concurrency libraries. [1] http://nanomsg.org/ [2] http://libmill.org/ [3] http://libdill.org/
- jey 10y ago
  Is nanomsg truly abandoned? I was hoping it was simply converging toward stability.
  - shabbyrobe 10y ago
    Hmm true, my language was a bit ambiguous. I meant it was abandoned by Sustrik [1], even if some of the community are doing the best they can manage to keep it alive. [1] http://www.freelists.org/post/nanomsg/New-fork-was-Re-Moving-forward
    - jey 10y ago
      That seems a bit outdated; looks like main nanomsg repo has regular commits from gdamore now[1], so maybe the fork was resolved? 1. https://github.com/nanomsg/nanomsg/commits/master https://github.com/nanomsg/nanomsg/commits/master
      - shabbyrobe 10y ago
        Yep it was eventually resolved midway through 2016, D'Amore stepped up, then stepped down, then stepped up again. The link I cited is not a commentary on the current status of nanomsg, it is more to illustrate Sustrik's role in the situation. If, ultimately, this discussion is about Sustrik's IETF proposal, it is important to consider his relationship with previous similar (zmq/nanomsg) or related (libmill/libdill) projects which have borne his name.
      - RickHull 10y ago
        Also https://github.com/go-mangos/mangos https://github.com/go-mangos/mangos
luchs 10y ago
Something this API gets right is having a unified interface for both IPv4 and IPv6. With the sockets API, you have to decide for one of them. Changing isn't easy as the constants and structures are all named differently. While it's possible to use IPv6 sockets for IPv4 connections, this doesn't cover all use-cases. For example, you can't do IPv4 broadcast with an IPv6 socket. Additionally, as most examples are written for the classic IPv4 API, that's what everyone uses per default. Later on, when people complain about missing IPv6 support, they are turned down because it's a ton of work to change.
- dfox 10y ago
  For majority of applications, supporting IPv6 boils down to using getaddrinfo()/getnameinfo() instead of gethostbyname()/gethostbyaddr(), which results in code that supports both IPv4 and IPv6 and is simpler than the IPv4-only original.
  - rwmj 10y ago
    The catch is you also have to be prepared to listen on multiple sockets. There are lots of servers out there which get this wrong, including qemu-nbd. For reference here's my implementation of this which (I think) gets it right: https://github.com/libguestfs/nbdkit/blob/master/src/sockets.c#L105 https://github.com/libguestfs/nbdkit/blob/master/src/sockets...
  - mvkg 10y ago
    A good dual-stack implementation should also support Happy Eyeballs which does not result in simpler code.
- deleted 10y ago
  [deleted]
- slrz 10y ago
  Changing isn't easy as the constants and structures are all named differently. They are all named "struct sockaddr_storage", no? You won't even need that most of the time, as it's all dealt with by getaddrinfo and friends (see sibling comment). Sure, there are some cases where it breaks down but the vast majority of software doesn't go there.
johnsmith21006 10y ago
A very long time ago and coming from VMS a big reason preferred BSD over Sys V was non blocking sockets. Now that was over 20 years ago but now going away?
HeadlessChild 10y ago
OT: How is text files like this (or like any RFC) written? Is there any standard format and is it done in a specific way? EDIT: Found it! There was actually a RFC for that. https://tools.ietf.org/html/rfc7322 https://tools.ietf.org/html/rfc7322
- hueving 10y ago
  So RFCs are self-hosting? ;)
- deleted 10y ago
  [deleted]
- kevin_thibedeau 10y ago
  Nroff traditionally but now there are any number of foo2rfc tools.
dbmikus 10y ago
Specifying a deadline timestamp instead of a countdown could run into problems if a given machine has an improperly configured clock. I think the risk of any machine having an incorrect timestamp is greater than a library implementer not recalculating the countdown correctly. Whenever possible, I prefer to stay away from timestamps. They're a fickle thing.
- slrz 10y ago
  Why would it matter? The deadline would be specified in terms of a monotonic clock that's unaffected by wall time adjustments. So even if some crude NTP implementation steps your clock by seconds (or a leap second is inserted), it shouldn't matter for the deadline calculation.
  - Bino 10y ago
    I think most of my timeout/deadlines relative to the monotonic time. Hence specifying the absolute monotonic time would just involve yet another syscall (to query the time). Is there a really good argument to specify a absolute monotonic time?
  - dbmikus 10y ago
    Then you need to send information to agree on a monotonic clock time.
bsder 10y ago
The problem with proposals like this is that they paper over the fundamental problem: Time is an input variable and really should be part of the API. The problem is that sometimes you want the end user to have control over time and sometimes you might not want the end user to have control over time. If I'm on Linux, I certainly should not be able to manipulate the "time" of the communication stack in order to send faster/more aggressively than I should be allowed. If I'm on embedded, I really want to be able to manipulate the "time" of the communication stack depending upon what I am doing.
- jstimpfle 10y ago
  Could you clarify? I don't get where you are heading with that. At least the concept of a "stream" isn't really linked with time. It's just a sequence of bytes.
  - bsder 10y ago
    The issue is that abstractions "leak". Let's take your "stream", for instance. It's a sequence of bytes. But is it a "maximum bandwidth" stream of bytes (gigantic file transfer) or is it a "minimum latency" (audio packet) stream of bytes? If I know or can control what the "time" the stack is working toward, the stack doesn't have to know the difference between those two. In addition, when on embedded, I quite often want a stack API which is "foo_init(), foo_deinit(), foo_queue_send(), foo_queue_action(), etc." which all update internal state And then "foo_make_incremental_progress(foo_state, foo_now, ...) where ONLY that incremental progress function actually carries out actions like reading hardware, writing hardware, timing out, etc. Now, I can use whatever concurrency construct I like without the stack getting in the way. Now, that isn't necessarily the fastest performance as the stack needs to be structured such that it only carries out one "action" at a time on each incremental call. However, it's very flexible. It also has the wonderful property of being repeatable and testable. Something which TCP stacks are notoriously resistant to.
Ono-Sendai 10y ago
Sounds good. Please add a flushHint() call as well: http://www.forwardscattering.org/post/3 http://www.forwardscattering.org/post/3
- astrange 10y ago
  Already exists in popular implementations: https://t37.net/nginx-optimization-understanding-sendfile-tcp_nodelay-and-tcp_nopush.html https://t37.net/nginx-optimization-understanding-sendfile-tc...
- rumcajz 10y ago
  The interesting question here is: Once we have flush() function does that means that protocols are allowed to delay data indefinitely if flush() is not called?
  - Ono-Sendai 10y ago
    Yeah that is an interesting question. I think if the provider of the socket interface could be sure that the client code using the interface would know about flush(), then it wouldn't need to set timers or anything like that to flush after X ms etc. However there wouldn't be any point, I think, to buffering up more than e.g. a MTU worth of bytes. So it wouldn't be delaying indefinitely.
eschaton 10y ago
I'd like to see some comparison with the STREAMS layered/stackable API from SVR4 which let a developer do just the sort of layering this API proposes. What makes it better than STREAMS, does it share the same pitfalls, etc.?
- yuhong 10y ago
  Also Plan 9.
notaplumber 10y ago
To clarify, none of the BSD projects have anything to do with this "revamp" not-a-real-RFC.
- rumcajz 10y ago
  "BSD sockets" is the name of the existing socket API.
- dchest 10y ago
  It's not some "not-a-real-RFC", it's https://en.wikipedia.org/wiki/Internet_Draft https://en.wikipedia.org/wiki/Internet_Draft
astrange 10y ago
There doesn't seem to be much of anything interesting here. I do think there's a lot to consider in a new system design, but the fact (brought up at the start) that you can't send any new protocol over the internet, and can't send anything but HTTP through half of it, really limits you. If I were designing an API, the RX side would be asynchronous and batch multiple connections, and the TX side would let you assemble your own TCP packets. This rules out ideas like sendfile() which IMO became nonstarters when we moved to HTTPS. And obviously the accidental complexity like fcntl vs. setsockopt, bind/connect, shutdown/close, multithreading+signals, have to go somehow. It doesn't really matter though. I can't imagine any major world crises are being caused by BSD sockets.
dcow 10y ago
There's a typo in section 3.4 paragraph 4 s/bystream/bytestream--unless it's an attempt to be clever.
- rumcajz 10y ago
  Fixed.
jheriko 10y ago
> To make the API less tedious to use, short protocol name, e.g. "ws", SHOULD be preferred to the long name, e.g. "websockets". Well that's a classic dumb mistake. Programming isn't about typing. Why not save everyone the effort of having to learn pointless things or look things up? There is no ambiguity with "websockets" where as "ws" describes nothing without context.
- rumcajz 10y ago
  Another example: use "tcp_connect" instead of "transmission_control_protocol_connect".
  - jheriko 10y ago
    i'd make an exception for that, but it has nothing to do with the length of the function name. TCP is much more well known as an abbreviation than long hand (most people who use it do not actually know what it stands for) communication is about getting the intent across and not obscuring it after all...
    - rumcajz 10y ago
      Yet one example: "dccp_send" or "datagram_congestion_control_protocol_send" ?

119 points by rumcajz 10y ago

comex 10y ago

Very interesting idea overall. I've long thought the current segregation of network protocols between userland and the kernel is rather arbitrary and inflexible - why should reliability (TCP) be in the kernel but security (TLS) be in userland? I don't think the idea of requiring Go-style lightweight threading is viable. Lightweight threading has many nice properties, but also inherently requires more expensive operations to deal with split stacks, and tends to use more memory than a manual approach. It's also rather poorly supported in general by existing languages and OSes (other than Go), while any new OS-level socket API should be as universally accessible as the existing one. In particular, many scripting-ish languages either have no concept of threading at all or only support isolated threads whose communication is limited to relatively slow message passing. In theory the language implementations could write a C shim to expose a non-blocking interface around blocking underlying operations - this is what libuv does today for file I/O, for instance - but the result would be a lot of unnecessary overhead.

tptacek 10y ago
How would a design with TCP done entirely in userland work? A given socket address might in one instant belong to uid 10, and in the next, after uid=10's connection closes, to uid 20. (I'm not challenging you, just asking what you think the semantics should look like). Worth adding for everyone else, though I'm sure you know: high speed network implementations already pull a lot of this stuff into userland.
- zrm 10y ago
  Basically the same as SOCK_DGRAM works except that send() and recv() expect/provide the TCP header before the transport payload. If a user process calls bind() on some address/port then it gets all TCP packets for that local address/port. If it calls connect() then it gets only the ones that match both the local and remote address/port.
  - 0x0 10y ago
    With packet retransmissions and out-of-order packet delivery, wouldn't there be a risk that the client software at uid=10 sees a clean shutdown and disconnects, but uid=20 instantly binds with the same ports and steals a bunch of delayed packets?
    - zrm 10y ago
      That happens with UDP already and is another reason everything ought to be encrypted with session keys. But the normal TCP solution of the OS holding released ports in TIME_WAIT for a couple of minutes still works as ever.
- comex 10y ago
  There are a few different ways to answer that question. First, I believe that for most applications, it's not that TCP should be in userland; rather, TLS should be in the kernel, or otherwise system managed. See the wall of text I just replied to someone else with. Second, if you do want to customize the transport layer, which is of course fairly common today - multiplayer games often go for some custom "reliable UDP" protocol that drops ordering guarantees to improve latency, and then there's Chromium with QUIC, and uTorrent ‎µTP, and WebRTC... well, the reality of networking today is that that has to be done on top of UDP rather than IP directly. I doubt there's much point writing a new API for TCP itself that exposes all the bells and whistles, because the design of TCP is dated. Rather, there should be more standardized APIs for applications to more easily use replacements for TCP that run over UDP. Ideally I should be able to switch from TCP to QUIC just by changing a few lines of code. (Edit: By the way, if we could redo networking protocols from scratch, I think ports ought to exist at the IP layer rather than being replicated in TCP and UDP, and then UDP could be abolished entirely. But we can't.) Actually, that ties into high performance networking too. As you know, those alternate stacks require userland to talk directly or near-directly to hardware, meaning you just have to give up on different ports belonging to different applications. But beyond that, one thing holding those stacks back is that they require a lot of custom code and don't work with existing applications. With a standardized networking API that supported userspace plugins, it could be possible to add "DPDK+rumptcpip" as an option next to "kernel TCP" and "QUIC", and configure any random process to use it.
  - zrm 10y ago
    > I doubt there's much point writing a new API for TCP itself that exposes all the bells and whistles, because the design of TCP is dated. Rather, there should be more standardized APIs for applications to more easily use replacements for TCP that run over UDP. The advantage TCP has over UDP is that middleboxes know what TCP FIN is and so are willing to use much longer timeouts for TCP sessions. For example the default Linux connection tracking timeout for established TCP connections is five days but for UDP streams it's three minutes. So if you need a long-lived session to receive event-based messages your choices are to use UDP with a mapped port using NAT-PMP or PCP (ideal but not always available), use UDP with frequent keepalives (expensive), or use TCP. Being able to do TCP in userspace would be very useful for any VPN-like thing because you could get the long timeouts but still deliver packets immediately even if an earlier one was lost, and avoid the TCP-over-TCP performance degradation by deferring congestion control to the inner packets.
- justincormack 10y ago
  The thing is, if addresses are scarce, you do need something (typically a kernel) to allocate them. Port numbers on a single IP are scarce. IPv6 addresses, if each machine has a /64 though are not, and it becomes much easier, each (64 bit!) pid can have its own IP with neither being reused, so routing can be dumb. (And/or encrypt everything, keys are not a scarce resource either).
- rumcajz 10y ago
  Can be done for example by having a address allocation daemon running in user space. The socket library would create an IPC connection to the daemon and ask for an address/port. The daemon would reserve the address/port and pass it back. The daemon would have to health check the applications so that it can return the addresses back to the pool in case of failure.
aftbit 10y ago
The big reason security is in userspace is because it requires policy decisions. How do we determine which remote users to trust? WoT, cert chains, trust on first use? There is kernel-mode security (IPsec) but it turns out that userspace is better equipped to make those decisions.
- comex 10y ago
  Actually, policy decisions are a big part of why I think TLS should be "in the kernel". It doesn't have to literally be in the kernel - it could be a userspace daemon instead - but that's an implementation detail. What matters is that just like with TCP, there should be one high-level API, provided by the system, which all applications are expected to use. Scenario: You want to temporarily ignore an invalid certificate. (Yes, I know this is usually a bad idea, but sometimes it’s necessary.) Today: Every application needs its own flag, environment variable, configuration option, and/or UI element to disable validation. If you’re unlucky, there won’t be one. If you’re really unlucky, there won’t be any way to enable validation; even in 2017 there are still lots of poorly written applications like that. Centralized: There should be a standard environment variable that works with everything, and/or maybe a “security center” applet where you can add a temporary system-wide exemption per-host, or something like that. Scenario: You want to sniff a secure connection for debugging purposes. Today: You have to use a MITM proxy, which is gross and needs setup and breaks things, and requires trusting invalid certificates, which doesn’t always work; see above and below. Centralized: The kernel could just hand decrypted packets to packet sniffing programs, allowing for purely passive observation. Scenario: You want to ensure that you’re using high-quality ciphersuites. Today: Up to the application, usually not customizable, so browsers will do the right thing and other applications will have some list the developer copy+pasted that seemed to work. Welp. Centralized: The system should be able to specify policy for all applications, with the option for the user to override it. After all, it’s almost never actually desirable for a cipher suite to be supported by one application and not another - like, it’s okay if my mail client connects to Google insecurely, but not if my instant messaging client does. Scenario: You want to add a custom trusted CA, or generally you want different applications to be consistent in their policy decisions. Today: Different TLS libraries have their own trust store formats and trust rules. If you’re lucky, your TLS library of choice will have some way to use the system trust store, on Windows and macOS where such a thing exists, although who knows if it’ll properly replicate the various rules browsers like to impose, e.g. requiring certificate transparency. If you’re unlucky, you end up with things like Java having its own trust store, or wget on macOS refusing to download any https URLs until you go install some .crt file and point wget to the right path. In both cases, you end up with a separate list of trusted certificates, that requires a separate process if you want to add a custom one, or upgrade to remove newly distrusted roots, etc. Centralized: Well, there’s already some centralization with system trust stores, but ideally not would those be supported universally, the entire process of “should this TLS connection be established” could be defined and maintained in one place. Scenario: On a server, you want to run a program that accepts TLS connections. Today: Each program needs its own copy of the private key and its own TLS configuration, which often doesn’t expose the full range of settings it’s desirable to customize. Centralized: The system holds the keys (reducing both configuration burden and the danger of leaks); applications can just say “listen on port X using TLS”, and send/receive plaintext on the resulting sockets, and encryption will be handled automatically. ... I admit there are also some downsides of a centralized design. One of them is updates: these days, on most systems, users update their browsers a lot more than they update the OS, so the browser is likely to be up to date while the OS is not. But "at least the browser is secure, even if other apps are not" is not exactly an ideal outcome, and this too seems like an artifact of history... there's no reason that OS components can't auto-update like browsers do.
jstimpfle 10y ago
I've come to think that the whole distinction "kernel vs userland" is inflexible. There was a thread here on HN a few days ago, and it showed some clear deficits of conventional OS, for example in processes. Speaking of lack of composability of clients and linked-in libraries, speaking of IPC and uncontrolled implicit environment inheritance (Docker... :<), speaking of the ad-hoc security model (Unix permissions, setuid hacks, and more recently extended attributes). There's just a single trust boundary in Unix. What if the system was instead just a graph of interacting entities, each managing control over some resource? The "userland process" would just be another entity that is in control of another (more ephemeral) resource. We wouldn't even separate address spaces! Security by careful propagation through fine-grained trust boundaries (instead of globally accessible resource managers that have a hard time deciding whether a requesting entity should be authorized). Think svn vs git. And my understanding is that that's much of what Lisp Machines were about...
- morb 10y ago
  Could you please link to the mentioned thread?
  - jstimpfle 10y ago
    https://news.ycombinator.com/item?id=13621623 https://news.ycombinator.com/item?id=13621623 If you search for my name there, you will find at least discussions about file modes and also why I claim that processes are a failed abstraction (it was really another user who pointed that out).
- nwmcsween 10y ago
  Define 'separate address spaces', you have linear address space (virtual) and kernel/user 'address spaces' which are address spaces but not in the way linear address spaces are. It would be easier to simply move almost everything not needed by the kernel into 'userspace' and have some sort of capability system to enforce access rights.
  - jstimpfle 10y ago

cafxx 10y ago

Am I missing something? It seems to propose to ditch all non-blocking APIs under the assumption that "runtimes" will provide lightweight concurrency. But AFAIK most (all?) runtimes that provide lightweight concurrency make extensive use of the non-blocking network APIs to work - including Golang, that is even cited as an example: During the decades since BSD sockets were first introduced the way they are used have changed significantly. While in the beginning the user was supposed to fork a new process for each connection and do all the work using simple blocking calls nowadays they are expected to keep a pool of connections, check them via functions like poll() or kqueue() and dispatch any work to be done to one of the worker threads in a thread pool. In other words, user is supposed to do both network and CPU scheduling. [...] To address this problem, this memo assumes that there already exists an efficient concurrency implementation where forking a new lightweight process takes at most hundreds of nanoseconds and context switch takes tens of nanoseconds. Note that there are already such concurrency systems deployed in the wild. One well-known example are Golang's goroutines but there are others available as well.

tptacek 10y ago
Apart from the layering, which really has nothing to do with the sockets interface and can be (and has been, repeatedly, as this author knows) built today on top of the sockets interface, I don't really see what's interesting at all about this interface; it seems like a simple restatement of the existing blocking socket API.
- rumcajz 10y ago
  It tries to describe in detail what are the minimal semantic preconditions for protocol implementations to be composable (both vertically and horizontally) while at the same time being efficient (so that a stack with 20 microprotocols won't grind to a halt). The most interesting parts are handshakes, how they are conflated among multiple protocols as well as semantics of error handling. But the real proposition here is to have an ecosystem of microprotocols (e.g. a CRLF protocol that splits bytestream into messages using CR+LF as a delimiter) which, by the virtue of having common, well understood semantics, could by composed into ad-hoc networking stacks according to users' needs.

Animats 10y ago

Why is this being proposed to the IETF? It's not an issue visible at the wire protocol level. It doesn't affect interoperability. Sockets belong to the POSIX spec, and to some extent to language specs.

luchs 10y ago
There are already other RFCs regarding the socket API, such as RFC3493 (IPv6 socket API).
- dfox 10y ago
  There even are RFCs (RFC 2783 comes to mind) that specify OS-level APIs that are only tangentialy related to networking.

shabbyrobe 10y ago

Didn't notice this on my first read, but this is by Martin Sustrik. Sustrik is a former ZeroMQ contributor and, more recently, author of the abandoned zmq alternative nanomsg [1], as well as the libmill [2] and libdill [3] lightweight concurrency libraries. [1] http://nanomsg.org/ [2] http://libmill.org/ [3] http://libdill.org/

jey 10y ago
Is nanomsg truly abandoned? I was hoping it was simply converging toward stability.
- shabbyrobe 10y ago
  Hmm true, my language was a bit ambiguous. I meant it was abandoned by Sustrik [1], even if some of the community are doing the best they can manage to keep it alive. [1] http://www.freelists.org/post/nanomsg/New-fork-was-Re-Moving-forward
  - jey 10y ago
    That seems a bit outdated; looks like main nanomsg repo has regular commits from gdamore now[1], so maybe the fork was resolved? 1. https://github.com/nanomsg/nanomsg/commits/master https://github.com/nanomsg/nanomsg/commits/master
    - shabbyrobe 10y ago
      Yep it was eventually resolved midway through 2016, D'Amore stepped up, then stepped down, then stepped up again. The link I cited is not a commentary on the current status of nanomsg, it is more to illustrate Sustrik's role in the situation. If, ultimately, this discussion is about Sustrik's IETF proposal, it is important to consider his relationship with previous similar (zmq/nanomsg) or related (libmill/libdill) projects which have borne his name.
    - RickHull 10y ago
      Also https://github.com/go-mangos/mangos https://github.com/go-mangos/mangos

luchs 10y ago

Something this API gets right is having a unified interface for both IPv4 and IPv6. With the sockets API, you have to decide for one of them. Changing isn't easy as the constants and structures are all named differently. While it's possible to use IPv6 sockets for IPv4 connections, this doesn't cover all use-cases. For example, you can't do IPv4 broadcast with an IPv6 socket. Additionally, as most examples are written for the classic IPv4 API, that's what everyone uses per default. Later on, when people complain about missing IPv6 support, they are turned down because it's a ton of work to change.

dfox 10y ago
For majority of applications, supporting IPv6 boils down to using getaddrinfo()/getnameinfo() instead of gethostbyname()/gethostbyaddr(), which results in code that supports both IPv4 and IPv6 and is simpler than the IPv4-only original.
- rwmj 10y ago
  The catch is you also have to be prepared to listen on multiple sockets. There are lots of servers out there which get this wrong, including qemu-nbd. For reference here's my implementation of this which (I think) gets it right: https://github.com/libguestfs/nbdkit/blob/master/src/sockets.c#L105 https://github.com/libguestfs/nbdkit/blob/master/src/sockets...
- mvkg 10y ago
  A good dual-stack implementation should also support Happy Eyeballs which does not result in simpler code.
deleted 10y ago
[deleted]
slrz 10y ago
Changing isn't easy as the constants and structures are all named differently. They are all named "struct sockaddr_storage", no? You won't even need that most of the time, as it's all dealt with by getaddrinfo and friends (see sibling comment). Sure, there are some cases where it breaks down but the vast majority of software doesn't go there.

johnsmith21006 10y ago

A very long time ago and coming from VMS a big reason preferred BSD over Sys V was non blocking sockets. Now that was over 20 years ago but now going away?

HeadlessChild 10y ago

OT: How is text files like this (or like any RFC) written? Is there any standard format and is it done in a specific way? EDIT: Found it! There was actually a RFC for that. https://tools.ietf.org/html/rfc7322 https://tools.ietf.org/html/rfc7322

hueving 10y ago
So RFCs are self-hosting? ;)
deleted 10y ago
[deleted]
kevin_thibedeau 10y ago
Nroff traditionally but now there are any number of foo2rfc tools.

dbmikus 10y ago

Specifying a deadline timestamp instead of a countdown could run into problems if a given machine has an improperly configured clock. I think the risk of any machine having an incorrect timestamp is greater than a library implementer not recalculating the countdown correctly. Whenever possible, I prefer to stay away from timestamps. They're a fickle thing.

slrz 10y ago
Why would it matter? The deadline would be specified in terms of a monotonic clock that's unaffected by wall time adjustments. So even if some crude NTP implementation steps your clock by seconds (or a leap second is inserted), it shouldn't matter for the deadline calculation.
- Bino 10y ago
  I think most of my timeout/deadlines relative to the monotonic time. Hence specifying the absolute monotonic time would just involve yet another syscall (to query the time). Is there a really good argument to specify a absolute monotonic time?
- dbmikus 10y ago
  Then you need to send information to agree on a monotonic clock time.

bsder 10y ago

The problem with proposals like this is that they paper over the fundamental problem: Time is an input variable and really should be part of the API. The problem is that sometimes you want the end user to have control over time and sometimes you might not want the end user to have control over time. If I'm on Linux, I certainly should not be able to manipulate the "time" of the communication stack in order to send faster/more aggressively than I should be allowed. If I'm on embedded, I really want to be able to manipulate the "time" of the communication stack depending upon what I am doing.

jstimpfle 10y ago
Could you clarify? I don't get where you are heading with that. At least the concept of a "stream" isn't really linked with time. It's just a sequence of bytes.
- bsder 10y ago
  The issue is that abstractions "leak". Let's take your "stream", for instance. It's a sequence of bytes. But is it a "maximum bandwidth" stream of bytes (gigantic file transfer) or is it a "minimum latency" (audio packet) stream of bytes? If I know or can control what the "time" the stack is working toward, the stack doesn't have to know the difference between those two. In addition, when on embedded, I quite often want a stack API which is "foo_init(), foo_deinit(), foo_queue_send(), foo_queue_action(), etc." which all update internal state And then "foo_make_incremental_progress(foo_state, foo_now, ...) where ONLY that incremental progress function actually carries out actions like reading hardware, writing hardware, timing out, etc. Now, I can use whatever concurrency construct I like without the stack getting in the way. Now, that isn't necessarily the fastest performance as the stack needs to be structured such that it only carries out one "action" at a time on each incremental call. However, it's very flexible. It also has the wonderful property of being repeatable and testable. Something which TCP stacks are notoriously resistant to.

Ono-Sendai 10y ago

Sounds good. Please add a flushHint() call as well: http://www.forwardscattering.org/post/3 http://www.forwardscattering.org/post/3

astrange 10y ago
Already exists in popular implementations: https://t37.net/nginx-optimization-understanding-sendfile-tcp_nodelay-and-tcp_nopush.html https://t37.net/nginx-optimization-understanding-sendfile-tc...
rumcajz 10y ago
The interesting question here is: Once we have flush() function does that means that protocols are allowed to delay data indefinitely if flush() is not called?
- Ono-Sendai 10y ago
  Yeah that is an interesting question. I think if the provider of the socket interface could be sure that the client code using the interface would know about flush(), then it wouldn't need to set timers or anything like that to flush after X ms etc. However there wouldn't be any point, I think, to buffering up more than e.g. a MTU worth of bytes. So it wouldn't be delaying indefinitely.

eschaton 10y ago

I'd like to see some comparison with the STREAMS layered/stackable API from SVR4 which let a developer do just the sort of layering this API proposes. What makes it better than STREAMS, does it share the same pitfalls, etc.?

yuhong 10y ago
Also Plan 9.

notaplumber 10y ago

To clarify, none of the BSD projects have anything to do with this "revamp" not-a-real-RFC.

rumcajz 10y ago
"BSD sockets" is the name of the existing socket API.
dchest 10y ago
It's not some "not-a-real-RFC", it's https://en.wikipedia.org/wiki/Internet_Draft https://en.wikipedia.org/wiki/Internet_Draft

astrange 10y ago

There doesn't seem to be much of anything interesting here. I do think there's a lot to consider in a new system design, but the fact (brought up at the start) that you can't send any new protocol over the internet, and can't send anything but HTTP through half of it, really limits you. If I were designing an API, the RX side would be asynchronous and batch multiple connections, and the TX side would let you assemble your own TCP packets. This rules out ideas like sendfile() which IMO became nonstarters when we moved to HTTPS. And obviously the accidental complexity like fcntl vs. setsockopt, bind/connect, shutdown/close, multithreading+signals, have to go somehow. It doesn't really matter though. I can't imagine any major world crises are being caused by BSD sockets.

dcow 10y ago

There's a typo in section 3.4 paragraph 4 s/bystream/bytestream--unless it's an attempt to be clever.

rumcajz 10y ago
Fixed.

jheriko 10y ago

> To make the API less tedious to use, short protocol name, e.g. "ws", SHOULD be preferred to the long name, e.g. "websockets". Well that's a classic dumb mistake. Programming isn't about typing. Why not save everyone the effort of having to learn pointless things or look things up? There is no ambiguity with "websockets" where as "ws" describes nothing without context.

rumcajz 10y ago
Another example: use "tcp_connect" instead of "transmission_control_protocol_connect".
- jheriko 10y ago
  i'd make an exception for that, but it has nothing to do with the length of the function name. TCP is much more well known as an abbreviation than long hand (most people who use it do not actually know what it stands for) communication is about getting the intent across and not obscuring it after all...
  - rumcajz 10y ago
    Yet one example: "dccp_send" or "datagram_congestion_control_protocol_send" ?