4 ms·

You can also sidestep that "quirky OS limitation" by just setting the first unprivileged port to 0(ip_unprivileged_port_start), no need for an new stack. https

You can also sidestep that "quirky OS limitation" by just setting the first unprivileged port to 0(ip_unprivileged_port_start), no need for an new stack. https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt https://www.kernel.org/doc/Documentation/networking/ip-sysct...

tptacek 2y ago
You can do that. Random programs cannot. Our CLI, which also does user-mode WireGuard and TCP/IP, doesn't even want to run under sudo. You're seeing the point, now: you want to build interesting network features that work the same way everywhere without demanding that your users be system administrators. Hence: user-mode TCP/IP.
- Xelynega 2y ago
  I'm still not seeing the point sadly. If you're running on a system you don't administrate that has ports under 1024 set as privileged, there's no way(with or without your cli) to have a userspace program receive TCP or UDP packets coming into the kernel from external devices for these ports(unless I'm completely mistaken). What can you accomplish with "user-mode TCP/IP" that you can't from userspace with system calls?
  - tptacek 2y ago
    You are completely mistaken. You can do literally anything you want with TCP/IP provided you can talk UDP on any port, by running a user-mode TCP/IP stack over WireGuard on that port.
    - Xelynega 2y ago
      I don't think you understand, and based on your reply it doesn't sound like I'm mistaken. With this CLI I am able to listen for external packets to port 80 from userspace without any elevated permissions and intercept traffic that's going to an application that's bound to that port on the OS? Edit: I think I understand what you're trying to do, but if I do then traffic is going from the kernel UDP stack to the userland TCP stack, back to the UDP kernel stack. Not sure how that avoids sending the packet to the kernel. If it's to get around the port restrictions, why can you not just use unprivileged ports?
      - tptacek 2y ago
        I'm not sure I can make this any simpler for you or easier to get your head around. If it helps: the idea here is giving an invocation of your program its own IP address. It can then do whatever it likes with TCP/IP for that address; its own routing, arbitrary protocols, whatever. The Go standard library makes it extremely easy to integrate. To the OS, it's all just ordinary socket code.
        taeric 2y ago
        I think it is a valid question on why not use unprivileged ports, though? Or am I also missing something?
        tptacek 2y ago
        "Unprivileged ports" is just a case in point for everything you would need privileges to do, from binding arbitrary ports to adding arbitrary addresses and running local servers on them, etc, etc, etc. The point is: turning "complicated" network features that require privilege into simple unprivileged socket code.
        taeric 2y ago
        Right, per my other thread below, my understanding was the "privileged" ports were mainly ones that were allowed for off machine communication by standard policy/convention long time ago. As such, using higher number ports should be just as easy in the code as using lower ports, outside of the discovery that was implied by following the other conventions. But, introducing new network addresses seems to have already side stepped the discovery affordances? I'll offer the same caveat here, btw, I am not trying to torpedo the idea of trying this. I'm genuinely curious why you would need to do this. Not necessarily why you would want to.
        tptacek 2y ago
        We did it because we run WireGuard gateways to the public cloud we operate, and our CLI wants to talk to things on customer networks (like remote Docker server instances to build new versions of apps). Our options were: * Do user-mode WireGuard (and thus TCP/IP) and talk "natively" to the infrastructure deployed on our platform. * Write case-by-case application gateways for each of those pieces of infrastructure tunneled somehow through HTTP.
anyfoo 2y ago
Getting that change onto the system sounds like "at best a big delay and at worst a nonstarter".
nonameiguess 2y ago
When they're talking about classified defense networks, the actual restrictions they mean is least privilege and separation of duties. Devs are not admins. They don't get root privilege on their machines. They can't create virtual network interfaces and they also can't change kernel settings. But if you put a full TCP/IP stack in userspace, well, they can run that and do whatever they want with it. To answer the upstream question about why arbitary outbound connections are allowed, they're not. This is connecting to a cloud development environment, and I would have to assume this service can be self-hosted, because on a classified network, the "cloud" isn't the cloud as Hacker News readers know it. Amazon et all run private data centers on US military installations that only the military and the IC can access and they're airgapped from the Internet. If you're on a workstation that can access this environment, that's all it can access. The only place you can exfiltrate data to is other military-controlled servers.
- Xelynega 2y ago
  If you're talking about developer machines, isn't the best(and easiest) solution to just run a VM that you administer so you can create virtual networks? If you're talking about production machines, a userspace application wouldn't be able to sniff privileged ports without elevated permissions, so I fail to see how this application would let you get around that limitation.
  - tptacek 2y ago
    No, because VMs are expensive and require some base level of system administration to operate, booting them usually requires privilege, and if the only problem you're trying to solve is reliably running (e.g.) Postgres and Redis protocol between your CLI and a server somewhere, it's extreme overkill.
    - Xelynega 2y ago
      VMs are free and can be run by any semi-competent developer wanting to host a test server on their development machine. Postgres and Redis can use non-privileged ports, so I don't understand why this would matter.

tptacek 2y ago

You can do that. Random programs cannot. Our CLI, which also does user-mode WireGuard and TCP/IP, doesn't even want to run under sudo. You're seeing the point, now: you want to build interesting network features that work the same way everywhere without demanding that your users be system administrators. Hence: user-mode TCP/IP.

Xelynega 2y ago
I'm still not seeing the point sadly. If you're running on a system you don't administrate that has ports under 1024 set as privileged, there's no way(with or without your cli) to have a userspace program receive TCP or UDP packets coming into the kernel from external devices for these ports(unless I'm completely mistaken). What can you accomplish with "user-mode TCP/IP" that you can't from userspace with system calls?
- tptacek 2y ago
  You are completely mistaken. You can do literally anything you want with TCP/IP provided you can talk UDP on any port, by running a user-mode TCP/IP stack over WireGuard on that port.
  - Xelynega 2y ago
    I don't think you understand, and based on your reply it doesn't sound like I'm mistaken. With this CLI I am able to listen for external packets to port 80 from userspace without any elevated permissions and intercept traffic that's going to an application that's bound to that port on the OS? Edit: I think I understand what you're trying to do, but if I do then traffic is going from the kernel UDP stack to the userland TCP stack, back to the UDP kernel stack. Not sure how that avoids sending the packet to the kernel. If it's to get around the port restrictions, why can you not just use unprivileged ports?
    - tptacek 2y ago
      I'm not sure I can make this any simpler for you or easier to get your head around. If it helps: the idea here is giving an invocation of your program its own IP address. It can then do whatever it likes with TCP/IP for that address; its own routing, arbitrary protocols, whatever. The Go standard library makes it extremely easy to integrate. To the OS, it's all just ordinary socket code.
      - taeric 2y ago
        I think it is a valid question on why not use unprivileged ports, though? Or am I also missing something?
        tptacek 2y ago
        "Unprivileged ports" is just a case in point for everything you would need privileges to do, from binding arbitrary ports to adding arbitrary addresses and running local servers on them, etc, etc, etc. The point is: turning "complicated" network features that require privilege into simple unprivileged socket code.
        taeric 2y ago
        Right, per my other thread below, my understanding was the "privileged" ports were mainly ones that were allowed for off machine communication by standard policy/convention long time ago. As such, using higher number ports should be just as easy in the code as using lower ports, outside of the discovery that was implied by following the other conventions. But, introducing new network addresses seems to have already side stepped the discovery affordances? I'll offer the same caveat here, btw, I am not trying to torpedo the idea of trying this. I'm genuinely curious why you would need to do this. Not necessarily why you would want to.
        tptacek 2y ago
        We did it because we run WireGuard gateways to the public cloud we operate, and our CLI wants to talk to things on customer networks (like remote Docker server instances to build new versions of apps). Our options were: * Do user-mode WireGuard (and thus TCP/IP) and talk "natively" to the infrastructure deployed on our platform. * Write case-by-case application gateways for each of those pieces of infrastructure tunneled somehow through HTTP.

anyfoo 2y ago

Getting that change onto the system sounds like "at best a big delay and at worst a nonstarter".

nonameiguess 2y ago

When they're talking about classified defense networks, the actual restrictions they mean is least privilege and separation of duties. Devs are not admins. They don't get root privilege on their machines. They can't create virtual network interfaces and they also can't change kernel settings. But if you put a full TCP/IP stack in userspace, well, they can run that and do whatever they want with it. To answer the upstream question about why arbitary outbound connections are allowed, they're not. This is connecting to a cloud development environment, and I would have to assume this service can be self-hosted, because on a classified network, the "cloud" isn't the cloud as Hacker News readers know it. Amazon et all run private data centers on US military installations that only the military and the IC can access and they're airgapped from the Internet. If you're on a workstation that can access this environment, that's all it can access. The only place you can exfiltrate data to is other military-controlled servers.

Xelynega 2y ago
If you're talking about developer machines, isn't the best(and easiest) solution to just run a VM that you administer so you can create virtual networks? If you're talking about production machines, a userspace application wouldn't be able to sniff privileged ports without elevated permissions, so I fail to see how this application would let you get around that limitation.
- tptacek 2y ago
  No, because VMs are expensive and require some base level of system administration to operate, booting them usually requires privilege, and if the only problem you're trying to solve is reliably running (e.g.) Postgres and Redis protocol between your CLI and a server somewhere, it's extreme overkill.
  - Xelynega 2y ago
    VMs are free and can be run by any semi-competent developer wanting to host a test server on their development machine. Postgres and Redis can use non-privileged ports, so I don't understand why this would matter.