12 ms·

NAT Is the Enemy of Low Power Devices

180 points by hasheddan 2y ago

justahuman74 2y ago
Interestingly, IPv6 is not listed as the solution
- api 2y ago
  The weird fear around it is crazy. It’s mostly just bigger IPs and it makes so much complexity and ugly hacks like NAT go away.
  - userbinator 2y ago
    IPv6 is its own complexity.
    - api 2y ago
      Why?
      - Joel_Mckay 2y ago
        In general, the IPv6 stack is not supported on many smaller mcu. It is not that options like FreeRTOS can't support the stack, but rather the resource constraints pose a challenge. That being said, most modern SoCs are competitively priced... and will boot Linux just fine for under $5/part. =3
        elcritch 2y ago
        Some of the new IoT protocols build on ipv6 natively. The resource overhead is minimal for modern mcus. Dropping dhcp and arp can save a lot of resources too. Also I have mcus with more ram than my first pc.
        jeroenhd 2y ago
        Disabling IPv6 in ESP-IDF can save about 40KiB of flash and 2KiB of RAM. Not enough that I'd do it by default, but I've hit the limit of some of my hobbyist ESP32s to the point where I've disabled modules to cram my code in there. Disabling IPv4 saves 25KiB of flash and less than a KiB of RAM. If you're down to the last kilobytes, disabling IPv6 makes more sense than disabling IPv4. Both options are choices of last resort, though.
        Joel_Mckay 2y ago
        A dual core 240MHz cpu with hardware paged flash and spi ram expansion support is not "small" in my opinion. However, I am a fan of their price point for hobby projects. Best regards =3
      - LegionMammal978 2y ago
        Eliminating NAT makes virtual networking much hairier. E.g., my desktop is currently connected to a big enterprise network that keeps track of all devices and will only allocate one IP address per MAC address. That leaves no IPs for any VMs on my desktop to use, so they must go through NAT if they want to communicate with the outside world.
        pcpuser 2y ago
        That's definitely a limitation of your network. I don't see how ipv6 can shoulder any responsibility here.
        Dylan16807 2y ago
        Are you talking about eliminating NAT on IPv4? IPv6, set up the normal way, does not allocate IPs. Devices can and will have several at once and their VMs can easily get some too.
        wmf 2y ago
        The enterprise corrupted IPv6 and disabled SLAAC so devices have to use whatever DHCPv6 gives them.
        somanyphotons 2y ago
        How does that work for android devices? I heard they don't allow dhcp provided addresses?
        wmf 2y ago
        The Android problem only occurs in an IPv6-only enterprise network which is very rare. Otherwise Android can use IPv4.
        pjc50 2y ago
        Shouldn't the VMs have their own MACs?
      - deleted 2y ago
        [deleted]
      - positr0n 2y ago
        Great question that I don't think the other commentors really answered. Sadly I don't really know "why" either. Second system effect? Instead of making IPv6 just "IPv4 with 128bit instead of 32bit address space" the designers changed a bunch of things. NDP is new and different than ARP. ICMPv6 is extended and different. MLD instead of IGMP for multicast group membership. DHCP is extended and has a bunch of other alternatives, like SLAAC. IPsec is mandatory now. All the new IPv6 protocols seem more secure, more powerful, and more efficient. So I guess that is "why". But I bet in an alternative history where backwards compatibility and the upgrade process were prioritized IPv6 would be a lot more widespread.
  - withinboredom 2y ago
    and also so many other things. ARP goes away, dhcp goes away -- yet people reinvented dhcp anyway and did it wrong (IHMO). I'm of the opinion that IPV6 changed some small things just enough to get people to have to learn new stuff -- and also forgot that NAT is not a firewall, somewhere along the way.
- gnabgib 2y ago
  It was 3 months ago: https://news.ycombinator.com/item?id=41884515 https://news.ycombinator.com/item?id=41884515
- LegionMammal978 2y ago
  An IPv6 router with a stateful firewall blocking incoming connections could have just the same issues with timeouts, I'd imagine. Switching to IPv6 doesn't just mean that anyone can make a P2P connection to anyone else (even STUN needs a third-party server to coordinate the two peers). (D)TLS session resumption (I'm not sure if their "Connection IDs" are that or something similar) seems like the most foolproof solution to this scenario, assuming that the remote host can support it.
  - namibj 2y ago
    But it'd be trivial to tell it to free the device from it, unlike with NAT, where you pretty much have to expire sessions to not run out of memory.
    - LegionMammal978 2y ago
      Not if the end user isn't in control of the firewall. (And if they were, then they could just forward dedicated ports for the devices they need.) It might not be as bad as the CGNAT situation, but there are plenty of big WANs that can't be reconfigured at will.
    - chgs 2y ago
      I have firewalls in v4 and v6 networks which don’t do any natting (well other than some 6-4 between them). They track sessions for security purposes, and they time them out for both security and memory reasons.
  - MerManMaid 2y ago
    >An IPv6 router with a stateful firewall blocking incoming connections could have just the same issues with timeouts, I'd imagine. You'd be surprised... PCP (Port Control Protocol) implemented by large vendors such as Cisco and Apple are able to punch through a firewall for up to 24 hours in a single session. https://github.com/Self-Hosting-Group/wiki/wiki/Port-Mapping-Protocols-Overview https://github.com/Self-Hosting-Group/wiki/wiki/Port-Mapping...
- vv_ 2y ago
  Few cellular modems commonly used in IoT support IPv6, and not all mobile network operators provide an IPv6 address. Since cellular connectivity plays a major role in the industry, IPv6 cannot be used as a blanket solution to this problem.
  - altairprime 2y ago
    Either it would or it wouldn’t help in most cases, but the absence of consideration of it at all weakens the article’s arguments from a carrier perspective. IPv6 adoption was at 90% by US mobile carriers a couple years ago, and the US is not known for its telco infrastructure investment; so, while using IPv6 may not be a uniform cure for their issues, the article’s total focus on legacy IPv4 NAT issues is in stark contrast to its availability carrier-side in one of the weakest examples available. China regulates that IPv6 be supported and enabled by default on all hardware sold for use in-country since last year, and telcos have six months left until a first-stage IPv4 new-hardware prohibition goes into effect later this year, so the assumption that most cellular modems don’t support IPv6 seems unlikely as well given their regulatory climate. This deserves more research or at least an explanation of why such was not done for the initial release of the paper.
    - kiwijamo 2y ago
      There are still countries with zero IPv6 on mobile, e.g. New Zealand. All three mobile carriers refuse to deploy IPv6 on mobile despite two of them doing so on their fixed line networks. Do agree with your point tho IPv6 should be considered a possible solution.
arcmechanica 2y ago
NAT was an good solution that the IETF came up with, we wouldn't be able to have the internet at our scale without it
- nrabulinski 2y ago
  Unless IPv6 were to be actually adopted as it was introduced
  - gamedever 2y ago
    I don't know networking all that well. In my mind, I have 50 devices connected to my router behind NAT. My Mac, My Apple TV, my iPhone, My PC, My Linux Box, My partner's versions of all of those. My video games. Etc From outside there's 1 IP address. With IPv6, every device would get it's own address outside. Why do I want that? That sounds less private to me. Am I mis-understanding something? Lots of traffic on one IP address sounds more obfuscated than all separate.
    - jeroenhd 2y ago
      With IPv6, every device has multiple IP addresses. One or more addresses that are rotated* to prevent you from being tracked easily, and one that's derived from your device's MAC address so you can make your devices easily accessible from WAN by opening ports in your firewall if you want to. You could disable the rotating addresses, or disable MAC-based ones by using DHCP, but there's usually no point. As for why you would want something like that: a whole bunch of software and hardware breaks because of NAT. Consumer NAT has some monkey patching inside of it rewriting some protocols to make them work again (which also allowed random websites to open arbitrary ports to arbitrary addresses in some Linux routers a while back, because NAT overrules firewall settings to work) but there are still limitations. For instance, if you're having issues with your Nintendo Switch, Nintendo will tell you to forward every single port to your Switch (https://en-americas-support.nintendo.com/app/answers/detail/a_id/22272/~/how-to-set-up-a-routers-port-forwarding-for-a-nintendo-switch-console https://en-americas-support.nintendo.com/app/answers/detail/..., hope that IP address doesn't get reassigned to an unpatched device later). Multiple Xbox consoles behind the same NAT requires tricking them into super-restricted-NAT mode to work, or enabling UPnP which allows devices to open ports in your firewall without any authentication. NAT just kind of sucks. IPv6 wasn't ready for deployment when NAT gained popularity, but all of the reasonable problems have been solved over a decade ago. *=default rotation happens daily, but your OS may allow you to pick a shorter duration. I've found out the hard way that setting this to five minutes will fill up Linux' route table real fast after a few days.
hermanubis 2y ago
Why not just have the server send a token to the client that the client includes in the next request?
- iforgotpassword 2y ago
  Because by that point the tcp session is already broken.
  - withinboredom 2y ago
    UDP?
gabesullice 2y ago
> This doesn’t solve the issue of cloud to device traffic being dropped after NAT timeout (check back for another post on that topic), but for many low power use cases, being able to sleep for an extended period of time is less important than being able to immediately push data to devices. Is "less important" a typo?
- hasheddan 2y ago
  Yes, thank you for calling out. It should read "more important". This has been corrected.
wavesound 2y ago
On modern firewalls/routers, NAT is only one cooks in the network kitchen raining on this author's parade. Stateful Packet Inspection has timeouts too!
- lxgr 2y ago
  But for TCP, you don't even need to be stateful just to prevent inbound connections, which is a huge win over NATs. UDP still needs state tracking, unfortunately.
vv_ 2y ago
The problem(-s) described in the blog post are really acute for IoT in general, especially if you want your device to run on batteries or you have a limited data budget. > Therefore, when you try to continue talking to the server over a previously established session, it will not recognize you. This means you’ll have to re-establish the session, which typically involves expensive cryptographic operations and sending a handful of messages back and forth before actually delivering the data you were interested in sending originally. The blog post mentions Session IDs as a solution, but these require servers to be stateful, which can be challenging in most deployments. An alternative is Session Tickets (https://datatracker.ietf.org/doc/html/rfc5077 https://datatracker.ietf.org/doc/html/rfc5077), but these may cause issues when offloading networking to another device, such as a cellular modem, as their implementation may be non-standard or faulty. incoming rant These issues could be mitigated—or even solved—by using mature software platforms like Zephyr RTOS and their native networking stacks, which receive more frequent updates than traditional “all-in-one-package” SoCs. However, many corporations choose to save a few dollars on hardware at the expense of incurring thousands in software engineering costs and bug-hunting. It is often seen as more cost-effective to purchase a cellular modem with an internal MCU rather than a separate cellular modem and a host MCU to run the networking stack. It is one of the many reasons why many IoT devices are utter garbage.
- marsven_422 2y ago
  [flagged]
  - SirSavary 2y ago
    Could you elaborate for someone who is unfamiliar?
    - shipp02 2y ago
      Zephyr RTOS[1] is an RTOS supported by the Linux foundation. It has similar structures to Linux for device configuration like a device tree and tries to emulate POSIXish APIs. I think some embedded people are put off by this configuration structure. Notably, Zephyr RTOS is the basis of Nordic Semiconductor's SDK[2]. Nordic is a major manufacturer of cellular and wireless MCUs. [1]:https://www.zephyrproject.org/ https://www.zephyrproject.org/ [2]:https://docs.nordicsemi.com/bundle/ncs-latest/page/nrf/index.html https://docs.nordicsemi.com/bundle/ncs-latest/page/nrf/index... Edit: GP seems to have a history of posting 1 line hot takes with no elaboration
dent9876543 2y ago
That NAT is a problem presumes that we actually want our IoT devices reaching out to the out-of-intranet zone. NAT gets the blame, and the intranet as a concept is generally a big corp term. But I prefer my IoT devices not to need to reach out of my network. For me, NAT is an unwitting ally in the fight against such nonsense.
- kstrauser 2y ago
  What NAT are you using that doesn’t have a firewall? I haven’t personally used one of those since the ‘90s.
  - kazinator 2y ago
    The first NAT I used in the middle 90's was IP Masquerading in the Linux kernel, by Pauline Middelink. That had a firewall.
    - kstrauser 2y ago
      Same, but ISTR we had some Cisco gear where NAT (or PAT as they insisted; yes, I know difference; no, no one cared) had a different license or hardware requirement or something from firewall rules.
- procaryote 2y ago
  Especially if the data is unencrypted and only authenticated by source ip and a long lived token-like thing
- rcxdude 2y ago
  If you don't want that, then complain about a lacking a configuration as such and configure your firewall so that that they can't. But don't cheer on something that's breaking functionality that others might want (especially if it doesn't actually achieve your own goals reliably).
  - dent9876543 2y ago
kazinator 2y ago
What you can do is port forwarding. You have a bunch of devices behind a 1:N NAT, so they share one IP address. For specific services on those devices, you can pair dedicated ports with this IP address, binding them to internal IP:port pairs. It's not a perfect solution for every scenario, and requires configuration, but there it is. This is how people on residential lines run web servers, mail servers, ... they map TCP ports like 443, 80, 22 and 25 on their router to go to specific internal hosts. Doh!
- procaryote 2y ago
  True in the general sense, but don't do this for IoT devices. Always remember that the S in IoT stands for Security
- bb88 2y ago
  With CG-NAT this doesn't work. Multiple customers are sharing the same IP address, all of which are sitting behind a NAT. Further the internet gateway is a NAT sitting behind the CG-NAT. And if you prefer to use a nice Mesh WiFi router, well that's a third NAT layer. Common suggestions I've heard: "Use a VPN" I tried to buy a computer from Apple directly. They detected the VPN and wouldn't let me purchase it. I turned off the VPN and the purchase worked. I then got a call from Apple asking me if I really did intend to buy the computer. "Use Tailscale" On my setup, tailscale can't really navigate the NAT setup correctly. I have a 200Mb downlink and I will often only get a 10th of that through tailscale. Also the routing table no longer works as routing is handled by a myriad of netfilter rules -- which may or may not conflict with docker on the same box. "Use Ubiquiti (or other networking gear) to get rid of the last NAT" Gah. I didn't need to perform a huge cash outlay 10 years ago and rewiring the house to do the same thing I'm trying to do today.
  - kazinator 2y ago
    I don't have my own external IP address (even if dynamic) + I want my devices to be spontaneously contactable from the network without polling anything from the inside = does not compute
philjohn 2y ago
Isn't this the whole point of Thread? You have a low power mesh network for inside a deployment (house, office, factory, whatever) and it's the border router which does the "communicate with the internet/lan/wider network" piece. These are typically plugged in (so no worries about having to be low power) have plenty of memory (so no worries about having to drop established routes like a router).
procaryote 2y ago
It seems brave to let an IoT device talk to someone over an unencrypted wan though. They're often of pretty varying software quality and rarely updated. If you really want IoT wifi devices, put them on a separate wifi, and only let them talk to a local device that you can keep up to date. Assume they're vulnerable to local attacks over wifi and act accordingly, e.g. don't give the IoT wifi access to your other devices beyond to that controller, and definitely not to the wider internet. If they're closed source, assume they're already compromised from the factory
- vv_ 2y ago
  The issue in IoT is that most people expect to be able to control their IoT (e.g. Smart Home) devices from outside of their network. This requires you to have a central server these devices communicate with or a similar deployment on-premises with a public IP address. I've always wondered how it is economically feasible to run these central services without a monthly subscription. If you stop selling devices you'll go under fairly quickly.
  - leptons 2y ago
    I have all my "Tuya" IoT devices on a separate network, isolated from my personal intranet, and the cloud servers these devices talk to are in China. I pay nothing to "Tuya" for their cloud service, never have. If the "Tuya" service ever goes down, my home automation is also down, and that sucks. I'm trying to replace it all with "Tasmota" devices now, but it's not quite as easy or as cheap to do.
  - procaryote 2y ago
    Yeah, sadly, people care about convenience and not security. So you get things like this: https://www.malwarebytes.com/blog/news/2024/04/ring-agrees-to-pay-5-6-million-after-cameras-were-used-to-spy-on-customers https://www.malwarebytes.com/blog/news/2024/04/ring-agrees-t... and somehow they are still in business, and popular. If you do care about security, keeping your home-automation within your own control is probably the only sane path. Homeassistant and similar open source things like openhab are pretty good if a bit fiddly, and a wireguard vpn like tailscale a fairly practical way to access it when away from home.
johnklos 2y ago
This has poorly considered generalizations and reads like an ad.
- hasheddan 2y ago
  An ad for... the IETF? All of the firmware discussed in this post is open source, and we even contributed DTLS Connection ID server side support to a popular open source library [0] so other folks can stand up secure cloud services for low power devices. Sure, we sell a product, but our broader mission is making the increasing number of internet connected devices more secure and reliable. When sharing knowledge and technology is in service of that mission, we do not hesitate to do so. [0]: https://blog.golioth.io/golioth-announces-connection-id-support/#cloud-changes https://blog.golioth.io/golioth-announces-connection-id-supp...
apple1417 2y ago
I have worked on a device with this exact same "send a tiny sensor reading every 30 minutes" use case, and this has not been my experience at all. We can run an STM32 and a few sensors at single digit microamps, add an LCD display and a few other niceties and it's one or two dozen. Simply turning on a modem takes hundreds of microamps, if not milliamps. In my experience it's always been better for power consumption to completely shut down the modem and start from scratch each time [1] - which means you're paying to start a new session every time anyway. Now I'll agree it's still inefficient to start up a full TLS session, a protocol like in the post will have it's uses, but I wouldn't blame it on NAT. [1] Doing this of course kills any chance at server-to-device comms, you can only ever apply changes when the device next checks in. This does cause us complaints from time to time, especially for those with longer intervals.
- hasheddan 2y ago
  Author of this post here -- thanks for sharing your experience! One thing I'll agree with immediately is that if you can afford to power down hardware that is almost always going to be your best option (see a previous post on this topic [0]). I believe the NAT post also calls this out, though I believe I could have gone further to disambiguate "sleeping" and "turning off": > This doesn’t solve the issue of cloud to device traffic being dropped after NAT timeout (check back for another post on that topic), but for many low power use cases, being able to sleep for an extended period of time is more important than being able to immediately push data to devices. (edit: there was originally an unfortunate typo here where the paragraph read "less important" rather than "more important") Depending on the device and the server, powering down the modem does not necessarily mean that a session has to be started from scratch when it is powered on again. In fact, this is one of the benefits of the DTLS Connection ID strategy. A cellular device, for example, could wake up the next time in a completely different location, connect to a new base station, be assigned a fresh IP address, and continue communication with the server without having to perform a full handshake. In reality, there is a spectrum of low power options with modems. We have written about many of them, including a post [1] that followed this one and describes using extended discontinuous reception (eDRX) [2] with DTLS Connection IDs and analyzing power consumption. [0]: https://blog.golioth.io/power-optimization-recommendations/ https://blog.golioth.io/power-optimization-recommendations/ [1]: https://blog.golioth.io/turn-off-subsystems-remotely-to-reduce-power-usage-with-golioth-and-joulescope/ https://blog.golioth.io/turn-off-subsystems-remotely-to-redu... [2]: https://www.everythingrf.com/community/what-is-edrx https://www.everythingrf.com/community/what-is-edrx
lxgr 2y ago
NAT breaks the central idea of IP (i.e. packet switching with stateless intermediary nodes). All of its problems are essentially downstream from there.
londons_explore 2y ago
> This doesn’t solve the issue of cloud to device traffic being dropped after NAT timeout (check back for another post on that topic), This is the key problem. I think it would be best solved by NAT devices having some way of probing their timeout policy, and then notifying the endpoints when they expire a mapping. One way to do that might be for a client to deliberately send a packet to the server with an insufficient TTL. The reply (via ICMP) could then contain fields specifying the timeout in minutes of the mapping, together with some generation number specifying if the mapping table has been cleared due to reboot or overflowed since last queried. There might even be a way to request a specific mapping become long lived - perhaps for months or years. The benefit of all this is that client devices can do push notifications from servers or P2P notifications, all with no polling - allowing for example coin cell devices to last for months or years (with appropriate WiFi protocol improvements too)
mannyv 2y ago
People have been complaining about NAT for decades. It's time to STFU. It's a well-known problem with a bunch of well-known solutions. If you don't like NAT you could go IPv6. But really, why do you need to talk to your device? If it's just reporting in NAT is irrelevant. If you want to do device management just write something into your protocol to check for updates/commands and deal with it on a periodic basis. You can even do that on startup, so you can tell the customer to power cycle the device. It's unlikely that any IoT device needs instant updates, so long periodic updates are probably fine.
- dave78 2y ago
  There are plenty of IoT devices that people want to execute commands on (anything remotely controlled, basically). Polling for commands on a periodic basis introduces lag into that process which is irritating. Furthermore, polling at a frequent interval can end up using a lot of power as well versus waiting in a receive-only mode for an incoming command.
  - raggi 2y ago
    The alternative to polling is unfortunately polling, which is what the article is about. You can avoid polling for messages, but you have to send packets outbound regularly in order to maintain a NAT mapping & connection, so that the external side can send messages inward. The latency is overcome this way, so latency is a solvable problem, but this need to constantly wake up a radio every <30s in order to keep a NAT session alive is a significant power draw. In theory you might be able to avoid this with NAT-PMP / UPnP however their deployments are inconsistent and their server side implementations are extremely buggy.
raggi 2y ago
What I want from platforms, and I fought for at one time in Google with no success: a platform API that provides applications a way to schedule packets when the radio turns on. The mobile platforms in particular continue to assume that you can live with their high level HTTP stuff, and it's just not good enough. The non-mobile platforms largely don't even approach the problem at all.
- freedomben 2y ago
  Indeed, that sounds like an obvious feature. Hard to believe it hasn't been implemented! I'd love to have that feature on Linux desktop/laptops. I think you could make lots of applications behave a whole lot better.
  - raggi 2y ago
    the arguments from folks on the "architecture review boards" was that multiple connections are always bad and that developers can't be trusted. i'm willing to accept that they did get beat up quite a bit over power at various points when at times applications were a big part of the problem. That said this is also a gross misunderstanding of the problem and overall solution space, as well as very much gatekeeping.
    - blitzar 2y ago
      Dump the entire queue to a google server (confirm receipt and shut down the connection) and then have the google server forward all the data to its destinations? Seems to me that would be the lowest power, lowest developer trust, lowest number of connections, maximum gatekeeping method.
clearint 2y ago
This article should clarify at the start whether TCP or UDP is under consideration. NAT idle timeouts for both are typically very different. RFC 5382 [0] specifies no less than 2 hours and 4 minutes for TCP. RFC 4787 [1] specifies no less than 2 minutes for UDP. Towards the end of the article it becomes clear that it's UDP. The example diagrams also incorrectly show port numbers exceeding 65535. The port fields in TCP and UDP headers are 16 bits [2]. [0]: https://www.rfc-editor.org/rfc/rfc5382 https://www.rfc-editor.org/rfc/rfc5382 [1]: https://www.rfc-editor.org/rfc/rfc4787 https://www.rfc-editor.org/rfc/rfc4787 [2]: https://textbook.cs161.org/network/transport.html https://textbook.cs161.org/network/transport.html
- apitman 2y ago
  That's interesting. Do NATs in the wild tend to be spec-compliant with their timeouts?
VagabundoP 2y ago
Wasn't IPv6 suppose to solve all this? I don't understand why that stalled. Also considering the state of iot security its probably not a great idea to have everything accessible anyway. But that's a slightly different problem to solve.
- chgs 2y ago
  It’s not going to solve a stateful firewall timing out. You try to continue a tcp session that’s timed out on my firewall and the packets will be dropped. This applies a fair amount t to me when I suspend my laptop, my ssh session will drop as both the server and the firewalls drop the session while it sits there peacefully. When it comes back the tcp packets get sent into the void. Meanwhile my WireGuard connection which runs through two separate ipv4 nats works just fine, as it doesn’t rely on sessions or a server timing out a socket. Nat is irrelevant to the problem.
  - VagabundoP 2y ago
    Why not let through all ipv6 traffic? It seems improbable that your device is going to get caught up in a ipv6 scan; the address space is too vast - I'm assuming here. Just have relatively secure devices using ipv6 - not shipping with standard default admin passwords would nearly be enough if there aren't obvious vulnerabilities.
eternityforest 2y ago
Why are we still making IoT devices that talk to anything other than a local hub? They don't need to care about NAT.

180 points by hasheddan 2y ago

justahuman74 2y ago

Interestingly, IPv6 is not listed as the solution

api 2y ago
The weird fear around it is crazy. It’s mostly just bigger IPs and it makes so much complexity and ugly hacks like NAT go away.
- userbinator 2y ago
  IPv6 is its own complexity.
  - api 2y ago
    Why?
    - Joel_Mckay 2y ago
      In general, the IPv6 stack is not supported on many smaller mcu. It is not that options like FreeRTOS can't support the stack, but rather the resource constraints pose a challenge. That being said, most modern SoCs are competitively priced... and will boot Linux just fine for under $5/part. =3
      - elcritch 2y ago
        Some of the new IoT protocols build on ipv6 natively. The resource overhead is minimal for modern mcus. Dropping dhcp and arp can save a lot of resources too. Also I have mcus with more ram than my first pc.
        jeroenhd 2y ago
        Disabling IPv6 in ESP-IDF can save about 40KiB of flash and 2KiB of RAM. Not enough that I'd do it by default, but I've hit the limit of some of my hobbyist ESP32s to the point where I've disabled modules to cram my code in there. Disabling IPv4 saves 25KiB of flash and less than a KiB of RAM. If you're down to the last kilobytes, disabling IPv6 makes more sense than disabling IPv4. Both options are choices of last resort, though.
        Joel_Mckay 2y ago
        A dual core 240MHz cpu with hardware paged flash and spi ram expansion support is not "small" in my opinion. However, I am a fan of their price point for hobby projects. Best regards =3
    - LegionMammal978 2y ago
      Eliminating NAT makes virtual networking much hairier. E.g., my desktop is currently connected to a big enterprise network that keeps track of all devices and will only allocate one IP address per MAC address. That leaves no IPs for any VMs on my desktop to use, so they must go through NAT if they want to communicate with the outside world.
      - pcpuser 2y ago
        That's definitely a limitation of your network. I don't see how ipv6 can shoulder any responsibility here.
      - Dylan16807 2y ago
        Are you talking about eliminating NAT on IPv4? IPv6, set up the normal way, does not allocate IPs. Devices can and will have several at once and their VMs can easily get some too.
        wmf 2y ago
        The enterprise corrupted IPv6 and disabled SLAAC so devices have to use whatever DHCPv6 gives them.
        somanyphotons 2y ago
        How does that work for android devices? I heard they don't allow dhcp provided addresses?
        wmf 2y ago
        The Android problem only occurs in an IPv6-only enterprise network which is very rare. Otherwise Android can use IPv4.
      - pjc50 2y ago
        Shouldn't the VMs have their own MACs?
    - deleted 2y ago
      [deleted]
    - positr0n 2y ago
      Great question that I don't think the other commentors really answered. Sadly I don't really know "why" either. Second system effect? Instead of making IPv6 just "IPv4 with 128bit instead of 32bit address space" the designers changed a bunch of things. NDP is new and different than ARP. ICMPv6 is extended and different. MLD instead of IGMP for multicast group membership. DHCP is extended and has a bunch of other alternatives, like SLAAC. IPsec is mandatory now. All the new IPv6 protocols seem more secure, more powerful, and more efficient. So I guess that is "why". But I bet in an alternative history where backwards compatibility and the upgrade process were prioritized IPv6 would be a lot more widespread.
- withinboredom 2y ago
  and also so many other things. ARP goes away, dhcp goes away -- yet people reinvented dhcp anyway and did it wrong (IHMO). I'm of the opinion that IPV6 changed some small things just enough to get people to have to learn new stuff -- and also forgot that NAT is not a firewall, somewhere along the way.
gnabgib 2y ago
It was 3 months ago: https://news.ycombinator.com/item?id=41884515 https://news.ycombinator.com/item?id=41884515
LegionMammal978 2y ago
An IPv6 router with a stateful firewall blocking incoming connections could have just the same issues with timeouts, I'd imagine. Switching to IPv6 doesn't just mean that anyone can make a P2P connection to anyone else (even STUN needs a third-party server to coordinate the two peers). (D)TLS session resumption (I'm not sure if their "Connection IDs" are that or something similar) seems like the most foolproof solution to this scenario, assuming that the remote host can support it.
- namibj 2y ago
  But it'd be trivial to tell it to free the device from it, unlike with NAT, where you pretty much have to expire sessions to not run out of memory.
  - LegionMammal978 2y ago
    Not if the end user isn't in control of the firewall. (And if they were, then they could just forward dedicated ports for the devices they need.) It might not be as bad as the CGNAT situation, but there are plenty of big WANs that can't be reconfigured at will.
  - chgs 2y ago
    I have firewalls in v4 and v6 networks which don’t do any natting (well other than some 6-4 between them). They track sessions for security purposes, and they time them out for both security and memory reasons.
- MerManMaid 2y ago
  >An IPv6 router with a stateful firewall blocking incoming connections could have just the same issues with timeouts, I'd imagine. You'd be surprised... PCP (Port Control Protocol) implemented by large vendors such as Cisco and Apple are able to punch through a firewall for up to 24 hours in a single session. https://github.com/Self-Hosting-Group/wiki/wiki/Port-Mapping-Protocols-Overview https://github.com/Self-Hosting-Group/wiki/wiki/Port-Mapping...
vv_ 2y ago
Few cellular modems commonly used in IoT support IPv6, and not all mobile network operators provide an IPv6 address. Since cellular connectivity plays a major role in the industry, IPv6 cannot be used as a blanket solution to this problem.
- altairprime 2y ago
  Either it would or it wouldn’t help in most cases, but the absence of consideration of it at all weakens the article’s arguments from a carrier perspective. IPv6 adoption was at 90% by US mobile carriers a couple years ago, and the US is not known for its telco infrastructure investment; so, while using IPv6 may not be a uniform cure for their issues, the article’s total focus on legacy IPv4 NAT issues is in stark contrast to its availability carrier-side in one of the weakest examples available. China regulates that IPv6 be supported and enabled by default on all hardware sold for use in-country since last year, and telcos have six months left until a first-stage IPv4 new-hardware prohibition goes into effect later this year, so the assumption that most cellular modems don’t support IPv6 seems unlikely as well given their regulatory climate. This deserves more research or at least an explanation of why such was not done for the initial release of the paper.
  - kiwijamo 2y ago
    There are still countries with zero IPv6 on mobile, e.g. New Zealand. All three mobile carriers refuse to deploy IPv6 on mobile despite two of them doing so on their fixed line networks. Do agree with your point tho IPv6 should be considered a possible solution.

arcmechanica 2y ago

NAT was an good solution that the IETF came up with, we wouldn't be able to have the internet at our scale without it

nrabulinski 2y ago
Unless IPv6 were to be actually adopted as it was introduced
- gamedever 2y ago
  I don't know networking all that well. In my mind, I have 50 devices connected to my router behind NAT. My Mac, My Apple TV, my iPhone, My PC, My Linux Box, My partner's versions of all of those. My video games. Etc From outside there's 1 IP address. With IPv6, every device would get it's own address outside. Why do I want that? That sounds less private to me. Am I mis-understanding something? Lots of traffic on one IP address sounds more obfuscated than all separate.
  - jeroenhd 2y ago
    With IPv6, every device has multiple IP addresses. One or more addresses that are rotated* to prevent you from being tracked easily, and one that's derived from your device's MAC address so you can make your devices easily accessible from WAN by opening ports in your firewall if you want to. You could disable the rotating addresses, or disable MAC-based ones by using DHCP, but there's usually no point. As for why you would want something like that: a whole bunch of software and hardware breaks because of NAT. Consumer NAT has some monkey patching inside of it rewriting some protocols to make them work again (which also allowed random websites to open arbitrary ports to arbitrary addresses in some Linux routers a while back, because NAT overrules firewall settings to work) but there are still limitations. For instance, if you're having issues with your Nintendo Switch, Nintendo will tell you to forward every single port to your Switch (https://en-americas-support.nintendo.com/app/answers/detail/a_id/22272/~/how-to-set-up-a-routers-port-forwarding-for-a-nintendo-switch-console https://en-americas-support.nintendo.com/app/answers/detail/..., hope that IP address doesn't get reassigned to an unpatched device later). Multiple Xbox consoles behind the same NAT requires tricking them into super-restricted-NAT mode to work, or enabling UPnP which allows devices to open ports in your firewall without any authentication. NAT just kind of sucks. IPv6 wasn't ready for deployment when NAT gained popularity, but all of the reasonable problems have been solved over a decade ago. *=default rotation happens daily, but your OS may allow you to pick a shorter duration. I've found out the hard way that setting this to five minutes will fill up Linux' route table real fast after a few days.

hermanubis 2y ago

Why not just have the server send a token to the client that the client includes in the next request?

iforgotpassword 2y ago
Because by that point the tcp session is already broken.
- withinboredom 2y ago
  UDP?

gabesullice 2y ago

> This doesn’t solve the issue of cloud to device traffic being dropped after NAT timeout (check back for another post on that topic), but for many low power use cases, being able to sleep for an extended period of time is less important than being able to immediately push data to devices. Is "less important" a typo?

hasheddan 2y ago
Yes, thank you for calling out. It should read "more important". This has been corrected.

wavesound 2y ago

On modern firewalls/routers, NAT is only one cooks in the network kitchen raining on this author's parade. Stateful Packet Inspection has timeouts too!

lxgr 2y ago
But for TCP, you don't even need to be stateful just to prevent inbound connections, which is a huge win over NATs. UDP still needs state tracking, unfortunately.

vv_ 2y ago

The problem(-s) described in the blog post are really acute for IoT in general, especially if you want your device to run on batteries or you have a limited data budget. > Therefore, when you try to continue talking to the server over a previously established session, it will not recognize you. This means you’ll have to re-establish the session, which typically involves expensive cryptographic operations and sending a handful of messages back and forth before actually delivering the data you were interested in sending originally. The blog post mentions Session IDs as a solution, but these require servers to be stateful, which can be challenging in most deployments. An alternative is Session Tickets (https://datatracker.ietf.org/doc/html/rfc5077 https://datatracker.ietf.org/doc/html/rfc5077), but these may cause issues when offloading networking to another device, such as a cellular modem, as their implementation may be non-standard or faulty. incoming rant These issues could be mitigated—or even solved—by using mature software platforms like Zephyr RTOS and their native networking stacks, which receive more frequent updates than traditional “all-in-one-package” SoCs. However, many corporations choose to save a few dollars on hardware at the expense of incurring thousands in software engineering costs and bug-hunting. It is often seen as more cost-effective to purchase a cellular modem with an internal MCU rather than a separate cellular modem and a host MCU to run the networking stack. It is one of the many reasons why many IoT devices are utter garbage.

marsven_422 2y ago
[flagged]
- SirSavary 2y ago
  Could you elaborate for someone who is unfamiliar?
  - shipp02 2y ago
    Zephyr RTOS[1] is an RTOS supported by the Linux foundation. It has similar structures to Linux for device configuration like a device tree and tries to emulate POSIXish APIs. I think some embedded people are put off by this configuration structure. Notably, Zephyr RTOS is the basis of Nordic Semiconductor's SDK[2]. Nordic is a major manufacturer of cellular and wireless MCUs. [1]:https://www.zephyrproject.org/ https://www.zephyrproject.org/ [2]:https://docs.nordicsemi.com/bundle/ncs-latest/page/nrf/index.html https://docs.nordicsemi.com/bundle/ncs-latest/page/nrf/index... Edit: GP seems to have a history of posting 1 line hot takes with no elaboration

dent9876543 2y ago

That NAT is a problem presumes that we actually want our IoT devices reaching out to the out-of-intranet zone. NAT gets the blame, and the intranet as a concept is generally a big corp term. But I prefer my IoT devices not to need to reach out of my network. For me, NAT is an unwitting ally in the fight against such nonsense.

kstrauser 2y ago
What NAT are you using that doesn’t have a firewall? I haven’t personally used one of those since the ‘90s.
- kazinator 2y ago
  The first NAT I used in the middle 90's was IP Masquerading in the Linux kernel, by Pauline Middelink. That had a firewall.
  - kstrauser 2y ago
    Same, but ISTR we had some Cisco gear where NAT (or PAT as they insisted; yes, I know difference; no, no one cared) had a different license or hardware requirement or something from firewall rules.
procaryote 2y ago
Especially if the data is unencrypted and only authenticated by source ip and a long lived token-like thing
rcxdude 2y ago
If you don't want that, then complain about a lacking a configuration as such and configure your firewall so that that they can't. But don't cheer on something that's breaking functionality that others might want (especially if it doesn't actually achieve your own goals reliably).
- dent9876543 2y ago

kazinator 2y ago

What you can do is port forwarding. You have a bunch of devices behind a 1:N NAT, so they share one IP address. For specific services on those devices, you can pair dedicated ports with this IP address, binding them to internal IP:port pairs. It's not a perfect solution for every scenario, and requires configuration, but there it is. This is how people on residential lines run web servers, mail servers, ... they map TCP ports like 443, 80, 22 and 25 on their router to go to specific internal hosts. Doh!

procaryote 2y ago
True in the general sense, but don't do this for IoT devices. Always remember that the S in IoT stands for Security
bb88 2y ago
With CG-NAT this doesn't work. Multiple customers are sharing the same IP address, all of which are sitting behind a NAT. Further the internet gateway is a NAT sitting behind the CG-NAT. And if you prefer to use a nice Mesh WiFi router, well that's a third NAT layer. Common suggestions I've heard: "Use a VPN" I tried to buy a computer from Apple directly. They detected the VPN and wouldn't let me purchase it. I turned off the VPN and the purchase worked. I then got a call from Apple asking me if I really did intend to buy the computer. "Use Tailscale" On my setup, tailscale can't really navigate the NAT setup correctly. I have a 200Mb downlink and I will often only get a 10th of that through tailscale. Also the routing table no longer works as routing is handled by a myriad of netfilter rules -- which may or may not conflict with docker on the same box. "Use Ubiquiti (or other networking gear) to get rid of the last NAT" Gah. I didn't need to perform a huge cash outlay 10 years ago and rewiring the house to do the same thing I'm trying to do today.
- kazinator 2y ago
  I don't have my own external IP address (even if dynamic) + I want my devices to be spontaneously contactable from the network without polling anything from the inside = does not compute

philjohn 2y ago

Isn't this the whole point of Thread? You have a low power mesh network for inside a deployment (house, office, factory, whatever) and it's the border router which does the "communicate with the internet/lan/wider network" piece. These are typically plugged in (so no worries about having to be low power) have plenty of memory (so no worries about having to drop established routes like a router).

procaryote 2y ago

It seems brave to let an IoT device talk to someone over an unencrypted wan though. They're often of pretty varying software quality and rarely updated. If you really want IoT wifi devices, put them on a separate wifi, and only let them talk to a local device that you can keep up to date. Assume they're vulnerable to local attacks over wifi and act accordingly, e.g. don't give the IoT wifi access to your other devices beyond to that controller, and definitely not to the wider internet. If they're closed source, assume they're already compromised from the factory

vv_ 2y ago
The issue in IoT is that most people expect to be able to control their IoT (e.g. Smart Home) devices from outside of their network. This requires you to have a central server these devices communicate with or a similar deployment on-premises with a public IP address. I've always wondered how it is economically feasible to run these central services without a monthly subscription. If you stop selling devices you'll go under fairly quickly.
- leptons 2y ago
  I have all my "Tuya" IoT devices on a separate network, isolated from my personal intranet, and the cloud servers these devices talk to are in China. I pay nothing to "Tuya" for their cloud service, never have. If the "Tuya" service ever goes down, my home automation is also down, and that sucks. I'm trying to replace it all with "Tasmota" devices now, but it's not quite as easy or as cheap to do.
- procaryote 2y ago
  Yeah, sadly, people care about convenience and not security. So you get things like this: https://www.malwarebytes.com/blog/news/2024/04/ring-agrees-to-pay-5-6-million-after-cameras-were-used-to-spy-on-customers https://www.malwarebytes.com/blog/news/2024/04/ring-agrees-t... and somehow they are still in business, and popular. If you do care about security, keeping your home-automation within your own control is probably the only sane path. Homeassistant and similar open source things like openhab are pretty good if a bit fiddly, and a wireguard vpn like tailscale a fairly practical way to access it when away from home.

johnklos 2y ago

This has poorly considered generalizations and reads like an ad.

hasheddan 2y ago
An ad for... the IETF? All of the firmware discussed in this post is open source, and we even contributed DTLS Connection ID server side support to a popular open source library [0] so other folks can stand up secure cloud services for low power devices. Sure, we sell a product, but our broader mission is making the increasing number of internet connected devices more secure and reliable. When sharing knowledge and technology is in service of that mission, we do not hesitate to do so. [0]: https://blog.golioth.io/golioth-announces-connection-id-support/#cloud-changes https://blog.golioth.io/golioth-announces-connection-id-supp...

apple1417 2y ago

I have worked on a device with this exact same "send a tiny sensor reading every 30 minutes" use case, and this has not been my experience at all. We can run an STM32 and a few sensors at single digit microamps, add an LCD display and a few other niceties and it's one or two dozen. Simply turning on a modem takes hundreds of microamps, if not milliamps. In my experience it's always been better for power consumption to completely shut down the modem and start from scratch each time [1] - which means you're paying to start a new session every time anyway. Now I'll agree it's still inefficient to start up a full TLS session, a protocol like in the post will have it's uses, but I wouldn't blame it on NAT. [1] Doing this of course kills any chance at server-to-device comms, you can only ever apply changes when the device next checks in. This does cause us complaints from time to time, especially for those with longer intervals.

hasheddan 2y ago
Author of this post here -- thanks for sharing your experience! One thing I'll agree with immediately is that if you can afford to power down hardware that is almost always going to be your best option (see a previous post on this topic [0]). I believe the NAT post also calls this out, though I believe I could have gone further to disambiguate "sleeping" and "turning off": > This doesn’t solve the issue of cloud to device traffic being dropped after NAT timeout (check back for another post on that topic), but for many low power use cases, being able to sleep for an extended period of time is more important than being able to immediately push data to devices. (edit: there was originally an unfortunate typo here where the paragraph read "less important" rather than "more important") Depending on the device and the server, powering down the modem does not necessarily mean that a session has to be started from scratch when it is powered on again. In fact, this is one of the benefits of the DTLS Connection ID strategy. A cellular device, for example, could wake up the next time in a completely different location, connect to a new base station, be assigned a fresh IP address, and continue communication with the server without having to perform a full handshake. In reality, there is a spectrum of low power options with modems. We have written about many of them, including a post [1] that followed this one and describes using extended discontinuous reception (eDRX) [2] with DTLS Connection IDs and analyzing power consumption. [0]: https://blog.golioth.io/power-optimization-recommendations/ https://blog.golioth.io/power-optimization-recommendations/ [1]: https://blog.golioth.io/turn-off-subsystems-remotely-to-reduce-power-usage-with-golioth-and-joulescope/ https://blog.golioth.io/turn-off-subsystems-remotely-to-redu... [2]: https://www.everythingrf.com/community/what-is-edrx https://www.everythingrf.com/community/what-is-edrx

lxgr 2y ago

NAT breaks the central idea of IP (i.e. packet switching with stateless intermediary nodes). All of its problems are essentially downstream from there.

londons_explore 2y ago

> This doesn’t solve the issue of cloud to device traffic being dropped after NAT timeout (check back for another post on that topic), This is the key problem. I think it would be best solved by NAT devices having some way of probing their timeout policy, and then notifying the endpoints when they expire a mapping. One way to do that might be for a client to deliberately send a packet to the server with an insufficient TTL. The reply (via ICMP) could then contain fields specifying the timeout in minutes of the mapping, together with some generation number specifying if the mapping table has been cleared due to reboot or overflowed since last queried. There might even be a way to request a specific mapping become long lived - perhaps for months or years. The benefit of all this is that client devices can do push notifications from servers or P2P notifications, all with no polling - allowing for example coin cell devices to last for months or years (with appropriate WiFi protocol improvements too)

mannyv 2y ago

People have been complaining about NAT for decades. It's time to STFU. It's a well-known problem with a bunch of well-known solutions. If you don't like NAT you could go IPv6. But really, why do you need to talk to your device? If it's just reporting in NAT is irrelevant. If you want to do device management just write something into your protocol to check for updates/commands and deal with it on a periodic basis. You can even do that on startup, so you can tell the customer to power cycle the device. It's unlikely that any IoT device needs instant updates, so long periodic updates are probably fine.

dave78 2y ago
There are plenty of IoT devices that people want to execute commands on (anything remotely controlled, basically). Polling for commands on a periodic basis introduces lag into that process which is irritating. Furthermore, polling at a frequent interval can end up using a lot of power as well versus waiting in a receive-only mode for an incoming command.
- raggi 2y ago
  The alternative to polling is unfortunately polling, which is what the article is about. You can avoid polling for messages, but you have to send packets outbound regularly in order to maintain a NAT mapping & connection, so that the external side can send messages inward. The latency is overcome this way, so latency is a solvable problem, but this need to constantly wake up a radio every <30s in order to keep a NAT session alive is a significant power draw. In theory you might be able to avoid this with NAT-PMP / UPnP however their deployments are inconsistent and their server side implementations are extremely buggy.

raggi 2y ago

What I want from platforms, and I fought for at one time in Google with no success: a platform API that provides applications a way to schedule packets when the radio turns on. The mobile platforms in particular continue to assume that you can live with their high level HTTP stuff, and it's just not good enough. The non-mobile platforms largely don't even approach the problem at all.

freedomben 2y ago
Indeed, that sounds like an obvious feature. Hard to believe it hasn't been implemented! I'd love to have that feature on Linux desktop/laptops. I think you could make lots of applications behave a whole lot better.
- raggi 2y ago
  the arguments from folks on the "architecture review boards" was that multiple connections are always bad and that developers can't be trusted. i'm willing to accept that they did get beat up quite a bit over power at various points when at times applications were a big part of the problem. That said this is also a gross misunderstanding of the problem and overall solution space, as well as very much gatekeeping.
  - blitzar 2y ago
    Dump the entire queue to a google server (confirm receipt and shut down the connection) and then have the google server forward all the data to its destinations? Seems to me that would be the lowest power, lowest developer trust, lowest number of connections, maximum gatekeeping method.

clearint 2y ago

This article should clarify at the start whether TCP or UDP is under consideration. NAT idle timeouts for both are typically very different. RFC 5382 [0] specifies no less than 2 hours and 4 minutes for TCP. RFC 4787 [1] specifies no less than 2 minutes for UDP. Towards the end of the article it becomes clear that it's UDP. The example diagrams also incorrectly show port numbers exceeding 65535. The port fields in TCP and UDP headers are 16 bits [2]. [0]: https://www.rfc-editor.org/rfc/rfc5382 https://www.rfc-editor.org/rfc/rfc5382 [1]: https://www.rfc-editor.org/rfc/rfc4787 https://www.rfc-editor.org/rfc/rfc4787 [2]: https://textbook.cs161.org/network/transport.html https://textbook.cs161.org/network/transport.html

apitman 2y ago
That's interesting. Do NATs in the wild tend to be spec-compliant with their timeouts?

VagabundoP 2y ago

Wasn't IPv6 suppose to solve all this? I don't understand why that stalled. Also considering the state of iot security its probably not a great idea to have everything accessible anyway. But that's a slightly different problem to solve.

chgs 2y ago
It’s not going to solve a stateful firewall timing out. You try to continue a tcp session that’s timed out on my firewall and the packets will be dropped. This applies a fair amount t to me when I suspend my laptop, my ssh session will drop as both the server and the firewalls drop the session while it sits there peacefully. When it comes back the tcp packets get sent into the void. Meanwhile my WireGuard connection which runs through two separate ipv4 nats works just fine, as it doesn’t rely on sessions or a server timing out a socket. Nat is irrelevant to the problem.
- VagabundoP 2y ago
  Why not let through all ipv6 traffic? It seems improbable that your device is going to get caught up in a ipv6 scan; the address space is too vast - I'm assuming here. Just have relatively secure devices using ipv6 - not shipping with standard default admin passwords would nearly be enough if there aren't obvious vulnerabilities.

eternityforest 2y ago

Why are we still making IoT devices that talk to anything other than a local hub? They don't need to care about NAT.