9 ms·
Self-hosting a NAT Gateway
- nodesocket 11mo agoI build my own NAT instances from Debian Trixie with Packer on AWS. AWS built-in NAT Gateways use an absurdly outdated and end-of-life version of Amazon Linux and are ridiculously expensive (especially traffic). The bash configuration is literally a few lines: cat <<'EOF' | sudo tee /etc/sysctl.d/99-ip-forwarding.conf > /dev/null net.ipv4.ip_forward=1 EOF sudo sysctl --system sudo iptables -t nat -A POSTROUTING -o ens5 -j MASQUERADE sudo iptables -F FORWARD sudo iptables -A FORWARD -i ens5 -m state --state RELATED,ESTABLISHED -j ACCEPT sudo iptables -A FORWARD -o ens5 -j ACCEPT sudo iptables-save | sudo tee /etc/iptables/rules.v4 > /dev/null Change ens5 with your instance network interface name. Also, VERY IMPORTANT you must set source_dest_check = false on the EC2 NAT instances. Also, don’t assign a EIP to your EC2 NAT instances (unless you absolutely must persist a given public IP) as that counterintuitively routes through public traffic. Just use a auto-assigned public IP (no EIP). NAT instance with EIP - AWS routes it through the public AWS network infrastructure (hairpinning). - You get charged $0.01/GB regional data transfer, even if in the same AZ.
- unquietwiki 11mo agoAssigning an IP is ideal if you're having to whitelist traffic to/from a data center, application, or service.
- nodesocket 11mo agoSure that one’s case, though you might be able to give out a host instead of IP to others to whitelist. Then you just set a low TTL and update the DNS record.
- Nextgrid 11mo agoOpenWrt is also a good option.
- topspin 11mo ago"NAT instances" That's what you did before AWS had the "NAT Gateway" managed service. It's literally called "NAT Instance" in current AWS documentation, and you can implement it in any way you wish. Of course, you don't have to limit yourself to iptables/nftables etc. OPNsense is a great way to do a NAT instance.
- nodesocket 11mo agoI believe the NAT instances also use super old and end-of-life Amazon Linux. I prefer Debian Trixie with Packer and EC2 instances and no EIP. Most secure, performant, and cost effective setup possible. > NAT AMI is built on the last version of the Amazon Linux AMI, 2018.03, which reached the end of standard support on December 31, 2020 and end of maintenance support on December 31, 2023.
- sarathyweb 11mo agoWe can follow the documentation for setting up the NAT instance on any distro. I tested with Rocky Linux 9 and it worked.
- vladvasiliu 11mo ago> Also, don’t assign a EIP to your EC2 NAT instances (unless you absolutely must persist a given public IP) as that counterintuitively routes through public traffic. Just use a auto-assigned public IP (no EIP). Could you point me to somewhere I can read more about this? I didn't know there was an extra charge for using an EIP (other than for the EIP itself).
- m1keil 11mo agoI'm highly skeptical of this claim as well. Going through NATGW with EIP or auto-assigned IP is the exact same cost for the actual traffic.
- notTooFarGone 11mo agoIt's honestly ridiculous that people now see that self hosting is stupidly cheaper and still 99.9% reliable. No your service does not need the extra .099% availability for 100x the price... Make your own VPN while you are at it, wireguard is basically the same config.
- radicaldreamer 11mo agoA lot of this is support. If you’re self hosting, when things don’t work the way they should, the team has no one to blame. On AWS, they can always lean on aws not working the way it should as an excuse.
- deleted 11mo ago[deleted]
- Numerlor 11mo agoFor company hosting cloud solutions gets you the various compliance stuff for free which can be worth it if you're not too large, and of course faster turnaround if you need to get a product out. For personal a cheap vps will end up costing around the same as something you can do on your own, without the risk of messing up your machine/network from a vulnerable endpont
- IgorPartola 11mo agoThis is really it: compliance. The cost is in having to prove that you did the right things. But I do wonder if we will see an easier path forward with that. After all if there was a way to pay someone a once a year fee for an audit and filling out the paperwork and the cost was lower than the cost of using AWS then surely people would do that and it is an opportunity for an audit business that is willing to work with self-hosted setups. Or just have GPT-5 fill out the compliance docs. I suspect it won’t be long until GPT-5 is reading them.
- wiredfool 11mo agoFor free, and 50% or more of your cloud spend.
- heinternets 11mo agoPlease can we do away with NAT forever. Why are we still encouraging this? It’s caused the world to do horrible kludges and continues to do so.
- somanyphotons 11mo agoPresumably the idea is that if you go ipv6-only you can avoid this cost and just use a firewall?
- abcdefg12 11mo agoDon’t even need firewall. Aws has egress only ipv6 gateway.
- waynesonfire 11mo agoIn theory.. but what happens when you want to change ISPs or your ISP doesnt assign static ipv6 blocks? Its recomnended but ISPs have no incentive to give a shit about you. Now all internal infra is not routable.
- ectospheno 11mo agoAn IPv6 allocation being static or dynamic has no bearing on its routability.
- unquietwiki 11mo agoThis shouldn't be mistaken for an anti-IPv6 post. There's also some steps you have to go through to enable IPv6 on your VPS networks, and there's still stuff like GitHub not handling IPv6. So, much as we need to migrate, we still have to support IPv4 connectivity for the foreseeable future. Shoutout to Hacker News for having IPv6 support!
- baby_souffle 11mo ago> and there's still stuff like GitHub not handling IPv6. And virtually everything inside of AWS still requires IPv4 so even if you have zero need to reach out to WAN, if you need any number of private AWS endpoints, you're going to be allocating some ipv4 blocks to your VPC :(.
- up2isomorphism 11mo agoI don't know what is the point of this kind of article. People care the cost and can do it already do it. It is a damn service, which is defined as "you pay someone to do it".
- gerdesj 11mo agoI do both: charge people to do it for them and do it for myself. (your second sentence is a bit confusing)
- stormbeard 11mo agoI can't believe people are paying these crazy amounts for what is basically a fleet of firewalls. What is the difficulty in running VMs with nftables rules?
- gerdesj 11mo agoOr if nft is too complicated (firewalld) then do ufw.
- Nextgrid 11mo ago1) You can't `npm install` it, which is a huge barrier to entry to the modern breed of "engineers". 2) Companies will happily pay thousands in recurring fees for the built-in NAT gateway, but if an engineer asks for even half that as a one-off sum to motivate them to learn Linux networking/firewalling, they'd get a hard no, so why should they bother?
- notatoad 11mo agorunning a VM where? on an ec2 instance? who's going to keep that updated for me? who's going to reprovision it when aws retires the underlying hardware? who's going to monitor it for PCI compliance for me? i don't want to deal with all that. i could dump it on fargate, but at that point it's barely cheaper than just using the official version. i've had to look at my nat gateway zero times since i set it up a couple years ago. i can't say that about any VM host i've got. to me, that's easily worth the few dollars a month that aws charges for it. it's cheaper than hiring somebody, and it's cheaper than me.
- deleted 11mo ago[deleted]
- moduspol 11mo agoIt costs a lot more than a few bucks when you’re putting a lot of traffic through it. And running your own NAT instance does not incur per-GB traffic costs. That said, the paid NAT gateways do also publish metrics. That can be nice when debugging a legitimate issue, such as when your gateway actually runs out of NAT ports to use.
- z7 11mo ago"You only live once." Why state this as absolute fact? Seems a bit lacking in epistemic humility.
- temptemptemp111 11mo ago[dead]
- creatonez 11mo agoEveryone has to address their spiritual beliefs every time they mention something vaguely related to them? Else they lack epistemic humility? ...Did it occur to you that most people have actually thought of this question? Wait, is "seems lacking in epistemic humility" just coded language for "I disagree, therefore you couldn't possibly be thoughtful"?
- deleted 11mo ago[deleted]
- JustExAWS 11mo agoYou only die once - you live everyday…
- api 11mo agoAs an OG networking person, developer, and Linux user, the state of modern dev culture just makes me sad. Modern devs are helpless in the face of things I taught myself to do in a day or two when I was fourteen, and they’re paralyzed with terror at the thought of running something. It’s “hard” goes the cliche. Networking is “hard.” Sys admin is “hard.” Everything is “hard” so you’d better pay an expert to do it. Where do we get these experts? Ever wonder that? It’s just depressing. Why even bother. It really makes me worry about who will keep all this stuff running or build anything new in the future if we are losing not only skills but spine and curiosity. Maybe AI.
- theideaofcoffee 11mo agoAll of this. I despair with some of the takes on basic technology being hard. And when you try to defend understanding just the most rudimentary things, you're labeled a problem because you should just be paying out the nose for the service and writing even more shit code to cover it up.
- amarant 11mo agoI actually kinda think ai will help with this, in a roundabout way. I think of AI as a kind of floor, a minimum required skill to be able to get a job as a professional anything. If you want to find paid work as a developer, you have to at least be better than AI at the job. Optimistically AI will filter out all the helpless Devs who can't get anything done from the job market. "Code monkeys" won't be a thing. Juniors will have to enter unpaid trainee programs I guess, but that might not be such a bad thing
- schrodinger 11mo agoBet you never thought you'd have a "when I was a kid" attitude… :)
- zamadatix 11mo agoFor those who DID think "I wonder what my 'when I was a kid' will be about when I'm old" what kind of things did you guess it'd be and what did it actually end up being? I'm only in my 30s but I was thinking recently "when I'm retired I feel like I'm going to be telling stories about how back in my day we had this thing called the filesystem and you'd just browse it directly..."
- mystraline 11mo agoYeah, I just use a VPS box I pay $20/year for. Only the most basic config goes on this machine. Basically load is 0.1 , and has no data. Then I run my stuff locally. And then I use ssh tunneling to forward the port to localhost of the remote machine. Its a unit file, and will reconstruct the tunnel every 30s if broken. So at most 30s downtime. Then nginx picks it up.
- rmunn 11mo agoBrilliant. I use Tailscale myself, but if you want everything totally under your control (and don't want to go to the trouble of setting up headscale or something similar) then that's one of the absolutely simplest, lowest-effort ways of doing it. EDIT: Well, except for the VPS box I suppose, but if that provider went down or you had any reason to suspect they were doing anything suspicious, it would be quite simple to jump to a different provider, so that's pretty darn close to controlling everything yourself.
- mystraline 11mo agoYep. Ive had the same provider for 3y going on 4. Particular things: I use letsencrypt wildcard, so my subdomains aren't leaked. If you register per subdomain, LE leaks all your subdomains as part of some transparency report. Learned that and had to burn that domain. The VPS is from LowEndBox. Like 2 core, 20GB storage 2GB ram. But runs perfectly fine. I run jellyfin, audiobookshelf, Navidrome, and Romm. Ssh tunnel per application. It would also be trivial to switch providers as well. But again, not a seed box, not doing torrents, not doing anything that would attract attention. And best of all, no evidence on the VPS. Its all SSL and SSH.
- BobbyTables2 11mo agoI do something similar using an openvpn tunnel (home->vps). Iptables rules on the VPS redirect services to the VPN client IP. Client automatically deals with reconnecting, never have to touch it. SSH tunnel would have been simpler, just didn’t want it open. SSH tunnel probably needs the keep alive on, otherwise connection loss may not be detected.
- hk1337 11mo agoI ran a NAT on a floppy back in college, in 2000.
- prmoustache 11mo agoHow do you plug an ethernet cable to the floppy?
- hk1337 10mo agoLinux boots and runs from a floppy disk. No hard drive, and 2 Ethernet cards
- nwellinghoff 11mo agoI think it might be as simple as ipv4 is just nicer to look at…maybe we should have just done “ipv5” and added another block. Eg 1.1.1.1.1. I know its stupid, but ipv6 addresses are just so hard to remember and look at that I think its just human nature to gravitate towards the simplicity of ipv4.
- dboreham 11mo agoThis was all discussed at length in 1993.
- ajsnigrutin 11mo agoThe problem with "add another block" is, that you have to change everything everywhere to make it work... and if you're changing everything, why not expand it properly. Only a tiny minority of people have to look at those addresses, the majority just types "facebook", enter, clicks on first google result and gets facebook (because ".com" is too hard to write).
- theideaofcoffee 11mo agoI have difficulty remembering ten numbers, why do I have to say 1-212-487-1965 when I can just say Santa Rosita 71965? Maybe we should have just done another exchange name and added another name. Eg Hawthorne Santa Rosita 71965. I know its stupid, but 10 digit phone numbers are just so hard to remember and look at that I think its just human nature to gravitate towards the simplicity of telephone exchange prefixes. Yet again, another fundamental misunderstanding (either genuine or not, I'm not sure) about the low-level technologies and their origins that underpin all of this. "Can't we just..."? No.
- denkmoon 11mo agodead::beef is just as memorable as 1.1.1.1, and my v6 delegated prefix is just as unmemorable as my public v4. The "easier to remember" argument just sucks hard.
- miyuru 11mo agoHow did you access HN? is it by typing its IPv4 address?
- mrsssnake 11mo ago> For those unfamiliar, a NAT Gateway acts as a one way door to your private subnet to access the internet without allowing traffic in Repeat after me: NAT is not a firewall. And we need to stop pretending it is.
- varenc 11mo agoAgreed. Assuming an AWS "NAT Gateway" is the same as a regular NAT? Security is not the purpose of a NAT. It's there to give you more IPs than you have. There's all sorts of NAT hole punching techniques. If you want a firewall, you need a firewall.
- Tractor8626 11mo agoBut NAT acts as a one way door to your private subnet, doesn't it?
- zamadatix 11mo agoThe firewall provides the stateful one way door, the router moves packets between the set of subnets it can see, and NAT makes it so things on the public internet think the conversations from one private address+port combo are actually coming from another public address. The last part isn't adding the security, and you can absolutely NAT without preventing the "outside" subnets from being allowed to route to the "inside" subnet, it's just that NAT is almost always done on the box providing the stateful firewall too so people tend to think of the 3 functions as combined in concept as well.
- Tractor8626 11mo ago> you can absolutely NAT without preventing the "outside" subnets from being allowed to route to the "inside" subnet Under very specific conditions. Technically if you send packet with destination 192.168.1.10 directly to wan port of router - yes it can route it inside. The problem - how to deliver this packet over internet. You need to be connected to exactly same network segment to pull it off. And you don't need statefull firewall to deny this kind of packets.
- tonymet 11mo agoIn aws you can use IPv6 with either security groups or EIGW to avoid NAT fees altogether (you still pay for transfer fees ) Death , taxes and transfer fees
- mannyv 11mo agoMoving to IPv6 works until it doesn't.
- t0mas88 11mo agoThat's quite recent. There was some time after AWS started charging for ipv4 addresses where you could not realistically go for an ipv6 only setup behind Cloudfront because it would for example not connect to a v6 only origin. This is probably a result of all AWS services being independent teams with their own release schedule. But it would have made sense for AWS to coordinate this better.
- tonymet 11mo agoYou’re right IPv6 has compatibility issues. But for instances needing NAT gateway (no public ip) , they are often good candidates for IPv6 egress.
- kenrose 11mo agoWe did this at OpsLevel a few years back. Went from AWS managed NAT gateway to fck-nat (Option 1 in the article). It’s a (small) moving part we now have to maintain. But it’s very much worth the massive cost savings in NATGateway-Bytes. A big part of OpsLevel is we receive all kinds of event and payload data from prod systems, so as we grew, so did our network costs. fck-nat turned that growing variable cost into an adorably small fixed one.
- nodesocket 11mo agoI looked at using fck-nat, but decided it was honestly easier to build my own Debian Trixie packer images. See my comment below[1]. How has your experience been with fck-nat? [1] https://news.ycombinator.com/item?id=46010302 https://news.ycombinator.com/item?id=46010302
- dboreham 11mo agoFor anyone else who is super confused as to wtf this is about: 1) it's not "NAT Gateway " but rather "The AWS service called NAT Gateway" and 2) it's not "self-hosting" but "hosting in EC2", in the same sense that "running postgresql on an EC2 instance" wouldn't be "self hosting aurora".
- Arch-TK 11mo agoThe article seems to perpetuate one of those age old myths that NAT has something to do with protection. Yes, in a very superficial sense, you can't literally route a packet over the internet backwards to a host behind NAT without matching a state entry or explicit port forwarding. But implementing NAT on it's own says nothing about the behavior of your router firewall with regards to receiving Martians, or with regards to whether the router firewall itself accepts connections and if the router firewall itself isn't running some service which causes exposure. To actually protect things behind NAT you still need firewall rules and you can keep those rules even when you are not using NAT. Thus those rules, and by extension the protection, are separable from the concept of NAT. This is the kind of weird argument that has caused a lot of people who hadn't ever used IPv6 to avoid trying it.
- gldrk 11mo ago>Yes, in a very superficial sense, you can't literally route a packet over the internet backwards to a host behind NAT without matching a state entry or explicit port forwarding. Don’t forget source routing. That said, depending on your threat model, it’s not entirely unreasonable to just rely on your ISP’s configuration to protect you from stuff like this, specifically behind an IANA private range.
- sedawkgrep 11mo agoI don't think source routing is a thing anymore. At least if you're talking about the ability of a source to specify a path to its destination. The last time I heard about source routing actually being a useful feature or a vulnerability used by hackers was the 1990's.
- globular-toast 11mo agoYeah, I keep meaning to write something about this. I've definitely noticed people wary of IPv6 because their machines get "real" IP addresses rather than the "safe" RFC1918 ones. Of course, having a real IP address is precisely the point of IPv6. It's like we've been collectively trained to think of RFC1918 as "safe" and forgotten what a firewall is. It's one of those "a little knowledge is a dangerous thing" things.
- 0xbadcafebee 11mo agoFwiw, the solutions mentioned here don't seem to properly secure the kernel's network stack against common attacks (rp_filter, accept_redirects, accept_source_route, syncookies, netfilter rules, etc). Ask your local security guru to harden the instance before deploying.
- drchaim 11mo agoI'm not to much into networks, although I've been sysadmin my vps for years. why I would need a NAT Gateway? it's not enough with a good set of rules of ufw or similar software?
- snowfield 11mo agoThis article is ment for companies who want to save a buck on aws costs.
- ygouzerh 11mo agoIt's mostly for AWS context, when you want to host for example your servers in a private subnet, but you want to allow access of small part: you can leverage a NAT Gateway to be the public entrypoint + some security groups as gatekeepers to filter the traffic. However, the fees from AWS are atrocious on the NAT Gateway.
- jimmar 11mo agoAWS already documents a solution to self-host a NAT instance: https://docs.aws.amazon.com/vpc/latest/userguide/work-with-nat-instances.html#create-nat-ami https://docs.aws.amazon.com/vpc/latest/userguide/work-with-n...
- garettmd 11mo agoI always find these discussions about AWS NAT gateways interesting because I recall way back in the day, before AWS had a manages NAT gateway, the recommendation was to roll your own anyway. Or at least that's what I heard. I took an ACloud Guru course and one of the first ec2 lessons was to create a simple NAT gateway in your VPC so that your other instances could reach the Internet.
- 7bit 11mo ago> Society would have you believe that self hosting a NAT Gateway is “crazy”, “irresponsible” and potentially even “dangerous”. I just can't take articles seriously when they lead with these kind of claims and then don't back them up. Typically to give their articles some sort of justification and/or weight. Did not bother to read the rest.
- switknee 10mo agoAm I the only one who read this and thought, "doesn't everyone self host a NAT gateway?" Mine's in the living room, it says TP Link. More seriously, NAT is fun and all but it can introduce unexpected behaviors that wouldn't exist in a firewall that doesn't do translation. Less is more.