19 ms·
netaddr.IP: a new IP address type for Go
- loa_in_ 6y ago> [it improves upon] 28-56 bytes [of IPv4 record size] that's still 28-56 MiB for 1024000 (~1M) IP records. Who needs to manipulate more than that?
- centimeter 6y agoWeb infrastructure companies keeping track of hundreds of millions of users.
- jerf 6y agoGoogle, Cisco, Microsoft, AWS, Linux, OpenBSD, ... ... Tailscale... You may not be in the class. I only live next to people in that class, I'm not in it myself. But it exists.
- deleted 6y ago[deleted]
- sleepydog 6y agoI would think that once you start working with even half that many ips it starts to make sense to use a trie-like data structure that shares common pieces of the ips.
- tptacek 6y agoThat doesn't help much if IP addresses are randomly distributed, as they are in a lot of our use cases. You use tries generally to do range and prefix lookups, not so much as a storage trick.
- tylersmith 6y agoWell they manage around 500 addresses for me, and I assume they'd like more than 2k users like me some day.
- dave_universetf 6y agoAlso, we run on platforms like iOS, where we have very severe memory constraints (the NetworkExtension is allowed 15MiB of total RAM). Given that we juggle a lot of IP-related data types, even modest optimizations help. That said, I'd think of the size thing as more of a constraint on the other desired features. We wanted the other nice things (less GC pressure, no allocations, comparable type...) while also being _lean enough_ that there's no reason not to use the netaddr types.
- monocasa 6y agoDo you have to do anything special from a runtime instance perspective to keep it within 15MB? That seems really small for a GCed language.
- dave_universetf 6y agoMostly we optimize at the application layer (e.g. we do lazy on-demand WireGuard tunnel configuration, to minimize how much state sits around idle). We also have a small fork of Go with some space-optimizing changes: https://github.com/golang/go/compare/master...tailscale:tailscale.go1.16 https://github.com/golang/go/compare/master...tailscale:tail... (most of these changes get upstreamed, albeit delayed by one release cycle). Aside from that, Go's also just not a big garbage generator in general, compared to older GC'd languages like Java, so with a little care you can keep the pressure off. That said, if anyone at Apple's listening, it would sure be nice to have, say, 30MiB ;). 15MiB makes it possible to cram a whole mesh networking stack in, but doesn't make it fun.
- gjulianm 6y agoNetwork code tends to be performance critical, so the more you can fit into a cache line the better. Exact numbers depend on the application, of course, but those memory overheads can give you a very serious performance hit in cache-heavy applications.
- petee 6y agoSize is a bonus though, the primary push seems to be all the other issues. Either way imho, needless bloat should be shorn regardless of the ability to accommodate it.
- hawski 6y agoYou should also think in terms of CPU caches.
- boring_twenties 6y agoHow the heck does one even turn an IPv4 address into 28-56 bytes? Obviously it "should" be exactly 4 bytes, but even if you store the whole thing as an ASCII string it's only up to 16 bytes, including a null terminator. UTF-16, perhaps? LOL
- zamadatix 6y ago"It’s large. A Go slice is 3 words (so 24 bytes total on 64-bit machines) just for the slice header, without counting the underlying array that the slice points to"
- bradfitz 6y ago> How the heck does one even turn an IPv4 address into 28-56 bytes? It's covered in the blog post.
- evil-olive 6y agofrom the article: > The underlying type of a net.IP is just a []byte > ... > It’s large. A Go slice is 3 words (so 24 bytes total on 64-bit machines) just for the slice header, without counting the underlying array that the slice points to (background). So an IP address with Go’s net.IP is two parts: the 24 byte slice header, and then also the 4 or 16 bytes of IP address. If you want an IPv6 zone, you have to use net.IPAddr with a 16 byte string header also.
- boring_twenties 6y agoThanks, that's a bit interesting. I only know a little bit of Go, but I would have assumed you could just use an array ([4]byte) directly without a slice ([]byte) on top of it. Guess not.
- ATsch 6y agoIt's less that you need to store many of them at the same time, but that it increases the amount of garbage collections. If you can create half as much garbage per packet or request, that's double the time until you hit the threshold for gc invocation.
- nynx 6y agoWhat an insane amount of work for something that would be fixed, in a better language, by better value types and algebraic enums.
- Thaxll 6y ago5 days of work is insane for you?
- nynx 6y agoNo, the time isn't really what I'm referring to—ip addressing is complicated. But it shouldn't require having to work around the language to do this kind of thing.
- tick_tock_tick 6y agoI'm not sure how this is a "work around the language" rather then the standard library in Golang is allocation happy for IP types. I imagine this would look very similar in almost any language.
- nynx 6y agoFor example, look at rust's [0]. It claims to vary in size depending on os, but on Linux, at least, it's 20 bytes. [0]: https://doc.rust-lang.org/std/net/enum.IpAddr.html https://doc.rust-lang.org/std/net/enum.IpAddr.html
- tick_tock_tick 6y agoThis type doesn't appear to support IPv6 Address Zones which was the most complicated part of the post. Actually from my limited googling Rust doesn't seem to support Address Zones at all in it's standard library.
- nynx 6y agoThat's a good point.
- tkiolp4 6y agoYear and a half of work. Impressive. I wonder if it’s actually worth it for a company to spend such amount of time on stuff like this. Nevertheless, I really appreciate it’s open source.
- gigatexal 6y agoI trust Brad and the team’s take on things. If they decide they needed a new library for this then so be it.
- xrd 6y agoI really wonder whether there is an executive at Google who has some connection to revenue for Google and how control of Golang ties to that. And, whether Brad leaving makes them really nervous. Google has never, AFAIK, invested in Jetbrains, but they care a lot about Android Studio and Kotlin. It's interesting to see this kind of post and wonder if this kind of work makes Google nervous at all. That would be a good book I'd like to read in 10 or 20 years.
- sdwvit 6y agoOff topic question: do you think go is the next big thing?
- 1f60c 6y agoCool! One thing that confused me, though, is why they used the inet.af import path. I get that it looks cool, but it'd be better if they just used tailscale.com for everything. On a related note, I'm not sure which problems Tailscale aims to solve (although I suspect it's one of those "you'll know when you need it" tools).
- edoceo 6y agoAF for Address Family, common in socket-y things
- nine_k 6y agoThis should point to AF_INET flag which you pass when creating a socket, and means that the address family is the internet (IP) address, as opposed to e.g. a domain socket, or some other transport.
- bradfitz 6y ago> Cool! One thing that confused me, though, is why they used the inet.af import path. I get that it looks cool, but it'd be better if they just used tailscale.com for everything. Why would it be better to use tailscale.com? It has nothing to do with Tailscale. It's just an IP address. A neutral package name seemed best.
- lnsp 6y agoLove the redirect from inet.af to the POSIX socket spec, nice easter egg :)
- luxurytent 6y agoI just want to say I love how the information is provided and broken down in this blog post. Masterclass.
- teraflop 6y agoNitpick: the correct title of the blog post starts with "netaddr.IP", but the one currently on HN starts with "netaddr.ip". The former looks like the name of a Go type (which is what it is) and the latter looks like a domain name or something.
- ATsch 6y agoReading the description of the stdlib type, I can't help but be reminded of the statement that the go way of solving a problem is to ignore real-world problems and half-ass things for the sake of a simplicity that merely pushes the complexity elsewhere. It certainly does seem to apply here too. Glad this library exists now though.
- staunch 6y agoThat's a pretty lazy criticism given how much of the most modern and popular networking and system software is written in Go (Kubernetes, Docker, all the related infrastructure, etc). The "net" standard library is still very nice and works extremely well. The proof of the pudding is in the eating, and the Go pudding has been eaten successfully many times.
- deleted 6y ago[deleted]
- zelon88 6y agoAnd Facebook was written in PHP. That hardly means that PHP doesn't have flaws or quirks or drawbacks. And to your other point, I've never used docker or kubernetes and I am alive and fully functional without them. As far as I'm concerned containers are a gimmick just like VPNs and just like go. It's a very hands off and ineffective approach to security that just makes you more comfortable being more lazy.
- blablabla123 6y agoI remember a time when Facebook would make a quite buggy appearance for a year or two. Must have been around 2015. Docker and k8s are tools to make Linux cgroups more accessible for the user. In total they add more structure to files and processes. I cannot really say that about VPNs since OpenVPN for instance just overwrites all my network settings. But I guess it's possible to write bad code in every language.
- kortilla 6y ago
- vlovich123 6y agoPerhaps I missed something (and I don’t know Go at all), but if you’re keeping a background map in-sync for the pointer equality stuff to work out, doesn’t that imply it’s not actually 0-allocation? Or is this a go’ism where allocations refer to some subset of things?
- dave_universetf 6y agoIt's a mild tradeoff the post glossed over: if you're handling IPv6 addresses with a zone specifier, the first use of a particular zone specifier will cause an allocation to intern that string. Thankfully, the vast majority of IP-wielding programs never touch zones, and IPs sans zone specifiers are always allocation-free. And even for the long tail of zone-aware programs (e.g. corerad, a router advertisement daemon), you end up with one allocation per unique zone string, not one per IP value, which generally amortizes to alloc-free in steady state.
- bradfitz 6y agoThat's only used for the IPv6 scopes and it's amortized over all parses to effectively zero. Only allocates once per unique new scope name seen. I could clarify in the post.
- benhoyt 6y agoI don't quite understand why you need the messy/unsafe string interning, instead of a simple (mutex-protected) map with the m[string(b)] optimization that avoids allocations, or josharian's sync.Pool approach? https://commaok.xyz/post/intern-strings/ https://commaok.xyz/post/intern-strings/
- zeeboo 6y agoA simple map was discussed in the post. The problem it has is that it has unbounded growth. The sync.Pool approach is a best effort to avoid allocations only. If its API was changed to return sentinel pointers rather than the string, it wouldn't have the guarantee that Intern(x) == Intern(x) when x == x.
- gyrgtyn 6y agoumm
- gilgad13 6y agoThe uintptr / SetFinalize trick to implement "weak" references is pretty cool, but I'd be curious to hear why type IP struct { hi, lo uint64 z string } with sentinel strings like "\04" and "\0\06" for IPv4 and IPv6-no-zone was rejected. Sure, this 8 bytes larger for the implicit string length, but its still not terribly large, and the string comparisons in the normal IPv4 / IPv6 case are very cheap (just length and pointer equality checks). And it doesn't require opening the unsafe jar. Was the extra 8 bytes enough to disqualify it? Or are the authors worried about the fact that this may allocate on each parse of an IPv6-with-zone?
- bradfitz 6y agoYeah, that would be the obvious representation if we were willing to go to 32 bytes. (on 64-bit) But paying 8x memory cost for IPv4 and 2x for IPv6 was a bit hard to swallow. Especially considering how rare IPv6+zones are. And we didn't want to have be worse in any regard compared to std's net.IP. If we went to 32 bytes then Go's net.IP would win if you parsed an IP and stored it multiple times. Because we'd be a 32 byte value type and Go would be a 24 byte value type containing a pointer to the shared-for-all-copies address bits. Sure, they'd all be aliasing the same memory (danger zone!) but in Go the culture is to behave and risk that. But staying at 24 bytes, we're never worse than net.IP and strictly better in all regards. (at least by the metrics we highlighted in the blog post... perhaps not by "don't play unsafe shenanigans with IPv6 zone interning")
- gilgad13 6y agoFair enough. I agree 8x for IPv4 would be hard to swallow. In either case, having the interning wrapped up in a reusable library is great.
- ash 6y ago> Especially considering how rare IPv6+zones are. Who uses IPv6+zones and why? I'm confused what zones are about. They don't seem to be part of IPv6 packets. Where do they come from? Why does IP type have to deal with them?
- mleonhard 6y agoThe brittle string interning implementation could be replaced with a proposed Golang weak reference map [0]. [0] https://github.com/golang/go/issues/43615 https://github.com/golang/go/issues/43615
- ericcholis 6y ago"But then I thought of something gross. Or awesome." The programmer's mantra.
- OJFord 6y ago> The story begins in April 2019 with 89476f8cb5 in which David Crawshaw, aware of all these problems, created an IP type like: It's in a PR here: https://github.com/WireGuard/wireguard-go/pull/11/files https://github.com/WireGuard/wireguard-go/pull/11/files Which mentions: > Typically throughout WireGuard, we've used [4]byte for v4 and [16]byte for v6, considering the Go standard library's choice of v6-mapped-v4 to be a mistake. It actually looks useful, there's something similar in wgtypes, and I wanted to use it recently [0] but that package unfortunately doesn't expose a way to write it out to a string/buffer, so I ended up almost replicating it. [0] https://github.com/OJFord/terraform-provider-wireguard/blob/master/provider/data_source_wireguard_config_document.go https://github.com/OJFord/terraform-provider-wireguard/blob/... (There's a lot of ways of presenting/parsing/constructing the same config options there! I'm sure if I were more fluent in Go there's some more succinct way.)
- trefil 6y agoThe amount of hoops the author has to jump through just because Go doesn't support overloading the == operator is ridiculous.
- skrebbel 6y agoI know nothing about Go or IP, but i do know Erlang where atoms (= interned strings) are a little bit of a security risk because it makes it very easy for an attacker to cause out-of-memory by flooding an endpoint that converts input strings to atoms with random data. Is that a risk here? Could i connect to a Go service that uses this library with a patched IP client and somehow push random data into that "zone" string and crash the server?
- kortex 6y ago> Basically, it’s playing unsafe games behind the Go garbage collector’s back, hiding pointers in untyped uintptr integers so Go will be forced to eventually garbage collect things which then causes the finalizer to be invoked to step in and either clean up its lies or clean up the map. I love this. I hate to love this. It has that fast inverse sqrt flavor to it. Does some crazy stuff, but cordons off the well-documented crazy behind a clean API. Bravo! Golang "worse is better" at it's finest. Gonna send this to my FP-obsessed friend who thinks humans can never be trusted to write code like this and all future languages should forbid any kind of manual memory management. I told him "have fun writing a kernel or driver in that kind of language", he wasn't having it. I'm sure this will irk him to no end.
- tapirl 6y agoI don't understand why SetFinalizer+resurrected could prevent a memory block which will be collected from being collected. Update: never mind, I re-read the docs of SetFinalizer to get the explanation: SetFinalizer sets the finalizer associated with obj to the provided finalizer function. When the garbage collector finds an unreachable block with an associated finalizer, it clears the association and runs finalizer(obj) in a separate goroutine. This makes obj reachable again, but now without an associated finalizer. Assuming that SetFinalizer is not called again, the next time the garbage collector sees that obj is unreachable, it will free obj.
- pjmlp 6y agoThen we will refer you to Midori powering the Asian nodes for Bing, until Microsoft management decided to pull the plug. Or how F-Secure is shipping secure USB keys with bare metal Go firmware.
- Fire-Dragon-DoL 6y agoIsn't intern essentially the behavior of a Ruby symbol? If that's the case it's a very powerful package, no doubt