5 ms·

> [it improves upon] 28-56 bytes [of IPv4 record size] that's still 28-56 MiB for 1024000 (~1M) IP records. Who needs to manipulate more than that?

centimeter 6y ago
Web infrastructure companies keeping track of hundreds of millions of users.
jerf 6y ago
Google, Cisco, Microsoft, AWS, Linux, OpenBSD, ... ... Tailscale... You may not be in the class. I only live next to people in that class, I'm not in it myself. But it exists.
- deleted 6y ago
  [deleted]
- sleepydog 6y ago
  I would think that once you start working with even half that many ips it starts to make sense to use a trie-like data structure that shares common pieces of the ips.
  - tptacek 6y ago
    That doesn't help much if IP addresses are randomly distributed, as they are in a lot of our use cases. You use tries generally to do range and prefix lookups, not so much as a storage trick.
tylersmith 6y ago
Well they manage around 500 addresses for me, and I assume they'd like more than 2k users like me some day.
- dave_universetf 6y ago
  Also, we run on platforms like iOS, where we have very severe memory constraints (the NetworkExtension is allowed 15MiB of total RAM). Given that we juggle a lot of IP-related data types, even modest optimizations help. That said, I'd think of the size thing as more of a constraint on the other desired features. We wanted the other nice things (less GC pressure, no allocations, comparable type...) while also being _lean enough_ that there's no reason not to use the netaddr types.
  - monocasa 6y ago
    Do you have to do anything special from a runtime instance perspective to keep it within 15MB? That seems really small for a GCed language.
    - dave_universetf 6y ago
      Mostly we optimize at the application layer (e.g. we do lazy on-demand WireGuard tunnel configuration, to minimize how much state sits around idle). We also have a small fork of Go with some space-optimizing changes: https://github.com/golang/go/compare/master...tailscale:tailscale.go1.16 https://github.com/golang/go/compare/master...tailscale:tail... (most of these changes get upstreamed, albeit delayed by one release cycle). Aside from that, Go's also just not a big garbage generator in general, compared to older GC'd languages like Java, so with a little care you can keep the pressure off. That said, if anyone at Apple's listening, it would sure be nice to have, say, 30MiB ;). 15MiB makes it possible to cram a whole mesh networking stack in, but doesn't make it fun.
gjulianm 6y ago
Network code tends to be performance critical, so the more you can fit into a cache line the better. Exact numbers depend on the application, of course, but those memory overheads can give you a very serious performance hit in cache-heavy applications.
petee 6y ago
Size is a bonus though, the primary push seems to be all the other issues. Either way imho, needless bloat should be shorn regardless of the ability to accommodate it.
hawski 6y ago
You should also think in terms of CPU caches.
boring_twenties 6y ago
How the heck does one even turn an IPv4 address into 28-56 bytes? Obviously it "should" be exactly 4 bytes, but even if you store the whole thing as an ASCII string it's only up to 16 bytes, including a null terminator. UTF-16, perhaps? LOL
- zamadatix 6y ago
  "It’s large. A Go slice is 3 words (so 24 bytes total on 64-bit machines) just for the slice header, without counting the underlying array that the slice points to"
- bradfitz 6y ago
  > How the heck does one even turn an IPv4 address into 28-56 bytes? It's covered in the blog post.
- evil-olive 6y ago
  from the article: > The underlying type of a net.IP is just a []byte > ... > It’s large. A Go slice is 3 words (so 24 bytes total on 64-bit machines) just for the slice header, without counting the underlying array that the slice points to (background). So an IP address with Go’s net.IP is two parts: the 24 byte slice header, and then also the 4 or 16 bytes of IP address. If you want an IPv6 zone, you have to use net.IPAddr with a 16 byte string header also.
  - boring_twenties 6y ago
    Thanks, that's a bit interesting. I only know a little bit of Go, but I would have assumed you could just use an array ([4]byte) directly without a slice ([]byte) on top of it. Guess not.
    - bradfitz 6y ago
      You can. But an IPv6 address won't fit in that.
    - zeeboo 6y ago
ATsch 6y ago
It's less that you need to store many of them at the same time, but that it increases the amount of garbage collections. If you can create half as much garbage per packet or request, that's double the time until you hit the threshold for gc invocation.

centimeter 6y ago

Web infrastructure companies keeping track of hundreds of millions of users.

jerf 6y ago

Google, Cisco, Microsoft, AWS, Linux, OpenBSD, ... ... Tailscale... You may not be in the class. I only live next to people in that class, I'm not in it myself. But it exists.

deleted 6y ago
[deleted]
sleepydog 6y ago
I would think that once you start working with even half that many ips it starts to make sense to use a trie-like data structure that shares common pieces of the ips.
- tptacek 6y ago
  That doesn't help much if IP addresses are randomly distributed, as they are in a lot of our use cases. You use tries generally to do range and prefix lookups, not so much as a storage trick.

tylersmith 6y ago

Well they manage around 500 addresses for me, and I assume they'd like more than 2k users like me some day.

dave_universetf 6y ago
Also, we run on platforms like iOS, where we have very severe memory constraints (the NetworkExtension is allowed 15MiB of total RAM). Given that we juggle a lot of IP-related data types, even modest optimizations help. That said, I'd think of the size thing as more of a constraint on the other desired features. We wanted the other nice things (less GC pressure, no allocations, comparable type...) while also being _lean enough_ that there's no reason not to use the netaddr types.
- monocasa 6y ago
  Do you have to do anything special from a runtime instance perspective to keep it within 15MB? That seems really small for a GCed language.
  - dave_universetf 6y ago
    Mostly we optimize at the application layer (e.g. we do lazy on-demand WireGuard tunnel configuration, to minimize how much state sits around idle). We also have a small fork of Go with some space-optimizing changes: https://github.com/golang/go/compare/master...tailscale:tailscale.go1.16 https://github.com/golang/go/compare/master...tailscale:tail... (most of these changes get upstreamed, albeit delayed by one release cycle). Aside from that, Go's also just not a big garbage generator in general, compared to older GC'd languages like Java, so with a little care you can keep the pressure off. That said, if anyone at Apple's listening, it would sure be nice to have, say, 30MiB ;). 15MiB makes it possible to cram a whole mesh networking stack in, but doesn't make it fun.

gjulianm 6y ago

Network code tends to be performance critical, so the more you can fit into a cache line the better. Exact numbers depend on the application, of course, but those memory overheads can give you a very serious performance hit in cache-heavy applications.

petee 6y ago

Size is a bonus though, the primary push seems to be all the other issues. Either way imho, needless bloat should be shorn regardless of the ability to accommodate it.

hawski 6y ago

You should also think in terms of CPU caches.

boring_twenties 6y ago

How the heck does one even turn an IPv4 address into 28-56 bytes? Obviously it "should" be exactly 4 bytes, but even if you store the whole thing as an ASCII string it's only up to 16 bytes, including a null terminator. UTF-16, perhaps? LOL

zamadatix 6y ago
"It’s large. A Go slice is 3 words (so 24 bytes total on 64-bit machines) just for the slice header, without counting the underlying array that the slice points to"
bradfitz 6y ago
> How the heck does one even turn an IPv4 address into 28-56 bytes? It's covered in the blog post.
evil-olive 6y ago
from the article: > The underlying type of a net.IP is just a []byte > ... > It’s large. A Go slice is 3 words (so 24 bytes total on 64-bit machines) just for the slice header, without counting the underlying array that the slice points to (background). So an IP address with Go’s net.IP is two parts: the 24 byte slice header, and then also the 4 or 16 bytes of IP address. If you want an IPv6 zone, you have to use net.IPAddr with a 16 byte string header also.
- boring_twenties 6y ago
  Thanks, that's a bit interesting. I only know a little bit of Go, but I would have assumed you could just use an array ([4]byte) directly without a slice ([]byte) on top of it. Guess not.
  - bradfitz 6y ago
    You can. But an IPv6 address won't fit in that.
  - zeeboo 6y ago

ATsch 6y ago

It's less that you need to store many of them at the same time, but that it increases the amount of garbage collections. If you can create half as much garbage per packet or request, that's double the time until you hit the threshold for gc invocation.