3 ms·

I mean, ultimately, thats how Google routes internally. IPSec-equivalent, VXLAN-equivalent, IPSec-equivalent. Prevents any compromised layer from knowing too

I mean, ultimately, thats how Google routes internally. IPSec-equivalent, VXLAN-equivalent, IPSec-equivalent. Prevents any compromised layer from knowing too much about the traffic.

pixl97 9mo ago
Internal is fine because you control things like MTU so you don't have to worry about packet fragmentation/partial loss.
als0 9mo ago
That seems like an awful amount of overhead for questionable gain.
- _bernd 9mo ago
  Links between, and in between data centers use so called jumbo frames with an mtu of over 9000. Not joking.
  - xoa 9mo ago
    Worth mentioning that links at home can use them too, jumbo frame support was rare at one point but now you can get them on really cheap basic switches if you're looking for it. Even incredibly cheap $30 (literally, that's what a 5 port UniFi flex mini lists for direct) switches support them now. Not just an exotic thing for data centers anymore, and it can cut down on overhead within a LAN particularly as you get into 10/25/40/100 Gbps stuff to your own NAS/SAN or whatever.
tucnak 9mo ago
What gave you that idea? Internally, Google uses GRE/GENEVE-like stuff but for reasons that have nothing to do with "preventing compromise" or whatever, but because they're carrying metadata (traces, latency budgets, billing ids.) That is to say, encapsulation is just transport. It's pretty much L3 semantics all the way down... In fact, this is more or less the point: L2 is intractable at scale, as broadcast/multicast doesn't work. However, it's hard to find comparisons to anything you're familiar with at Google scale. They have a myriad of proprietary solutions and custom protocols for routing, even though it's all L3 semantics. To learn more: Andromeda https://research.google/pubs/andromeda-performance-isolation-and-velocity-at-scale-in-cloud-network-virtualization/ https://research.google/pubs/andromeda-performance-isolation... Orion https://research.google/pubs/orion-googles-software-defined-networking-control-plane/ https://research.google/pubs/orion-googles-software-defined-...
- DiabloD3 9mo ago
  What gave me that idea? Talks and research papers from Google network engineers over the past decade.
  - tucnak 9mo ago
    Where are you getting at VXLAN-equiv, IPsec-equiv, etc. specifically? ALTS/PSP is not "IPsec-equivalent"
- zeroxfe 9mo ago
  The last time I was there, there were many layers of encap, including MPLS, GRE, PSP, with very tightly managed MTU. Traffic engineering was mostly SDN-managed L3, but holy hell was it complex. Considering that Google (at the time) carried more traffic than the rest of the Internet combined, maybe it was worth it.

I mean, ultimately, thats how Google routes internally. IPSec-equivalent, VXLAN-equivalent, IPSec-equivalent. Prevents any compromised layer from knowing too much about the traffic.

pixl97 9mo ago

Internal is fine because you control things like MTU so you don't have to worry about packet fragmentation/partial loss.

als0 9mo ago

That seems like an awful amount of overhead for questionable gain.

_bernd 9mo ago
Links between, and in between data centers use so called jumbo frames with an mtu of over 9000. Not joking.
- xoa 9mo ago
  Worth mentioning that links at home can use them too, jumbo frame support was rare at one point but now you can get them on really cheap basic switches if you're looking for it. Even incredibly cheap $30 (literally, that's what a 5 port UniFi flex mini lists for direct) switches support them now. Not just an exotic thing for data centers anymore, and it can cut down on overhead within a LAN particularly as you get into 10/25/40/100 Gbps stuff to your own NAS/SAN or whatever.

tucnak 9mo ago

What gave you that idea? Internally, Google uses GRE/GENEVE-like stuff but for reasons that have nothing to do with "preventing compromise" or whatever, but because they're carrying metadata (traces, latency budgets, billing ids.) That is to say, encapsulation is just transport. It's pretty much L3 semantics all the way down... In fact, this is more or less the point: L2 is intractable at scale, as broadcast/multicast doesn't work. However, it's hard to find comparisons to anything you're familiar with at Google scale. They have a myriad of proprietary solutions and custom protocols for routing, even though it's all L3 semantics. To learn more: Andromeda https://research.google/pubs/andromeda-performance-isolation-and-velocity-at-scale-in-cloud-network-virtualization/ https://research.google/pubs/andromeda-performance-isolation... Orion https://research.google/pubs/orion-googles-software-defined-networking-control-plane/ https://research.google/pubs/orion-googles-software-defined-...

DiabloD3 9mo ago
What gave me that idea? Talks and research papers from Google network engineers over the past decade.
- tucnak 9mo ago
  Where are you getting at VXLAN-equiv, IPsec-equiv, etc. specifically? ALTS/PSP is not "IPsec-equivalent"
zeroxfe 9mo ago
The last time I was there, there were many layers of encap, including MPLS, GRE, PSP, with very tightly managed MTU. Traffic engineering was mostly SDN-managed L3, but holy hell was it complex. Considering that Google (at the time) carried more traffic than the rest of the Internet combined, maybe it was worth it.