5 ms·

Is there a tl;dr? Who would be affected? Sounds scary.

by derFunk 11y ago

djb_hackernews 11y ago
You'll be affected if you use a veth (becoming more popular with docker/container schedulers) and have a corrupt packet floating through your network.
- devit 11y ago
  ... and are not using authenticated encryption as you should.
  - djb_hackernews 11y ago
    In my experience, there is a bit of hardware (which was the root cause in the articles case) between SSL termination and application servers. So even using encryption, you are still vulnerable.
  - tyingq 11y ago
    In many situations, you might still have unencrypted traffic, even if your app is using authenticated encryption. Like, for example, if you're doing DNS lookups, or syslog to a remote host, etc.
  - deleted 11y ago
    [deleted]
geofft 11y ago
Virtual ethernet devices, which are used in some container deployments (you / your ops team probably know if you're using them), do not check TCP checksums. This appears to be because the original programmer was thinking about applications on the same machine communicating across virtual ethernet devices, but the optimization also affected traffic from a physical network that was routed onto a virtual network. If your physical network corrupts data, TCP is supposed to notice the checksum mismatch and drop the packet, and wait for it to be retransmitted. Because of this bug, Linux's TCP implementation was not validating checksums, which allowed corrupt data to reach the application. This requires a faulty physical network, which is rare but nowhere near nonexistent. (The kernel is not introducing corruption to these packets.)
tyingq 11y ago
It does mention toggling off the veth device "checksum offloading" as a valid workaround.
- vijayp 11y ago
  yes, the code is as follows in the broken veth: if (skb->ip_summed == CHECKSUM_NONE && rcv->features & NETIF_F_RXCSUM) checksum offloading is encapsulated in the rcv-features bitmap, so disabling it will hide this bug. You can do something like this within your container to disable it (from memory, might be slightly off): $ ethtool --offload VETH_DEVICE_NAME rx off tx off $ ethtool -K VETH_DEVICE_NAME gso off
- evanj 11y ago
  I've been trying to get Docker to include this workaround when it creates containers to ensure people don't run into it, but this has not gotten any attention: https://github.com/docker/docker/issues/18776 https://github.com/docker/docker/issues/18776 Mesos has a workaround like this in it now.

Is there a tl;dr? Who would be affected? Sounds scary.

by derFunk 11y ago

Is there a tl;dr? Who would be affected? Sounds scary.

You'll be affected if you use a veth (becoming more popular with docker/container schedulers) and have a corrupt packet floating through your network.

devit 11y ago
... and are not using authenticated encryption as you should.
- djb_hackernews 11y ago
  In my experience, there is a bit of hardware (which was the root cause in the articles case) between SSL termination and application servers. So even using encryption, you are still vulnerable.
- tyingq 11y ago
  In many situations, you might still have unencrypted traffic, even if your app is using authenticated encryption. Like, for example, if you're doing DNS lookups, or syslog to a remote host, etc.
- deleted 11y ago
  [deleted]

geofft 11y ago

Virtual ethernet devices, which are used in some container deployments (you / your ops team probably know if you're using them), do not check TCP checksums. This appears to be because the original programmer was thinking about applications on the same machine communicating across virtual ethernet devices, but the optimization also affected traffic from a physical network that was routed onto a virtual network. If your physical network corrupts data, TCP is supposed to notice the checksum mismatch and drop the packet, and wait for it to be retransmitted. Because of this bug, Linux's TCP implementation was not validating checksums, which allowed corrupt data to reach the application. This requires a faulty physical network, which is rare but nowhere near nonexistent. (The kernel is not introducing corruption to these packets.)

tyingq 11y ago

It does mention toggling off the veth device "checksum offloading" as a valid workaround.

vijayp 11y ago
yes, the code is as follows in the broken veth: if (skb->ip_summed == CHECKSUM_NONE && rcv->features & NETIF_F_RXCSUM) checksum offloading is encapsulated in the rcv-features bitmap, so disabling it will hide this bug. You can do something like this within your container to disable it (from memory, might be slightly off): $ ethtool --offload VETH_DEVICE_NAME rx off tx off $ ethtool -K VETH_DEVICE_NAME gso off
evanj 11y ago
I've been trying to get Docker to include this workaround when it creates containers to ensure people don't run into it, but this has not gotten any attention: https://github.com/docker/docker/issues/18776 https://github.com/docker/docker/issues/18776 Mesos has a workaround like this in it now.