4 ms·

I was under the impression there was enough layers of checksums once you get to the level of UDP/TCP that these kind of single bit flips should be detected and

I was under the impression there was enough layers of checksums once you get to the level of UDP/TCP that these kind of single bit flips should be detected and dropped before you get to read them. What's going on here? Is networking gear not calculating checksums like it should?

noncoml 6y ago
TCP header does have a checksum that is supposed to check both the header and the payload. So either the bit corruption is such that is not detected by the checksum Or ATT is doing something nefarious and touches layer 4 and corrupts the data while doing so
- willis936 6y ago
  I don’t have numbers off hand, but from a feeling I have from memory, I would think it is extremely unlikely that TCP checksums are consistently failing to trigger retransmission. Someone must be altering packets along the way.
  - _-___________-_ 6y ago
    It is triggering retransmission. It just retransmits until it gets lucky with the (pretty weak) TCP checksum.
  - cranekam 6y ago
    TCP checksums are notoriously weak: https://www.evanjones.ca/tcp-checksums.html https://www.evanjones.ca/tcp-checksums.html With enough packets passing the dodgy RAM a noticeable number will manage to get mangled in such a way that the checksum is still correct.
    - ericbarrett 6y ago
      Checksums are also often recomputed on transit if the packet is intercepted, e.g. to limit TTL, unflag odd/unused TCP features, that kind of ISP-ish preening. So if it was a software error or even a hardware problem in the right (wrong) spot it’s possible to get this kind of corruption without retransmits.
      - noncoml 6y ago
        I think you are confusing the IP header with TCP header. Routers don’t touch the TCP header at all
        Avery3R 6y ago
        if your router does NAT it touches the tcp header
        noncoml 6y ago
        You are right, although NAT could be classified as a firewall feature, that runs on routers. But in any case it's irrelevant in this context as ATT shouldn't be doing any NAT
        ericbarrett 6y ago
        No. URG, “Christmas tree” packets, etc. all can be mangled by the ISP. Or more commonly dropped
        deleted 6y ago
        [deleted]
    - bytesandbots 6y ago
      In that case, won't there be significant packet loss causing throughput to be very slow? I don't know if this is possible without something messing with TCP headers.
anon946 6y ago
The checksum should be checked by your computer. So somehow the packet is being repackaged with the correct checksum, but for the wrong data. In other words, when your computer checks the checksum, it matches. Another possibility is that somehow only errors that result in the same checksum are being generated.
- toast0 6y ago
  Or their computer isn't checking the checksum. As is apparently the case on mac os (as reported elsewhere in the thread). Another quality product by Apple ;P
  - holycrapwhodat 6y ago
    Except it was trivial to reproduce with the script on non-Apple devices, and people in one of the many Twitter threads surrounding this showed that on their Mac there was MANY tcp retransmits due to invalid checksums, and the bit-flipped packet did have the correct checksum. https://twitter.com/KielO/status/1335640544496689152 https://twitter.com/KielO/status/1335640544496689152
    - toast0 6y ago
      OK, that tweet does show the checksum is OK. I didn't see a whole lot of tcpdumps, so had to go with what was reported in the thread (I tried to reproduce with a few people, but my server wasn't in the broken path, so I couldn't get a lot of real data). That tweet in particular doesn't show any retransmits. tcpdump/wireshark gets a little hard to read at times; especially when the packet dump is a lie: all those packets marked red for bad checksums are from the dumping machine, and the checksums are wrong because the NIC is filling them in, and the capture interface doesn't get to see what they are). Perhaps the other people in the thread who said mac os was ignoring bad checksums were also confused; or perhaps it does ignore bad checksums, it's pretty bad at networking (it can't handle a synflood in 2020 because it's got synhandling code from 2000)
  - cranekam 6y ago
    There’s no proof of this. And what possible reason would macOS have for not checking the checksum? Although the checksum is weak it presumably catches at least some corrupt traffic. Do you really think Apple would just skip the TCP checksum and make its network performance less reliable when they have already implemented (or maintained if it came from BSD) the rest of a TCP/IP stack, which is vastly more complex, just because its developers are lazy?

noncoml 6y ago

TCP header does have a checksum that is supposed to check both the header and the payload. So either the bit corruption is such that is not detected by the checksum Or ATT is doing something nefarious and touches layer 4 and corrupts the data while doing so

willis936 6y ago
I don’t have numbers off hand, but from a feeling I have from memory, I would think it is extremely unlikely that TCP checksums are consistently failing to trigger retransmission. Someone must be altering packets along the way.
- _-___________-_ 6y ago
  It is triggering retransmission. It just retransmits until it gets lucky with the (pretty weak) TCP checksum.
- cranekam 6y ago
  TCP checksums are notoriously weak: https://www.evanjones.ca/tcp-checksums.html https://www.evanjones.ca/tcp-checksums.html With enough packets passing the dodgy RAM a noticeable number will manage to get mangled in such a way that the checksum is still correct.
  - ericbarrett 6y ago
    Checksums are also often recomputed on transit if the packet is intercepted, e.g. to limit TTL, unflag odd/unused TCP features, that kind of ISP-ish preening. So if it was a software error or even a hardware problem in the right (wrong) spot it’s possible to get this kind of corruption without retransmits.
    - noncoml 6y ago
      I think you are confusing the IP header with TCP header. Routers don’t touch the TCP header at all
      - Avery3R 6y ago
        if your router does NAT it touches the tcp header
        noncoml 6y ago
        You are right, although NAT could be classified as a firewall feature, that runs on routers. But in any case it's irrelevant in this context as ATT shouldn't be doing any NAT
      - ericbarrett 6y ago
        No. URG, “Christmas tree” packets, etc. all can be mangled by the ISP. Or more commonly dropped
        deleted 6y ago
        [deleted]
  - bytesandbots 6y ago
    In that case, won't there be significant packet loss causing throughput to be very slow? I don't know if this is possible without something messing with TCP headers.

anon946 6y ago

The checksum should be checked by your computer. So somehow the packet is being repackaged with the correct checksum, but for the wrong data. In other words, when your computer checks the checksum, it matches. Another possibility is that somehow only errors that result in the same checksum are being generated.

toast0 6y ago
Or their computer isn't checking the checksum. As is apparently the case on mac os (as reported elsewhere in the thread). Another quality product by Apple ;P
- holycrapwhodat 6y ago
  Except it was trivial to reproduce with the script on non-Apple devices, and people in one of the many Twitter threads surrounding this showed that on their Mac there was MANY tcp retransmits due to invalid checksums, and the bit-flipped packet did have the correct checksum. https://twitter.com/KielO/status/1335640544496689152 https://twitter.com/KielO/status/1335640544496689152
  - toast0 6y ago
    OK, that tweet does show the checksum is OK. I didn't see a whole lot of tcpdumps, so had to go with what was reported in the thread (I tried to reproduce with a few people, but my server wasn't in the broken path, so I couldn't get a lot of real data). That tweet in particular doesn't show any retransmits. tcpdump/wireshark gets a little hard to read at times; especially when the packet dump is a lie: all those packets marked red for bad checksums are from the dumping machine, and the checksums are wrong because the NIC is filling them in, and the capture interface doesn't get to see what they are). Perhaps the other people in the thread who said mac os was ignoring bad checksums were also confused; or perhaps it does ignore bad checksums, it's pretty bad at networking (it can't handle a synflood in 2020 because it's got synhandling code from 2000)
- cranekam 6y ago
  There’s no proof of this. And what possible reason would macOS have for not checking the checksum? Although the checksum is weak it presumably catches at least some corrupt traffic. Do you really think Apple would just skip the TCP checksum and make its network performance less reliable when they have already implemented (or maintained if it came from BSD) the rest of a TCP/IP stack, which is vastly more complex, just because its developers are lazy?