11 ms·

Forwarding issues related to MACs starting with 4 or 6

TeMPOraL 10y ago
> I believe IEEE changed their strategy to attempt to purposefully higher the chance of collisions with MAC squatters, to encourage people to register and pay the fee. TIL: MAC squatting is a thing.
- greglindahl 10y ago
  Whenever you have any kind of assignment process, people will squat. If there's a monetary fee involved, more people squat. IEEE's new policy is uncomfortable in another way, it makes it impossible to guess what a new, not-seen-before MAC prefix might be.
  - dfc 10y ago
    Why is it a problem to nit be able to guess a never seen before MAC prefix?
    - greglindahl 10y ago
      If I have a network scanning tool that tells me what's on my network, it's nice to know that an unknown prefix is likely an Apple product, or an HP printer, etc etc.
      - wolrah 10y ago
        http://standards-oui.ieee.org/oui/oui.txt http://standards-oui.ieee.org/oui/oui.txt There you go, straight from the source. Convert that to whatever your scanning tool wants and you can update your list as often as you feel necessary. No need to make assumptions about unknowns when you can just make them known.
        greglindahl 10y ago
        Last time I did this, I found that the downloadable list ran behind reality. That was 2007, maybe it's updated more quickly these days.
        dfc 10y ago
        I have never looked up a MAC and had it not show up. Maybe in 2007 you were using an old copy of the file that came with your distribution.
  - _wmd 10y ago
    There are MAC addresses specifically designed for local use: https://en.wikipedia.org/wiki/MAC_address#Universal_vs._local https://en.wikipedia.org/wiki/MAC_address#Universal_vs._loca... Not sure why you'd otherwise want a list of never-before-seen OUIs
linkregister 10y ago
Can someone break this down for me? Are 4 and 6 previously reserved MAC prefixes?
- pritambaral 10y ago
  No, but it does sound suspiciously like it's related to 'IPv4' and 'IPv6'.
- bandrami 10y ago
  No, but datalink stacks have acted like they are for decades, and it's finally catching up with them.
  - tener 10y ago
    I guess someone calculated that given current rate of sequential assignment of MAC ranges, we won't get to 4 and 6 prefixes in a very long time. That reasoning was broken as it assumed sequential assignment is the only way to go.
    - packetized 10y ago
      Until fairly recently (in IEEE/OUI terms, ~2012), the distribution has been entirely sequential. FTA: "I believe IEEE changed their strategy to attempt to purposefully higher the chance of collisions with MAC squatters, to encourage people to register and pay the fee"
      - bandrami 10y ago
        Nonetheless it was a broken parse: they needed to expend the memory to figure out what the actual protocol is and not jump up from the datalink layer if it's not called for. I guess I'm just saying I really can't blame IEEE here.
        packetized 10y ago
        Oh, I don't blame the IEEE wholly - but I think that they probably changed tack after realizing that a number of manufacturers were simply not paying the fees to actually be assigned a block of OUIs. I view it as kind of a cudgel to be used against vendors, and end-users (read: netops/sysops) are caught in the middle.
kibwen 10y ago
Can someone more knowledgeable explain why the digits 4 or 6 specifically would be causing issues?
- SloopJon 10y ago
  The email references BCP 128: https://tools.ietf.org/html/bcp128 https://tools.ietf.org/html/bcp128 "any application that defines an FEC that does not take measures to prevent the values 0x4 and 0x6 from occurring in the first nibble of the payload may be subject to IP ECMP and thus having their flows take multiple paths and arriving with considerable jitter and possibly out of order."
  - jsjohnst 10y ago
    That's not enough context I feel for most readers. I'm quoting a bit more here which I think will better explain the issue: "In the early days of MPLS, the payload was almost exclusively IP. Even today the overwhelming majority of carried traffic remains IP. Providers of MPLS equipment sought to continue this IP ECMP behavior. As shown above, it is not possible to know whether the payload of an MPLS packet is IP at every place where IP ECMP needs to be performed. Thus vendors have taken the liberty of guessing the payload. By inspecting the first nibble beyond the label stack, existing equipment infers that a packet is not IPv4 or IPv6 if the value of the nibble (where the IP version number would be found) is not 0x4 or 0x6 respectively. Most deployed LSRs will treat a packet whose first nibble is equal to 0x4 as if the payload were IPv4 for purposes of IP ECMP. A consequence of this is that any application that defines an FEC that does not take measures to prevent the values 0x4 and 0x6 from occurring in the first nibble of the payload may be subject to IP ECMP and thus having their flows take multiple paths and arriving with considerable jitter and possibly out of order."
    - deleted 10y ago
      [deleted]
    - jlgaddis 10y ago
      > "In the early days of MPLS, the payload was almost exclusively IP. Even today the overwhelming majority of carried traffic remains IP." You can pretty much carry "Any Transport over MPLS" (AToM). As mentioned, IP is the most common protocol, of course, but you can "tunnel" almost anything you can think of over an MPLS path: Ethernet, ATM, PPP, HDLC, SONET, etc. Basically, you dump a frame into one side and it comes out the other unmodified -- regardless of the network in between. As noted in the linked e-mail, an ISP providing (layer 2) transport has (little to) no control over the payload so there's nothing they can do to prevent running into this issue.
wolfgang42 10y ago
I wonder how many bugs there are like this in low-level networking infrastructure that cause issues which go undiagnosed for years before someone with enough knowledge to diagnose them figures it out. I keep reading about weird networking glitches triggered by specific bit patterns, but certainly if I ever encountered such a think I don't think I'd be able to diagnose it.
- qb45 10y ago
  One WTF I've seen: a consumer router which failed to NAT outgoing packets if their endian-swapped destination IP matched the LAN subnet, i.e. anything sent to X.1.168.192 would be forwarded to WAN with 192.168.1.Y source IP and get lost. Took me a few weeks to figure out why certain random websites don't work.
  - heywire 10y ago
    Reminds me of an issue [1] I ran into with Symantec Firewall looking at UDP fragments as if they were full packets, and blocking them if the value where the port number would be on a full packet matched a rule. [1] https://www.symantec.com/connect/forums/firewall-incorrectly-blocks-fragments-udp-traffic https://www.symantec.com/connect/forums/firewall-incorrectly...
- Rapzid 10y ago
  Not a network code issue but my favorite anecdote for this type of thing. There was a Xen hypervisor bug for YEARS that would cause the clock to jump 20(?) minutes occasionally. The root cause was found in inline assembly due to a compiler oddity. It was found during some bug day/week thing by somebody with the assembly knowledge who just happened to look at that code and see an issue. He had no idea of the significance of what he found :D
  - simcop2387 10y ago
    I'd love to read more about that. Got a link or anything more to google on?
deleted 10y ago
[deleted]
job 10y ago
Good news everyone, Cisco re-examined the bug and concluded this particular problem was not a hardware issue after all but can be remedied with a software update: http://mailman.nanog.org/pipermail/nanog/2016-December/089416.html http://mailman.nanog.org/pipermail/nanog/2016-December/08941...
ChuckMcM 10y ago
Wow it blows my mind. Both that a switch might have issues forwarding packets with a perfectly valid MAC, and that there are so many MAC squatters out there that the IEEE felt this was a good strategy. I guess its going to get harder to use MACs like CA:FE:00:C0:FF:EE to easily spot your packets coming from your home grown FPGA ethernet MAC implementation :-)
- akira2501 10y ago
  > CA:FE:00:C0:FF:EE That's already a locally-administered address as the first byte ends in the hex nibble 'A'. Aside from that, the IEEE's actions are a little odd here, but when you read BCP128[1]; you can see, it's really not their fault. This is due to vendors choosing a wacky and poorly thought out mechanism for identifying packet types on MPLS networks. [1]: https://tools.ietf.org/html/bcp128 https://tools.ietf.org/html/bcp128
bogomipz 10y ago
Wow that's some serious persistence to work a Cisco TAC case all the way up to it becoming a documented bug. I feel for those network engineers. I can only imagine how many hours they spent on the phone with TAC. I've been there, its not fun.
tinus_hn 10y ago
It's wonderful to see how bugs in network hardware are suddenly the problem of the registration authority and they need to delay a few years more. How about the vendors start fixing their bugs instead? And if you have hardware and there will be no fix, consider that the next time your going to spend your money on hardware.
gkafkg8y8 10y ago
> "This 4/6 MAC issue was well documented in BCP128 back in 2007. The control-word drafts mentioned that there would be dragons related to 4 and 6 back in 2004." This reminds me a little of what Mr. Prosser said to Arthur Dent in the Hitchhiker's Guide to the Galaxy about the demolition notice for Arthur's house: "It was on display at the bottom of a locked filing cabinet stuck in a disused lavatory with a sign on the door saying beware of the leopard." You can't make the assumption that engineers will be able to keep all of this in their heads. The IEEE is not going to review every document and put together a compendium of bear traps awaiting them in the future, and they aren't going to review every historic document before every decision, either. If the few that knew what would happen made an incorrect assumption of prior knowledge, then they dropped the ball in reminding those that needed to know. Since nobody remembered, they should have just been informed of the mistake and a possible solution or assistance provided.

> I believe IEEE changed their strategy to attempt to purposefully higher the chance of collisions with MAC squatters, to encourage people to register and pay the fee. TIL: MAC squatting is a thing.

greglindahl 10y ago
Whenever you have any kind of assignment process, people will squat. If there's a monetary fee involved, more people squat. IEEE's new policy is uncomfortable in another way, it makes it impossible to guess what a new, not-seen-before MAC prefix might be.
- dfc 10y ago
  Why is it a problem to nit be able to guess a never seen before MAC prefix?
  - greglindahl 10y ago
    If I have a network scanning tool that tells me what's on my network, it's nice to know that an unknown prefix is likely an Apple product, or an HP printer, etc etc.
    - wolrah 10y ago
      http://standards-oui.ieee.org/oui/oui.txt http://standards-oui.ieee.org/oui/oui.txt There you go, straight from the source. Convert that to whatever your scanning tool wants and you can update your list as often as you feel necessary. No need to make assumptions about unknowns when you can just make them known.
      - greglindahl 10y ago
        Last time I did this, I found that the downloadable list ran behind reality. That was 2007, maybe it's updated more quickly these days.
        dfc 10y ago
        I have never looked up a MAC and had it not show up. Maybe in 2007 you were using an old copy of the file that came with your distribution.
- _wmd 10y ago
  There are MAC addresses specifically designed for local use: https://en.wikipedia.org/wiki/MAC_address#Universal_vs._local https://en.wikipedia.org/wiki/MAC_address#Universal_vs._loca... Not sure why you'd otherwise want a list of never-before-seen OUIs

linkregister 10y ago

Can someone break this down for me? Are 4 and 6 previously reserved MAC prefixes?

pritambaral 10y ago
No, but it does sound suspiciously like it's related to 'IPv4' and 'IPv6'.
bandrami 10y ago
No, but datalink stacks have acted like they are for decades, and it's finally catching up with them.
- tener 10y ago
  I guess someone calculated that given current rate of sequential assignment of MAC ranges, we won't get to 4 and 6 prefixes in a very long time. That reasoning was broken as it assumed sequential assignment is the only way to go.
  - packetized 10y ago
    Until fairly recently (in IEEE/OUI terms, ~2012), the distribution has been entirely sequential. FTA: "I believe IEEE changed their strategy to attempt to purposefully higher the chance of collisions with MAC squatters, to encourage people to register and pay the fee"
    - bandrami 10y ago
      Nonetheless it was a broken parse: they needed to expend the memory to figure out what the actual protocol is and not jump up from the datalink layer if it's not called for. I guess I'm just saying I really can't blame IEEE here.
      - packetized 10y ago
        Oh, I don't blame the IEEE wholly - but I think that they probably changed tack after realizing that a number of manufacturers were simply not paying the fees to actually be assigned a block of OUIs. I view it as kind of a cudgel to be used against vendors, and end-users (read: netops/sysops) are caught in the middle.

kibwen 10y ago

Can someone more knowledgeable explain why the digits 4 or 6 specifically would be causing issues?

SloopJon 10y ago
The email references BCP 128: https://tools.ietf.org/html/bcp128 https://tools.ietf.org/html/bcp128 "any application that defines an FEC that does not take measures to prevent the values 0x4 and 0x6 from occurring in the first nibble of the payload may be subject to IP ECMP and thus having their flows take multiple paths and arriving with considerable jitter and possibly out of order."
- jsjohnst 10y ago
  That's not enough context I feel for most readers. I'm quoting a bit more here which I think will better explain the issue: "In the early days of MPLS, the payload was almost exclusively IP. Even today the overwhelming majority of carried traffic remains IP. Providers of MPLS equipment sought to continue this IP ECMP behavior. As shown above, it is not possible to know whether the payload of an MPLS packet is IP at every place where IP ECMP needs to be performed. Thus vendors have taken the liberty of guessing the payload. By inspecting the first nibble beyond the label stack, existing equipment infers that a packet is not IPv4 or IPv6 if the value of the nibble (where the IP version number would be found) is not 0x4 or 0x6 respectively. Most deployed LSRs will treat a packet whose first nibble is equal to 0x4 as if the payload were IPv4 for purposes of IP ECMP. A consequence of this is that any application that defines an FEC that does not take measures to prevent the values 0x4 and 0x6 from occurring in the first nibble of the payload may be subject to IP ECMP and thus having their flows take multiple paths and arriving with considerable jitter and possibly out of order."
  - deleted 10y ago
    [deleted]
  - jlgaddis 10y ago
    > "In the early days of MPLS, the payload was almost exclusively IP. Even today the overwhelming majority of carried traffic remains IP." You can pretty much carry "Any Transport over MPLS" (AToM). As mentioned, IP is the most common protocol, of course, but you can "tunnel" almost anything you can think of over an MPLS path: Ethernet, ATM, PPP, HDLC, SONET, etc. Basically, you dump a frame into one side and it comes out the other unmodified -- regardless of the network in between. As noted in the linked e-mail, an ISP providing (layer 2) transport has (little to) no control over the payload so there's nothing they can do to prevent running into this issue.

wolfgang42 10y ago

I wonder how many bugs there are like this in low-level networking infrastructure that cause issues which go undiagnosed for years before someone with enough knowledge to diagnose them figures it out. I keep reading about weird networking glitches triggered by specific bit patterns, but certainly if I ever encountered such a think I don't think I'd be able to diagnose it.

qb45 10y ago
One WTF I've seen: a consumer router which failed to NAT outgoing packets if their endian-swapped destination IP matched the LAN subnet, i.e. anything sent to X.1.168.192 would be forwarded to WAN with 192.168.1.Y source IP and get lost. Took me a few weeks to figure out why certain random websites don't work.
- heywire 10y ago
  Reminds me of an issue [1] I ran into with Symantec Firewall looking at UDP fragments as if they were full packets, and blocking them if the value where the port number would be on a full packet matched a rule. [1] https://www.symantec.com/connect/forums/firewall-incorrectly-blocks-fragments-udp-traffic https://www.symantec.com/connect/forums/firewall-incorrectly...
Rapzid 10y ago
Not a network code issue but my favorite anecdote for this type of thing. There was a Xen hypervisor bug for YEARS that would cause the clock to jump 20(?) minutes occasionally. The root cause was found in inline assembly due to a compiler oddity. It was found during some bug day/week thing by somebody with the assembly knowledge who just happened to look at that code and see an issue. He had no idea of the significance of what he found :D
- simcop2387 10y ago
  I'd love to read more about that. Got a link or anything more to google on?

deleted 10y ago

[deleted]

job 10y ago

Good news everyone, Cisco re-examined the bug and concluded this particular problem was not a hardware issue after all but can be remedied with a software update: http://mailman.nanog.org/pipermail/nanog/2016-December/089416.html http://mailman.nanog.org/pipermail/nanog/2016-December/08941...

ChuckMcM 10y ago

Wow it blows my mind. Both that a switch might have issues forwarding packets with a perfectly valid MAC, and that there are so many MAC squatters out there that the IEEE felt this was a good strategy. I guess its going to get harder to use MACs like CA:FE:00:C0:FF:EE to easily spot your packets coming from your home grown FPGA ethernet MAC implementation :-)

akira2501 10y ago
> CA:FE:00:C0:FF:EE That's already a locally-administered address as the first byte ends in the hex nibble 'A'. Aside from that, the IEEE's actions are a little odd here, but when you read BCP128[1]; you can see, it's really not their fault. This is due to vendors choosing a wacky and poorly thought out mechanism for identifying packet types on MPLS networks. [1]: https://tools.ietf.org/html/bcp128 https://tools.ietf.org/html/bcp128

bogomipz 10y ago

Wow that's some serious persistence to work a Cisco TAC case all the way up to it becoming a documented bug. I feel for those network engineers. I can only imagine how many hours they spent on the phone with TAC. I've been there, its not fun.

tinus_hn 10y ago

It's wonderful to see how bugs in network hardware are suddenly the problem of the registration authority and they need to delay a few years more. How about the vendors start fixing their bugs instead? And if you have hardware and there will be no fix, consider that the next time your going to spend your money on hardware.

gkafkg8y8 10y ago

> "This 4/6 MAC issue was well documented in BCP128 back in 2007. The control-word drafts mentioned that there would be dragons related to 4 and 6 back in 2004." This reminds me a little of what Mr. Prosser said to Arthur Dent in the Hitchhiker's Guide to the Galaxy about the demolition notice for Arthur's house: "It was on display at the bottom of a locked filing cabinet stuck in a disused lavatory with a sign on the door saying beware of the leopard." You can't make the assumption that engineers will be able to keep all of this in their heads. The IEEE is not going to review every document and put together a compendium of bear traps awaiting them in the future, and they aren't going to review every historic document before every decision, either. If the few that knew what would happen made an incorrect assumption of prior knowledge, then they dropped the ball in reminding those that needed to know. Since nobody remembered, they should have just been informed of the mistake and a possible solution or assistance provided.