5 ms·

Mixing 1Gbps, 2.5Gbps and 10Gbps has sort of been a nightmare for me on Mikrotik at least. My WAN port is 2.5Gbps and you would expect that I get full speed 112

Mixing 1Gbps, 2.5Gbps and 10Gbps has sort of been a nightmare for me on Mikrotik at least. My WAN port is 2.5Gbps and you would expect that I get full speed 112MB/s Internet on the 1Gbps LAN ports, but that only happens if I force the WAN port to negotiate at 1Gbps. If I leave it at 2.5Gbps, then I get something like 50MB/s instead, and it oscillates +- >20MB/s during the life of that TCP connection. I don't think it's entirely a Mikrotik issue as I believe this is just a physics thing - you're firing off one port at a certain speed so the buffer of the other port fills up too quickly unless the faster one slows down (resulting in excessive packet loss.) But it looks like Mikrotik has the most complaints about this. The solution appears to be to enable Flow Control, but it's never clear in which direction or which port it needs to be enabled on, and I haven't really had any success with any combination. [0] https://forum.mikrotik.com/viewtopic.php?t=182691 https://forum.mikrotik.com/viewtopic.php?t=182691 [1] https://forum.mikrotik.com/viewtopic.php?t=181881 https://forum.mikrotik.com/viewtopic.php?t=181881 [2] https://www.reddit.com/r/mikrotik/comments/rq7ytu/rb5009_25gbe_slower_than_1gbe/ https://www.reddit.com/r/mikrotik/comments/rq7ytu/rb5009_25g... [3] https://www.reddit.com/r/mikrotik/comments/rza5u3/slow_multigigi_performance_on_rb5009/ https://www.reddit.com/r/mikrotik/comments/rza5u3/slow_multi...

csunbird 4y ago
I wonder, what is stopping people from buying a x64 PC with a good CPU, slapping multiple ethernet cards over pcie (which support amazing speeds, good enough for 10G) and installing openwrt/pfsense on it? This is something I really want to do, actually.
- arghwhat 4y ago
  Cost, both from power consumption and purchase. Hard to compete in those fronts with a mikrotik or ubiquity router doing 1Gb/s through hardware offload with a tiny MIPS or ARM chip.
- zamadatix 4y ago
  Somewhere around HN there is a guy that did just this for the 25G internet to his home, full talk with slide deck and all. I think he wrote some custom Go code to simplify running it as well. In general though unless you need >10G you'll come behind on (good) COTS offerings in price and performance. Particularly if you need features like NAT or firewalling where software starts adding latency or performance cliffs at certain intervals while things like a low end Fortigate have high levels of hardware offload.
- toast0 4y ago
  One thing I've seen some people run into is if their ISP insists on PPPoE, inbound packets will tend to stick to a single RX queue, which lends to single threaded handling of that traffic, and it can too much for one core, especially if you get a lower power cpu. It might be possible to convince the network card to look a bit farther into the packet to hash packets to different queues, but at least it doesn't happen out of the box.
mmastrac 4y ago
I suspect as these ports get faster, flow control is going to be more important. I had a network issue on my local homelab that required enabling flow control to work around. This was particularly annoying on the UniFi interface as I had to drop back to a legacy interface to even find it. We're probably going to be in for a bit of pain until everything can reliably function at 10Gb/s
InTheArena 4y ago
I have a 2.5GBP/s WAN (as well as a second 1GB/s WAN) going into UDM-Pro, with a trunk link going at 10GB/second to a main switch via DAC, and a second 10GB/s going to a 10GB/s switch in my office. That just works (at least since Unifi solidified the firmware for dealing with funky fiber PPPoE situations). Transceivers matter. This is the one I used: https://www.amazon.com/gp/product/B06XQBFHNL/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&psc=1 https://www.amazon.com/gp/product/B06XQBFHNL/ref=ppx_yo_dt_b...
- mmastrac 4y ago
  I'm still struggling to get my Ubiquiti devices to negotiate 10Gbps between themselves, hah. I suspect my EdgeSwitch is just too old to reliably hit that mark.
Melatonic 4y ago
They should always auto negotiate down to whatever the slowest direct link in the chain is. So like if you connect a 2.5Gbs port directly to a 1Gbps switch then the driver should negotiate it down to 1Gbs.
- herpderperator 4y ago
  Ethernet auto negotiation does not look past one single endpoint. It can only negotiate with the immediate peer.
  - lillecarl 4y ago
    There's nothing preventing a software implementation to change this. I don't however think it's a good idea.
zamadatix 4y ago
It's a MikroTik issue in the end, their buffers are too simply too small for the interface speeds they support. It's part of being budget chips. Flow control is a poor solution as it blocks all flows whenever one flow makes things busy, best is just a properly scaled chip. I swapped a more expensive "low end" (for the family) Broadcom Trident 3 based switch in in place of 2 different 10G MikroTik boxes and the mixed speed transfers are now behaving exactly like one would expect without any flow control.
- herpderperator 4y ago
  Yeah, I did get the feeling that Flow Control (while it may alleviate some issues and/or fix others entirely) is just a hack. If what you're saying is true regarding the buffer sizes, I don't understand why they would knowingly make that choice as it causes poor performance which is not what their devices should be known for. It also completely negates the whole point of one 2.5Gbps port in the first place! I'm curious though: Going by your statement, wouldn't even a large buffer fill eventually? And by eventually, in computer terms, that might be milliseconds/a few seconds if we're talking transfer speeds hundreds of megabytes or gigabytes per second. How big is a buffer supposed to be exactly?
  - xen2xen1 4y ago
    I've replaced several MikroTik 's with something more expensive several times due to just being too slow under load , it's just how they are. You need to spend real $ to get real speed, even for gigabit connections. I know people to use them extensively who say the same.
  - zamadatix 4y ago
    Packet buffers get a bit complicated but the hand wavy explanation is you want enough buffer to be able to smooth over situations like mixed speed goes-intos and goes-outtas or jitter from small bursty streams but not too much buffer you end up delaying bandwidth/loss discovery algorithms or holding onto packets already considered stale more than you are actually helping. In the end for local switching you generally want a very small buffer but not necessarily just the smallest buffer you can find. There are endless rabbit holes of details though. You want a buffer with at least 1 hardware queue per network queue (in advanced situations you may even want a hardware queue set per flow) and you want these queues to be intelligent - priority queues, weighted round robin queues, best effort queues. In each of these queues you also probably want things like WRED which counterintuitively helps avoid congestion by starting to drop packets _before_ the buffer is full. You also want the buffer to be as "flat" among interfaces as possible, e.g. if the switch has 12 interfaces it's best it's all 1 line rate ASIC with 1 shared buffer not an 8 port + 4 port with interconnect and it's own set of internal QoS. For non local things latency also starts to become as big a factor in buffer sizes as interface speeds. I forget what I ended up finding the buffer size of my MikroTik switch to be but I remember thinking it should have been 4x-8x the size - whatever it was it was truly tiny and didn't seem to be operating well. As for why MikroTik makes things with known too small buffers... it's cheaper than proper high speed interfaces but more performant than just using low speed interfaces. Sure, it's not a perfect 2.5 Gbps solid but for the price it'd take for them to do that "right" they would be able to do 10 Gbps ports "wrong" instead. I still use my MikroTiks though I just make sure not to mix interface speeds or if I do it's with something on the side where I don't really care about having perfect performance to it just a generally fast connection when I do use it.
toast0 4y ago
You don't really want flow control here. Flow control on the 1G interface isn't needed, your PC can manage line rate packets (I assume); asking the WAN to stop when the 1G is full will cause trouble for flows to other ports. What you want is a small amount of buffer, but when the WAN is sending packets too fast, they should be dropped and TCP will figure it out. But maybe the buffers are too big and you get latency spikes. Or maybe something else is funky when packets are getting dropped? Flow control usually causes more issues as things will get delayed or dropped in bunches.
geerlingguy 4y ago
I've had this issue internally too, and found that only Mikrotik's own S+RJ10 transceivers handle the 2.5/5 Gbps speeds gracefully on their switches. Others sometimes negotiate at 10 Gbps and you get like 1.2 Gbps maximum over the 2.5G interfaces. It's kind of infuriating, to the point I bought a separate dedicated 2.5G switch with 10G uplinks and plugged that over to the Mikrotik with a DAC.
dave_taht 4y ago
Mikrotik RouterOS7 supports fq_codel running native. Does that not work on 2.5Gbit -> 1Gbit?

csunbird 4y ago

I wonder, what is stopping people from buying a x64 PC with a good CPU, slapping multiple ethernet cards over pcie (which support amazing speeds, good enough for 10G) and installing openwrt/pfsense on it? This is something I really want to do, actually.

arghwhat 4y ago
Cost, both from power consumption and purchase. Hard to compete in those fronts with a mikrotik or ubiquity router doing 1Gb/s through hardware offload with a tiny MIPS or ARM chip.
zamadatix 4y ago
Somewhere around HN there is a guy that did just this for the 25G internet to his home, full talk with slide deck and all. I think he wrote some custom Go code to simplify running it as well. In general though unless you need >10G you'll come behind on (good) COTS offerings in price and performance. Particularly if you need features like NAT or firewalling where software starts adding latency or performance cliffs at certain intervals while things like a low end Fortigate have high levels of hardware offload.
toast0 4y ago
One thing I've seen some people run into is if their ISP insists on PPPoE, inbound packets will tend to stick to a single RX queue, which lends to single threaded handling of that traffic, and it can too much for one core, especially if you get a lower power cpu. It might be possible to convince the network card to look a bit farther into the packet to hash packets to different queues, but at least it doesn't happen out of the box.

mmastrac 4y ago

I suspect as these ports get faster, flow control is going to be more important. I had a network issue on my local homelab that required enabling flow control to work around. This was particularly annoying on the UniFi interface as I had to drop back to a legacy interface to even find it. We're probably going to be in for a bit of pain until everything can reliably function at 10Gb/s

InTheArena 4y ago

I have a 2.5GBP/s WAN (as well as a second 1GB/s WAN) going into UDM-Pro, with a trunk link going at 10GB/second to a main switch via DAC, and a second 10GB/s going to a 10GB/s switch in my office. That just works (at least since Unifi solidified the firmware for dealing with funky fiber PPPoE situations). Transceivers matter. This is the one I used: https://www.amazon.com/gp/product/B06XQBFHNL/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&psc=1 https://www.amazon.com/gp/product/B06XQBFHNL/ref=ppx_yo_dt_b...

mmastrac 4y ago
I'm still struggling to get my Ubiquiti devices to negotiate 10Gbps between themselves, hah. I suspect my EdgeSwitch is just too old to reliably hit that mark.

Melatonic 4y ago

They should always auto negotiate down to whatever the slowest direct link in the chain is. So like if you connect a 2.5Gbs port directly to a 1Gbps switch then the driver should negotiate it down to 1Gbs.

herpderperator 4y ago
Ethernet auto negotiation does not look past one single endpoint. It can only negotiate with the immediate peer.
- lillecarl 4y ago
  There's nothing preventing a software implementation to change this. I don't however think it's a good idea.

zamadatix 4y ago

It's a MikroTik issue in the end, their buffers are too simply too small for the interface speeds they support. It's part of being budget chips. Flow control is a poor solution as it blocks all flows whenever one flow makes things busy, best is just a properly scaled chip. I swapped a more expensive "low end" (for the family) Broadcom Trident 3 based switch in in place of 2 different 10G MikroTik boxes and the mixed speed transfers are now behaving exactly like one would expect without any flow control.

herpderperator 4y ago
Yeah, I did get the feeling that Flow Control (while it may alleviate some issues and/or fix others entirely) is just a hack. If what you're saying is true regarding the buffer sizes, I don't understand why they would knowingly make that choice as it causes poor performance which is not what their devices should be known for. It also completely negates the whole point of one 2.5Gbps port in the first place! I'm curious though: Going by your statement, wouldn't even a large buffer fill eventually? And by eventually, in computer terms, that might be milliseconds/a few seconds if we're talking transfer speeds hundreds of megabytes or gigabytes per second. How big is a buffer supposed to be exactly?
- xen2xen1 4y ago
  I've replaced several MikroTik 's with something more expensive several times due to just being too slow under load , it's just how they are. You need to spend real $ to get real speed, even for gigabit connections. I know people to use them extensively who say the same.
- zamadatix 4y ago
  Packet buffers get a bit complicated but the hand wavy explanation is you want enough buffer to be able to smooth over situations like mixed speed goes-intos and goes-outtas or jitter from small bursty streams but not too much buffer you end up delaying bandwidth/loss discovery algorithms or holding onto packets already considered stale more than you are actually helping. In the end for local switching you generally want a very small buffer but not necessarily just the smallest buffer you can find. There are endless rabbit holes of details though. You want a buffer with at least 1 hardware queue per network queue (in advanced situations you may even want a hardware queue set per flow) and you want these queues to be intelligent - priority queues, weighted round robin queues, best effort queues. In each of these queues you also probably want things like WRED which counterintuitively helps avoid congestion by starting to drop packets _before_ the buffer is full. You also want the buffer to be as "flat" among interfaces as possible, e.g. if the switch has 12 interfaces it's best it's all 1 line rate ASIC with 1 shared buffer not an 8 port + 4 port with interconnect and it's own set of internal QoS. For non local things latency also starts to become as big a factor in buffer sizes as interface speeds. I forget what I ended up finding the buffer size of my MikroTik switch to be but I remember thinking it should have been 4x-8x the size - whatever it was it was truly tiny and didn't seem to be operating well. As for why MikroTik makes things with known too small buffers... it's cheaper than proper high speed interfaces but more performant than just using low speed interfaces. Sure, it's not a perfect 2.5 Gbps solid but for the price it'd take for them to do that "right" they would be able to do 10 Gbps ports "wrong" instead. I still use my MikroTiks though I just make sure not to mix interface speeds or if I do it's with something on the side where I don't really care about having perfect performance to it just a generally fast connection when I do use it.

toast0 4y ago

You don't really want flow control here. Flow control on the 1G interface isn't needed, your PC can manage line rate packets (I assume); asking the WAN to stop when the 1G is full will cause trouble for flows to other ports. What you want is a small amount of buffer, but when the WAN is sending packets too fast, they should be dropped and TCP will figure it out. But maybe the buffers are too big and you get latency spikes. Or maybe something else is funky when packets are getting dropped? Flow control usually causes more issues as things will get delayed or dropped in bunches.

geerlingguy 4y ago

I've had this issue internally too, and found that only Mikrotik's own S+RJ10 transceivers handle the 2.5/5 Gbps speeds gracefully on their switches. Others sometimes negotiate at 10 Gbps and you get like 1.2 Gbps maximum over the 2.5G interfaces. It's kind of infuriating, to the point I bought a separate dedicated 2.5G switch with 10G uplinks and plugged that over to the Mikrotik with a DAC.

dave_taht 4y ago

Mikrotik RouterOS7 supports fq_codel running native. Does that not work on 2.5Gbit -> 1Gbit?