4 ms·

Yeah, I did get the feeling that Flow Control (while it may alleviate some issues and/or fix others entirely) is just a hack. If what you're saying is true rega

Yeah, I did get the feeling that Flow Control (while it may alleviate some issues and/or fix others entirely) is just a hack. If what you're saying is true regarding the buffer sizes, I don't understand why they would knowingly make that choice as it causes poor performance which is not what their devices should be known for. It also completely negates the whole point of one 2.5Gbps port in the first place! I'm curious though: Going by your statement, wouldn't even a large buffer fill eventually? And by eventually, in computer terms, that might be milliseconds/a few seconds if we're talking transfer speeds hundreds of megabytes or gigabytes per second. How big is a buffer supposed to be exactly?

xen2xen1 4y ago
I've replaced several MikroTik 's with something more expensive several times due to just being too slow under load , it's just how they are. You need to spend real $ to get real speed, even for gigabit connections. I know people to use them extensively who say the same.
zamadatix 4y ago
Packet buffers get a bit complicated but the hand wavy explanation is you want enough buffer to be able to smooth over situations like mixed speed goes-intos and goes-outtas or jitter from small bursty streams but not too much buffer you end up delaying bandwidth/loss discovery algorithms or holding onto packets already considered stale more than you are actually helping. In the end for local switching you generally want a very small buffer but not necessarily just the smallest buffer you can find. There are endless rabbit holes of details though. You want a buffer with at least 1 hardware queue per network queue (in advanced situations you may even want a hardware queue set per flow) and you want these queues to be intelligent - priority queues, weighted round robin queues, best effort queues. In each of these queues you also probably want things like WRED which counterintuitively helps avoid congestion by starting to drop packets _before_ the buffer is full. You also want the buffer to be as "flat" among interfaces as possible, e.g. if the switch has 12 interfaces it's best it's all 1 line rate ASIC with 1 shared buffer not an 8 port + 4 port with interconnect and it's own set of internal QoS. For non local things latency also starts to become as big a factor in buffer sizes as interface speeds. I forget what I ended up finding the buffer size of my MikroTik switch to be but I remember thinking it should have been 4x-8x the size - whatever it was it was truly tiny and didn't seem to be operating well. As for why MikroTik makes things with known too small buffers... it's cheaper than proper high speed interfaces but more performant than just using low speed interfaces. Sure, it's not a perfect 2.5 Gbps solid but for the price it'd take for them to do that "right" they would be able to do 10 Gbps ports "wrong" instead. I still use my MikroTiks though I just make sure not to mix interface speeds or if I do it's with something on the side where I don't really care about having perfect performance to it just a generally fast connection when I do use it.
- bitbckt 4y ago
  "goes-intos and goes-outtas" - love it. Buffer counts and buffer sizing is also a notoriously cut-throat game in vendor marketing, almost on par with (mis-)use of the term "wire speed". This is not an easy/cheap problem to solve, and I fully agree with the conclusion: Mikrotik hardware is good up to the point where the price/performance curve falls the wrong way in a particular situation. I also get a lot of use from Mikrotik hardware, particularly the passively-cooled models where performance isn't as important as price + silence.
- greyhair 4y ago
  I had to fight this battle in 1998 with a DSL SoHo modem/NAT router (single box) with a 10Mb/S Ethernet jack routing through an ADSL modem with a maximum speed of 8Mb/S Down and 1Mb/S Up. The 'Up' path was killer with the PC trying to stream 10Mb/S Ethernet into a NAT router that could (at best) dump 1Mb/S up through DSL. So a 10/1 ratio. And yes, make your buffers too big (we had relatively tons of memory) and all you get is weird latency cycles as the higher level protocols over-send, then fall back. Make your buffers to small, and the total throughput falls off, bu the latency is far more 'normal'. We ended up buffer scaling based on what the upstream connection really achieved, so larger buffers for a 1Mb/S link than for a 128Kb/S link. We would start the router at 1Mb/S buffer sizing, and scale down if the DSL link throughput was less than that. At the time we were working through that, a number of people would scream 'just give it more buffer', not understanding how badly that broke something like TCP.

xen2xen1 4y ago

I've replaced several MikroTik 's with something more expensive several times due to just being too slow under load , it's just how they are. You need to spend real $ to get real speed, even for gigabit connections. I know people to use them extensively who say the same.

zamadatix 4y ago

Packet buffers get a bit complicated but the hand wavy explanation is you want enough buffer to be able to smooth over situations like mixed speed goes-intos and goes-outtas or jitter from small bursty streams but not too much buffer you end up delaying bandwidth/loss discovery algorithms or holding onto packets already considered stale more than you are actually helping. In the end for local switching you generally want a very small buffer but not necessarily just the smallest buffer you can find. There are endless rabbit holes of details though. You want a buffer with at least 1 hardware queue per network queue (in advanced situations you may even want a hardware queue set per flow) and you want these queues to be intelligent - priority queues, weighted round robin queues, best effort queues. In each of these queues you also probably want things like WRED which counterintuitively helps avoid congestion by starting to drop packets _before_ the buffer is full. You also want the buffer to be as "flat" among interfaces as possible, e.g. if the switch has 12 interfaces it's best it's all 1 line rate ASIC with 1 shared buffer not an 8 port + 4 port with interconnect and it's own set of internal QoS. For non local things latency also starts to become as big a factor in buffer sizes as interface speeds. I forget what I ended up finding the buffer size of my MikroTik switch to be but I remember thinking it should have been 4x-8x the size - whatever it was it was truly tiny and didn't seem to be operating well. As for why MikroTik makes things with known too small buffers... it's cheaper than proper high speed interfaces but more performant than just using low speed interfaces. Sure, it's not a perfect 2.5 Gbps solid but for the price it'd take for them to do that "right" they would be able to do 10 Gbps ports "wrong" instead. I still use my MikroTiks though I just make sure not to mix interface speeds or if I do it's with something on the side where I don't really care about having perfect performance to it just a generally fast connection when I do use it.

bitbckt 4y ago
"goes-intos and goes-outtas" - love it. Buffer counts and buffer sizing is also a notoriously cut-throat game in vendor marketing, almost on par with (mis-)use of the term "wire speed". This is not an easy/cheap problem to solve, and I fully agree with the conclusion: Mikrotik hardware is good up to the point where the price/performance curve falls the wrong way in a particular situation. I also get a lot of use from Mikrotik hardware, particularly the passively-cooled models where performance isn't as important as price + silence.
greyhair 4y ago
I had to fight this battle in 1998 with a DSL SoHo modem/NAT router (single box) with a 10Mb/S Ethernet jack routing through an ADSL modem with a maximum speed of 8Mb/S Down and 1Mb/S Up. The 'Up' path was killer with the PC trying to stream 10Mb/S Ethernet into a NAT router that could (at best) dump 1Mb/S up through DSL. So a 10/1 ratio. And yes, make your buffers too big (we had relatively tons of memory) and all you get is weird latency cycles as the higher level protocols over-send, then fall back. Make your buffers to small, and the total throughput falls off, bu the latency is far more 'normal'. We ended up buffer scaling based on what the upstream connection really achieved, so larger buffers for a 1Mb/S link than for a 128Kb/S link. We would start the router at 1Mb/S buffer sizing, and scale down if the DSL link throughput was less than that. At the time we were working through that, a number of people would scream 'just give it more buffer', not understanding how badly that broke something like TCP.