5 ms·

I'd like this to go one layer deeper... Simulate a network with a variable latency, and see how typical TCP implementations behave over it. Far too many rea

I'd like this to go one layer deeper... Simulate a network with a variable latency, and see how typical TCP implementations behave over it. Far too many realtime things, like games and videoconferencing (both of which frequently use TCP, despite it being badly suited to the application), perform really badly when the bandwidth increases and decreases, for example as I walk around a building with wifi and pass concrete pillars. I want not a single dropped frame in those circumstances. Sure - you can have some lower res frames, but I don't expect to see 30 frames dropped in a row and a big glitch.

clbrmbr 3y ago
I wrote such a tool[1] for my own use some years ago. It uses libpcap to grab all the packets on one interface and then re-emit them from another interface after some delay, with optional packet loss. If there’s real interest I could dust it off… maybe rewrite in Rust. [1] https://github.com/chrismerck/lagsim https://github.com/chrismerck/lagsim
ajsnigrutin 3y ago
You can do this with a linux router and 'tc' https://www.pico.net/kb/how-can-i-simulate-delayed-and-dropped-packets-in-linux/ https://www.pico.net/kb/how-can-i-simulate-delayed-and-dropp... (just a short example)
- laserbeam 3y ago
  Can't you just do it locally with tc without a router? I have artificially throttled / simulated delays with tc before and I don't see why you need a router. PS. My knowledge of tc is very rudimentary. I only had to use it once.
  - jrockway 3y ago
    You don't need a router. I've used it on `lo` before.
ysleepy 3y ago
IIRC FreeBSD had dummynet which allows creating slow, laggy and lossy virtual networks.
derhuerst 3y ago
AFAIK mininet [0] can help you with this. It allows configuring latency and/or jitter dynamically using Python [1]. [0] https://mininet.org https://mininet.org [1] https://mininet.org/api/classmininet_1_1link_1_1TCIntf.html#a3854309a926ae4bc40f4fc15895838fa https://mininet.org/api/classmininet_1_1link_1_1TCIntf.html#...
ahoka 3y ago
What do you think TCP should do when bandwidth lowers?
oneplane 3y ago
GNS3 can do that
jsnell 3y ago
If you're specifically looking to probe the behavior of different TCP implementations, I think you want something slightly different than this. First, you want the faults to be deterministic (e.g. not random 1% packet loss but packet loss at a certain point in the connection); and you want the limits to be per TCP connection, not per link. If you don't have that determinism or connection isolation there are just going to be too many confounding variables in addition to the TCP stacks you're supposed to be testing. I wrote a tool like that a few years back[0], but didn't publish most of what I discovered about e.g. how different CDNs tuned their TCP stacks (just a couple of anonymized examples[1]). [0] https://github.com/teclo/flow-disruptor https://github.com/teclo/flow-disruptor [1] https://www.snellman.net/blog/archive/2015-10-01-flow-disruptor/ https://www.snellman.net/blog/archive/2015-10-01-flow-disrup...
networkchad 3y ago
[dead]
lelanthran 3y ago
> I want not a single dropped frame in those circumstances. Sure - you can have some lower res frames, but I don't expect to see 30 frames dropped in a row and a big glitch. This is quite difficult, I think. You only know how bad the connection is after dropping a frame or two, and the intervening network will detect it before the endpoints do. From an application level, I suspect that on the socket layer level you can't do anything about TCP anyway, the underlying layer will retransmit after a delay. [EDIT: Just realised you meant video frames, and not network frames. I think my comment still applies though - dropping a single transmission unit that is part of an I video frame still means discarding that entire frame, whilst dropping a B or P frame is going to result in glitches.]
- londons_explore 3y ago
  With closer integration between the application and the network layers, you can see things like the RSSI dropping (basically signal strength). This can trigger the application to start sending less data before any actual data loss happens - and the application at the local end can respond in under a millisecond rather than a whole network roundtrip. For the traffic from the remote end, data packets can consist of "here is the low res data" and another packet for "and here is the extra data to turn the low res into higher res". The high res would be sent with QoS markers so that network hardware knows that if anything is to be dropped, drop that first. The different QoS streams could even be sent with different transmit powers and modulation schemes on the WiFi network.
  - withinboredom 3y ago
    RSSI goes up and down all the time for a variety of reasons. That's not really an indication of network speed though, only link speed. You can have a wifi connection over several km's if nobody else has wifi routers (in real life, they do, so you can't go much further than your house) and still have a really good link speed (from experience). Network speed is a function of link speed AND the success of packets to reach you. When you are on wifi, your device gets a very tiny slice of time to send packets. If you have more devices, your device gets less time to send packets. Further, a "loud router" nearby (or self-interference ... or radar in 5G space -- or a microwave, or Bluetooth in 2G space) can cause a frame to be interfered with and never be received. Thus it has to be retransmitted. RSSI is only a very small part of your overall network conditions and doesn't really tell anyone anything.
jrockway 3y ago
This little tool is showing its age: https://gfblip.appspot.com/ https://gfblip.appspot.com/ but as you walk around you'll probably notice that you aren't getting any packets through to the Internet at all. That's why your video drops out, you aren't connected to the Internet anymore. What Zoom does is notice this and play the video frames in its buffer back more slowly in an attempt to smooth over the "no data coming from the Internet" period. I don't find this that helpful because if you comment on something someone just said, they've moved on to a new topic by the time Zoom unbuffers and displays all the frames. Clever hack, I prefer the complete dropout.

clbrmbr 3y ago

I wrote such a tool[1] for my own use some years ago. It uses libpcap to grab all the packets on one interface and then re-emit them from another interface after some delay, with optional packet loss. If there’s real interest I could dust it off… maybe rewrite in Rust. [1] https://github.com/chrismerck/lagsim https://github.com/chrismerck/lagsim

ajsnigrutin 3y ago

You can do this with a linux router and 'tc' https://www.pico.net/kb/how-can-i-simulate-delayed-and-dropped-packets-in-linux/ https://www.pico.net/kb/how-can-i-simulate-delayed-and-dropp... (just a short example)

laserbeam 3y ago
Can't you just do it locally with tc without a router? I have artificially throttled / simulated delays with tc before and I don't see why you need a router. PS. My knowledge of tc is very rudimentary. I only had to use it once.
- jrockway 3y ago
  You don't need a router. I've used it on `lo` before.

ysleepy 3y ago

IIRC FreeBSD had dummynet which allows creating slow, laggy and lossy virtual networks.

derhuerst 3y ago

AFAIK mininet [0] can help you with this. It allows configuring latency and/or jitter dynamically using Python [1]. [0] https://mininet.org https://mininet.org [1] https://mininet.org/api/classmininet_1_1link_1_1TCIntf.html#a3854309a926ae4bc40f4fc15895838fa https://mininet.org/api/classmininet_1_1link_1_1TCIntf.html#...

ahoka 3y ago

What do you think TCP should do when bandwidth lowers?

oneplane 3y ago

GNS3 can do that

jsnell 3y ago

If you're specifically looking to probe the behavior of different TCP implementations, I think you want something slightly different than this. First, you want the faults to be deterministic (e.g. not random 1% packet loss but packet loss at a certain point in the connection); and you want the limits to be per TCP connection, not per link. If you don't have that determinism or connection isolation there are just going to be too many confounding variables in addition to the TCP stacks you're supposed to be testing. I wrote a tool like that a few years back[0], but didn't publish most of what I discovered about e.g. how different CDNs tuned their TCP stacks (just a couple of anonymized examples[1]). [0] https://github.com/teclo/flow-disruptor https://github.com/teclo/flow-disruptor [1] https://www.snellman.net/blog/archive/2015-10-01-flow-disruptor/ https://www.snellman.net/blog/archive/2015-10-01-flow-disrup...

networkchad 3y ago

[dead]

lelanthran 3y ago

> I want not a single dropped frame in those circumstances. Sure - you can have some lower res frames, but I don't expect to see 30 frames dropped in a row and a big glitch. This is quite difficult, I think. You only know how bad the connection is after dropping a frame or two, and the intervening network will detect it before the endpoints do. From an application level, I suspect that on the socket layer level you can't do anything about TCP anyway, the underlying layer will retransmit after a delay. [EDIT: Just realised you meant video frames, and not network frames. I think my comment still applies though - dropping a single transmission unit that is part of an I video frame still means discarding that entire frame, whilst dropping a B or P frame is going to result in glitches.]

londons_explore 3y ago
With closer integration between the application and the network layers, you can see things like the RSSI dropping (basically signal strength). This can trigger the application to start sending less data before any actual data loss happens - and the application at the local end can respond in under a millisecond rather than a whole network roundtrip. For the traffic from the remote end, data packets can consist of "here is the low res data" and another packet for "and here is the extra data to turn the low res into higher res". The high res would be sent with QoS markers so that network hardware knows that if anything is to be dropped, drop that first. The different QoS streams could even be sent with different transmit powers and modulation schemes on the WiFi network.
- withinboredom 3y ago
  RSSI goes up and down all the time for a variety of reasons. That's not really an indication of network speed though, only link speed. You can have a wifi connection over several km's if nobody else has wifi routers (in real life, they do, so you can't go much further than your house) and still have a really good link speed (from experience). Network speed is a function of link speed AND the success of packets to reach you. When you are on wifi, your device gets a very tiny slice of time to send packets. If you have more devices, your device gets less time to send packets. Further, a "loud router" nearby (or self-interference ... or radar in 5G space -- or a microwave, or Bluetooth in 2G space) can cause a frame to be interfered with and never be received. Thus it has to be retransmitted. RSSI is only a very small part of your overall network conditions and doesn't really tell anyone anything.

jrockway 3y ago

This little tool is showing its age: https://gfblip.appspot.com/ https://gfblip.appspot.com/ but as you walk around you'll probably notice that you aren't getting any packets through to the Internet at all. That's why your video drops out, you aren't connected to the Internet anymore. What Zoom does is notice this and play the video frames in its buffer back more slowly in an attempt to smooth over the "no data coming from the Internet" period. I don't find this that helpful because if you comment on something someone just said, they've moved on to a new topic by the time Zoom unbuffers and displays all the frames. Clever hack, I prefer the complete dropout.