5 ms·

Audio/video streams are a completely different problem - lost data is just lost. This article is discussing a very specific problem where no data loss can be to

Audio/video streams are a completely different problem - lost data is just lost. This article is discussing a very specific problem where no data loss can be tolerated. The solution described only works when very high bandwidth is available, because it aggressively sends duplicate copies of data until acked, using on average 2-15x more bandwidth and 120x more bandwidth in the worst case (numbers taken directly from the article). If you happened to have that kind of spare bandwidth available then yes, this is an excellent solution. When do you have high latency, packet loss, and bandwidth overprovisioned by a factor of 15?

ANTSANTS 12y ago
Deterministic lockstep is primarily used in RTS games, where the only data that is transmitted per tick are tiny commands like "player 1 is constructing a war factory at this location" or "player 1 told unit 6 to move to this tile." If we're ridiculously generous and say that a kilobyte of commands are sent per tick, then 120x bandwidth would be 120 kb/tick. 10 ticks/s is a common RTS tickrate, so that's about a megabyte per second. In practice, far, far less bandwidth would be used than this, given that this architecture scales down to 90's dialup connections (https://www.codeofhonor.com/blog/the-making-of-warcraft-part-3 https://www.codeofhonor.com/blog/the-making-of-warcraft-part...), but even a megabyte/s is not a completely ridiculous amount of upload bandwidth for much of the first world (but sadly not the USA). > Audio/video streams are a completely different problem - lost data is just lost. This article is discussing a very specific problem where no data loss can be tolerated. I know that, which is why I separated deterministic lockstep from other solutions in my comment. You're missing my point, which is that soft-realtime problems aren't suited for TCP, and cannot be addressed by any single protocol because constraints differ.
- asuffield 12y ago
  You appear to be missing my point, which is that this proposed protocol responds to network congestion by increasing the amount of bandwidth that it uses by a factor of more than 10. Consider what happens when many clients are using this protocol over a shared transit link and it approaches 80% utilisation, so starts to drop packets at a low rate: all the clients will start batching up larger amounts of history and sending them, which increases the utilisation, which increases the packet loss, which increases the amount of history each client sends... it's a destructive feedback loop that launches a DoS attack against the congested link. The absolute size of each individual client stream is unimportant, as we are merely assuming that the number of clients sharing a link is high enough to congest it. The last-mile user connection is not the worst problem here.
  - ANTSANTS 12y ago
    Network congestion is not the only reason for packets to be dropped. Wifi and mobile connections can be hampered by radio interference or have trouble transmitting through thick walls, hardware can be faulty, wires and fiber can be damaged, routers (especially home routers) can be poorly configured, cosmic rays can flip bits, etc. The internet is not perfect, and network programmers in a certain soft-realtime, low-bandwidth domain have noted that sending duplicates of their tiny packets instead of waiting around for ACKs results in an improvement in reliability and average latency at negligible cost. You're trying to extrapolate this idea way too far. No one is saying we should swap TCP out tomorrow with an alternative that works that way.
  - jtown_ 12y ago
    This isn't a theoretical, unproven suggestion he's recklessly advocating -- this technique is implemented in several games played by millions without your DoS scenario playing out. Games are typically low bandwidth applications -- they use that extra wiggle room to increase reliability and responsiveness with reliable real world success. He's also written about flow control to properly handle congestion when it does happen: http://gafferongames.com/networking-for-game-programmers/reliability-and-flow-control/ http://gafferongames.com/networking-for-game-programmers/rel...
    - infinite8s 12y ago
      I wonder if something like Aeron (https://github.com/real-logic/Aeron https://github.com/real-logic/Aeron) would work.
    - gafferongames 12y ago
      To be clear, not all games are networked using the deterministic lockstep technique but most realtime games (FPS and so on) do use UDP to send time critical data, and implement some aspects of reliability, ordering and congestion avoidance within their own custom protocol.
      - jtown_ 12y ago
NickPollard 12y ago
The point is though that sending 15x more data is not an issue when you're only sending a few bytes per tick. When optimizing, you always profile and optimize the bottleneck first. For a lot of people in a lot of situations, latency, not bandwidth, is the bottleneck, and that's exactly what this solution solves. Compared to e.g. Standard web usage, most games are actually very small amounts of data sent, but very high time criticality, and hence very sensitive to latency.
bartwe 12y ago
In the case of wifi and such this seems like a sensible approach, too bad its being applied end to end.
- gafferongames 12y ago
  All that would be required IMO for this to be a perfect solution for time critical data would be if we had an IP packet header flag to indicate to routers that our packet is time critical, so if it can't be passed on immediately, don't buffer it like a TCP packet, just drop it. -- cheers

ANTSANTS 12y ago

Deterministic lockstep is primarily used in RTS games, where the only data that is transmitted per tick are tiny commands like "player 1 is constructing a war factory at this location" or "player 1 told unit 6 to move to this tile." If we're ridiculously generous and say that a kilobyte of commands are sent per tick, then 120x bandwidth would be 120 kb/tick. 10 ticks/s is a common RTS tickrate, so that's about a megabyte per second. In practice, far, far less bandwidth would be used than this, given that this architecture scales down to 90's dialup connections (https://www.codeofhonor.com/blog/the-making-of-warcraft-part-3 https://www.codeofhonor.com/blog/the-making-of-warcraft-part...), but even a megabyte/s is not a completely ridiculous amount of upload bandwidth for much of the first world (but sadly not the USA). > Audio/video streams are a completely different problem - lost data is just lost. This article is discussing a very specific problem where no data loss can be tolerated. I know that, which is why I separated deterministic lockstep from other solutions in my comment. You're missing my point, which is that soft-realtime problems aren't suited for TCP, and cannot be addressed by any single protocol because constraints differ.

asuffield 12y ago
You appear to be missing my point, which is that this proposed protocol responds to network congestion by increasing the amount of bandwidth that it uses by a factor of more than 10. Consider what happens when many clients are using this protocol over a shared transit link and it approaches 80% utilisation, so starts to drop packets at a low rate: all the clients will start batching up larger amounts of history and sending them, which increases the utilisation, which increases the packet loss, which increases the amount of history each client sends... it's a destructive feedback loop that launches a DoS attack against the congested link. The absolute size of each individual client stream is unimportant, as we are merely assuming that the number of clients sharing a link is high enough to congest it. The last-mile user connection is not the worst problem here.
- ANTSANTS 12y ago
  Network congestion is not the only reason for packets to be dropped. Wifi and mobile connections can be hampered by radio interference or have trouble transmitting through thick walls, hardware can be faulty, wires and fiber can be damaged, routers (especially home routers) can be poorly configured, cosmic rays can flip bits, etc. The internet is not perfect, and network programmers in a certain soft-realtime, low-bandwidth domain have noted that sending duplicates of their tiny packets instead of waiting around for ACKs results in an improvement in reliability and average latency at negligible cost. You're trying to extrapolate this idea way too far. No one is saying we should swap TCP out tomorrow with an alternative that works that way.
- jtown_ 12y ago
  This isn't a theoretical, unproven suggestion he's recklessly advocating -- this technique is implemented in several games played by millions without your DoS scenario playing out. Games are typically low bandwidth applications -- they use that extra wiggle room to increase reliability and responsiveness with reliable real world success. He's also written about flow control to properly handle congestion when it does happen: http://gafferongames.com/networking-for-game-programmers/reliability-and-flow-control/ http://gafferongames.com/networking-for-game-programmers/rel...
  - infinite8s 12y ago
    I wonder if something like Aeron (https://github.com/real-logic/Aeron https://github.com/real-logic/Aeron) would work.
  - gafferongames 12y ago
    To be clear, not all games are networked using the deterministic lockstep technique but most realtime games (FPS and so on) do use UDP to send time critical data, and implement some aspects of reliability, ordering and congestion avoidance within their own custom protocol.
    - jtown_ 12y ago

NickPollard 12y ago

The point is though that sending 15x more data is not an issue when you're only sending a few bytes per tick. When optimizing, you always profile and optimize the bottleneck first. For a lot of people in a lot of situations, latency, not bandwidth, is the bottleneck, and that's exactly what this solution solves. Compared to e.g. Standard web usage, most games are actually very small amounts of data sent, but very high time criticality, and hence very sensitive to latency.

bartwe 12y ago

In the case of wifi and such this seems like a sensible approach, too bad its being applied end to end.

gafferongames 12y ago
All that would be required IMO for this to be a perfect solution for time critical data would be if we had an IP packet header flag to indicate to routers that our packet is time critical, so if it can't be passed on immediately, don't buffer it like a TCP packet, just drop it. -- cheers