10 ms·

Fixing the Internet for Games

172 points by gafferongames 8y ago

tschwimmer 8y ago
This was an awesome write up. The most interesting thing for me was learning that bottlenecks are not on the edge, but across the backbone because T1 carriers are optimizing for low cost, not low latency.
- Jare 8y ago
  The Riot links are pretty great too, very much worth it.
- kingosticks 8y ago
  The thing I'm still a bit confused about is: is this a limitation with how their networks fundamentally work, or an artefact of my product choice when I buy a relatively cheap home Internet service? i.e. if I want a specialist low-latency and low-jitter link, could I not just pay for those benefits by purchasing a business-grade service? Or is there no distinction once the data gets onto a peered-network? Or is the idea here that I don't have to cough up for all the benefits of that superior product when I only want some of them and only some of the time? And this whole thing does rely on the bottleneck not being at then edge, which I don't see any proof of or reason for. EDIT: is there any proof/data to support this?
  - gafferongames 8y ago
    We are seeing players with Google Fiber get 100ms improvements. It’s not bad ISPs, it’s poor routing decisions and these are pretty much universal.
ndmrcf 8y ago
That reminds me of QuakeWorld's proxy servers.
mjevans 8y ago
Other things that can help: * A national policy that actually encourages competition > Also one that classifies 'the last mile' as public infrastructure to compete on top of. * Fiber to the home/MDU (Ethernet the rest of the way) * Where cable Internet must be used, the anti-buffer bloat parts of DOCSIS 3.1 (DOCSIS-PIE). * Actually having peer to peer focused ISPs, not content consumption ones. Upload should be no less than 25% of download; ideally symmetric. * Better visibility in to peering between users and actual effort applied to removing the bottlenecks.
- weberc2 8y ago
  > A national policy that actually encourages competition I'm all for improving the Internet and dunking on large ISPs, but what does this mean? Specifically what kind of national policy would foster competition (regulation tends to favor monopolies, right?)? Also, don't these private networks and marketplaces constitute competition? Or are you talking specifically about the network edge while the article is talking about the backbone? This article has me thinking that there is potentially a lot of economic efficiency to be unlocked by these sorts of marketplaces--basically routing based on economics, but maybe this is antithetical to net neutrality ("pay to play" internet routing)?
  - mjevans 8y ago
    Ideally competition in all areas, but different areas might require different policy and this is a question that should actually be studied by experts, or at least in academia summarizing the results of policy in different nations (as well as how closely they might match similar housing density for actually lived in areas). Some places that I think should be at least initially studied / considered for LAST MILE* competition include: * LAST MILE (connectivity) being the connection between citizen's houses and peering/point-of-presence locations: connecting citizens to a marketplace of different service providers. * Remove legal barriers that prevent municipalities and utility entities from providing last mile connectivity (often laws lobbied (paid) for by existing monopoly or near monopoly companies). * Things similar to "Local Loop Un-bundling" > Require the existing providers break out last mile connectivity from their back-haul package. > Require they sell access to their private PoP at a cost which they also pay at a similar rate (to the PoP on the balance sheet). * Things equivalent to 'right of way' grants (poles on public land, trenches, etc) must be owned by the citizens. > Also leased at FRAND rates to any entities > Serviceable with 'one touch' by any entity licensed with the owners (citizens via the City/County/State/etc that owns it) Possibly other things I can't think of in a short break.
VectorLock 8y ago
Clever concept. I wonder if they'll publish any more information on how their service works. Things like what this 'route shader' is that people can read without having to go through Email-Our-Sales-Team shit.
- politician 8y ago
  It's a custom bidding script that tells their real-time auction platform how much you want to pay for route improvements for a particular user session. Typically, shaders are used to move vertices in 3d meshes or color pixels in the GPU. Network Next is just using the term in a creative way to help connect to their audience, but it's nothing different than a bidding model.
  - gafferongames 8y ago
    Route shaders select the best route out of multiple ones available. Right now they could be used to for example, specify basic things like: acceptable latency (eg. I don’t want to pay to optimize a player’s session from 20ms down to 19ms, 20 is good enough), minimum improvement vs. public internet before a session takes network next (eg. 5ms improvement or greater), and of course maximum price for the bid. If we don’t find a suitable route at the price specified, the session takes the public internet and is free. Right now it’s fixed function like OpenGL 1.0 (eg. Parameters), but long term, it could even be LUA bytecode. Hence, route shader is not just a marketing term, it’s a real thing :)
    - kbenson 8y ago
      The really interesting thing I see for this is a possible other route for game revenue in the future. Instead of paying up front (traditional) pricing or advertising, or cosmetic (or not so cosmetic) in game purchases, you could have two tiers, free and optimized latency for a cost. Or some combination. Cosmetic in-game purchases aren't necessarily bad, but the amount my son has spent and tries to continue to spend (I strictly limit what of his money he can waste on it now) on fortnite stuff is ridiculous and concerning. A sane middle ground would be helpful.
NelsonMinar 8y ago
See also Haste, which has been doing this for a couple of years now: https://haste.net/ https://haste.net/ One thing that's interesting is how different these services need to be in different countries.
- grenoire 8y ago
  I think Network Next is developer-targeted, as opposed to Haste which offers a subscription service for gamers.
  - gafferongames 8y ago
    This is correct. We work with game developers to sponsor the best route for their players. If we can’t significantly beat the public internet for that player (according to what the game developer specifies in their route shader), then we take the public internet and that player is free.
AlimJaffer 8y ago
As someone who works in the games space, I find this to be an interesting proposition. My primary concern revolves around similar 'net neutrality' aspects - more and more of the internet is becoming pay to play, this just further entrenches that theory if you want the best networking for your games as a studio.
- politician 8y ago
  That's a good observation. Given the alternative between bidding for performance from Network Next vs paying some flat fee to Comcast, et al. directly, I'd go with NN and let them deal with Comcast. ... and then maybe add "faster gameplay" power ups to our loot boxes (kidding).
  - gafferongames 8y ago
    When we work with game developers, they find this aspect of the system incredibly compelling. Does it really make sense for every game company to negotiate with every transit provider and ISP to make sure traffic is delivered properly for their game? We think not. Plus, how could they get a fair price if they’re the small guy. Suppliers on Network Next cannot identify buyers and cannot have one price for one buyer and one for another. So now the small guys get the same price the big guys do and competition should drive the price down over time.
stcredzero 8y ago
The Internet is optimized for throughput at lowest cost. No amount of good netcode that you write can compensate for this. The problem is the internet itself. The internet doesn’t care about your game. The Internet did change, such that streaming movies became viable. Given that games are now bigger than movies, could it be argued that that the network lag we're seeing is the industry lagging behind the culture? (Which it generally does, at least a little.) A part of this might well involve special deals between large companies involving specialized infrastructure. There are such things now involved with movie streaming. Google's plans involve the infrastructure part of that already. What happens to net neutrality? Once game streaming becomes mature, then it's only a matter of time for App streaming to follow. We'll be back in the mainframe days, just with much larger companies with much larger multi-box cloud "mainframes", commanding the attention of a much larger portion of the population and the culture. "The computer is your friend. Trust the computer!" https://en.wikipedia.org/wiki/Paranoia_(role-playing_game) https://en.wikipedia.org/wiki/Paranoia_(role-playing_game)
- politician 8y ago
  > What happens to net neutrality? Counterintuitively, the market that Network Next is making shifts us closer to actual net neutrality than relying on the benevolence of FCC or Comcast by putting power into the hands of content providers where previously they were at the mercy of last-mile ISPs. If businesses can collectively bid for performance from private networks to offset actions by last-mile ISPs to cut costs and reduce service, then they also have the power to organize a general strike by refusing to bid for performance from traffic from specific ISPs until they work out better rates throughout the value chain. So I see this as a mechanism to restore the spirit of net neutrality despite the best wishes of the FCC.
  - mrosett 8y ago
    > If businesses can collectively bid for performance from private networks You realize this is exactly what net neutrality prohibits?
    - 8y ago
AntiRush 8y ago
Valve recently announced partner access to their network backbone which addresses a lot of the problems mentioned here: * Access to our network, giving your players protection from attack, 100% reliable NAT traversal, and improved connectivity. * Tools for instantly estimating the ping between two arbitrary hosts without sending any packets. * A high quality end-to-end encrypted reliable-over-UDP protocol. >> Second, clients can select a route that gets off of the public Internet and onto our dedicated links as early as possible. On our backbone we can ensure that the routing is optimal, since we have peered with over 2,500 ISPs. We also prioritize the latency-sensitive game traffic over HTTP content downloads, which we can afford to do because game traffic makes up a relatively small percentage of our overall bandwidth utilization. And on our backbone, a sudden surge of traffic unrelated to gaming won’t degrade the experience. https://steamcommunity.com/groups/steamworks#announcements/detail/1791775741704351698 https://steamcommunity.com/groups/steamworks#announcements/d... They started using it in some of their games a couple of years ago (Counterstrike Global Offensive and Dota2 if I recall correctly).
- Impossible 8y ago
  Which is great if Steam is your only platform and you want to be tied to Valve forever, but as soon as you want your game on mobile, console or even another PC store like Epic or GOG you need another solution.
  - soup10 8y ago
    i hate these underhanded "use this cool feature that we made to lock you in to our platform" tactics
    - bilbo0s 8y ago
      ??? Isn't the entire point of the original article to lock you into the Network Next platform by forcing you to implement their SDK? I think every network provider is going to work to lock you into their platform.
      - deleted 8y ago
the_mitsuhiko 8y ago
I'm just glad to see someone implements a real time bidding market without dropping the word blockchain. I really like the idea of it.
- gafferongames 8y ago
  I really hate blockchain :)
tinco 8y ago
This is epic. Great to see that someone who's been such a good support to the gaming community with his excellent learning resources is also being successful in his business. It's also interesting to see these "private internet" endeavours have popped up to solve centralization issues on the internet. Streaming sites like Youtube and Netflix have solved it in similar way, striking deals with ISPs, as have the big CDN's, simply putting their machines at the ISP's datacenters. Network.Next is solving this for the gaming domain, but there is a more general preferential routing on the internet problem this idea could be expanded to.
- gafferongames 8y ago
  Thank you very much.
  - jay_kyburz 8y ago
    How about some Tribes Jetpack action!
jauer 8y ago
This reads like a traffic targeting system ala FB's cartographer (c10r) bolted onto an ad exchange (continuous bidding etc.) Interesting to see if they can keep the 10-second update intervals while scaling to more users. Curious to see if the market actually wants to pay for marginal improvements in gaming latency. I would have expected something like this targeting VoIP to be more lucrative.
robterrell 8y ago
Glenn, if you're still hanging around -- can the shader be written to account for the flow of P2P network traffic? Let's say I'm using an old-school configuration of a centralized lobby server that simply connects players to each other and the players connect directly over UDP?
- gafferongames 8y ago
  We currently only support client/server topology with dedicated servers.
petermcneeley 8y ago
The primary issue with the internet networking is the lack of Quality of Service (QoS) controls at the application layer.
- gafferongames 8y ago
  This is true, but without a marketplace and somebody paying, how can we tell whether the QoS requests are legitimate? I mean, “yeah, my traffic is always real-time priority”. People just will cheat it... so Network Next is a way of doing this, while saying, OK, for the truly real-time traffic, applications are willing to sponsor it. Since it’s paid, it won’t be exploited.
mkj 8y ago
This could be useful for long-distance remote desktop and SSH too. Curious how to build on top of it for diverse apps though (use it as a VPN somehow?)
- gafferongames 8y ago
  Right now we are an SDK that embeds in the client, server and backend for applications. It is open source too (BSD license), but not yet publicly released. Will be released soon, I look forward to seeing what people do with it, outside of games.
tarikjn 8y ago
Great reading this, I just want to drop a line to say that I have been working to solve precisely this problem -- for gaming but also other applications -- at the root. I have been working on a paper on this, here is the pre-proposal I published a whiled ago: https://docs.google.com/document/d/1xWaSB-3VSOMaiyxfjDqTalDCein0V46EbVrO6mFGPQs/edit?usp=sharing https://docs.google.com/document/d/1xWaSB-3VSOMaiyxfjDqTalDC... -- a lot of progress has been made since and I am looking for funding/grant before moving to publishing the actual paper and moving to implementation. The root of the issue is that the IP protocol is incomplete, in that is relies on private agreement and ISP servicing that cannot react to a real-time, real-world demand. IPv4 had some attempts at public QoS features that were mostly dropped in IPv6 because people realized it didn't work in practice. The only way to fix this for real is to merge ISP servicing and peering into a per-packet peer-to-peer network market. This solves a great many issues at the same time and essentially achieves the promise of mesh networking and more. If you would like to help, please drop me a line: tarik@graphnet.org (sorry no website at the moment).
openloop 8y ago
Get rid of TCP/IP. I’m sure you’re right sort of.
openloop 8y ago
Non uniform Data Protocol.
Animats 8y ago
Games need some low latency data, but not much. Player actions, character movement, and maybe bullets need low latency. Asset downloading does not. If you had a dial up modem running over a non-packet phone network, and didn't overload the buffering, you'd have better latency than the Internet. Many games make this distinction already. The latency-critical stuff goes over UDP, and it's limited to data where a missing packet is superseded by the next packet. Nothing is retransmitted. Bulk data goes over TCP, with reliable retransmission but more delay. It's too bad ISDN died off. 64Kb/s end to end, not packetization delay, and no jitter. The ideal gamer network would be an ISDN connection and a IP connection in parallel. If we had QoS systems where 1% of your maximum data rate could be at high priority, this, and the VoIP problem, would be mostly solved.
- gsich 8y ago
  Mostly it's just UDP with some TCP like reliable transfer built on top of it.
  - gafferongames 8y ago
    This is correct. Most FPS game protocols are 90%+ unreliable UDP acting as a carrier wave for a very small subset of reliable/ordered data. The key here is to get this unreliable time series data across with the lowest latency and the greatest consistency (least jitter), and packet loss. That’s what we do at Network Next.
    - gafferongames 8y ago
      And also, we only accelerate the part of the game that needs the low latency. Game downloads and everything else... ehh, that’s a solved problem for a while now. We accelerate only the game traffic between client and server!
peteretep 8y ago
Where can I read more about League of Legends private internet as mentioned in the article?
- gafferongames 8y ago
  https://technology.riotgames.com/news/fixing-internet-real-time-applications-part-i https://technology.riotgames.com/news/fixing-internet-real-t... https://technology.riotgames.com/news/fixing-internet-real-time-applications-part-ii https://technology.riotgames.com/news/fixing-internet-real-t... https://technology.riotgames.com/news/fixing-internet-real-time-applications-part-iii https://technology.riotgames.com/news/fixing-internet-real-t...
Insanity 8y ago
Modern games have a problem that I believe is a direct consequence of consoles. The lack of a server browser. Just let me choose a server to which I have a good connection and I will be much happier. Instead of dropping me in games with random people all the time with random connection issues. One of the main reasons I don't play fps on the consoles is missing this. (And there are social advantages of playing with the same 'random' people as well). Having a server browser does not solve all issues mentioned in his presentation, but it does at least inform the player of the connection before they join a game.
- gsich 8y ago
  Random players can work with proper servers too, and many games just do that. The problem with server browsers, or user hosted servers are/were arbitrary rules enforced by the admin of that server. It's what killed Black Ops 1 on the PC.
  - cjslep 8y ago
    The benefit of the server browsers is that there were tons of positive gaming communities that competed around making the best and friendliest server possible in the early 2000's. I fondly remember playing Call of Duty as a tween with other community members that were adults and learned how to have fun, manage disagreements, and deal with problematic people without screaming profanities and raging about other peoples' mother. The death of self-hosted servers means the death of admin powers which means the death of building a gaming community one wants to see come to fruition. Those admin powers are key to protecting vulnerable people from general assholes. And the "modern" online "report a person" is woefully laughable in comparison.
    - gsich 8y ago
      Often times admins are the assholes here (kick for using weapon X, perk Y, attachment Z) and you have no way prior to connecting to know which stupid rules are now in place. (that and the fact that dedicated servers in BO1 meant you could rent them just from one or two places, no files for you)
simonsays2 8y ago
This sort of stuff has been around for a while, it helps some, but realistically your game needs to be designed from the ground up with latency and stutter in mind. If you do this, you wont need the complexity and expense of this sort of thing. First you need client side prediction and smoothing. Next, you must design your game so it follows certain very strict rules about player interaction. These rules allow the players to feel the have real time synchronization when in fact synchronization is lazy. The specific rules required are different for each game, and must be nailed down before game design begins.
- kaoD 8y ago
  Are you teaching game networking to Glenn Fiedler? You probably learnt it from him, or someone who read his articles! Now seriously: client side prediction is not magic and it always introduces its own problems, like getting killed after getting into cover (especially in low TTK shooters) or the opposite: bullets not registering when if they hit in your screen. Client side prediction is sleight of hand. It will never replace actually perfect network conditions and will still feel better the lower your latency is. It's necessary, but not sufficient.
  - gafferongames 8y ago
    Sure you can client side predict, but since the real game lives on the server, you want the minimal latency to that server so you see the most up to date version of what’s really happening, and your actions take effect as quickly as possible. You still want the lowest latency here.
  - simonsays2 8y ago
    Fuck yeah, I'm trying to warn people to go it right and not wait for a magic bullet that will never come.

172 points by gafferongames 8y ago

This was an awesome write up. The most interesting thing for me was learning that bottlenecks are not on the edge, but across the backbone because T1 carriers are optimizing for low cost, not low latency.

Jare 8y ago
The Riot links are pretty great too, very much worth it.
kingosticks 8y ago
The thing I'm still a bit confused about is: is this a limitation with how their networks fundamentally work, or an artefact of my product choice when I buy a relatively cheap home Internet service? i.e. if I want a specialist low-latency and low-jitter link, could I not just pay for those benefits by purchasing a business-grade service? Or is there no distinction once the data gets onto a peered-network? Or is the idea here that I don't have to cough up for all the benefits of that superior product when I only want some of them and only some of the time? And this whole thing does rely on the bottleneck not being at then edge, which I don't see any proof of or reason for. EDIT: is there any proof/data to support this?
- gafferongames 8y ago
  We are seeing players with Google Fiber get 100ms improvements. It’s not bad ISPs, it’s poor routing decisions and these are pretty much universal.

ndmrcf 8y ago

That reminds me of QuakeWorld's proxy servers.

mjevans 8y ago

Other things that can help: * A national policy that actually encourages competition > Also one that classifies 'the last mile' as public infrastructure to compete on top of. * Fiber to the home/MDU (Ethernet the rest of the way) * Where cable Internet must be used, the anti-buffer bloat parts of DOCSIS 3.1 (DOCSIS-PIE). * Actually having peer to peer focused ISPs, not content consumption ones. Upload should be no less than 25% of download; ideally symmetric. * Better visibility in to peering between users and actual effort applied to removing the bottlenecks.

weberc2 8y ago
> A national policy that actually encourages competition I'm all for improving the Internet and dunking on large ISPs, but what does this mean? Specifically what kind of national policy would foster competition (regulation tends to favor monopolies, right?)? Also, don't these private networks and marketplaces constitute competition? Or are you talking specifically about the network edge while the article is talking about the backbone? This article has me thinking that there is potentially a lot of economic efficiency to be unlocked by these sorts of marketplaces--basically routing based on economics, but maybe this is antithetical to net neutrality ("pay to play" internet routing)?
- mjevans 8y ago
  Ideally competition in all areas, but different areas might require different policy and this is a question that should actually be studied by experts, or at least in academia summarizing the results of policy in different nations (as well as how closely they might match similar housing density for actually lived in areas). Some places that I think should be at least initially studied / considered for LAST MILE* competition include: * LAST MILE (connectivity) being the connection between citizen's houses and peering/point-of-presence locations: connecting citizens to a marketplace of different service providers. * Remove legal barriers that prevent municipalities and utility entities from providing last mile connectivity (often laws lobbied (paid) for by existing monopoly or near monopoly companies). * Things similar to "Local Loop Un-bundling" > Require the existing providers break out last mile connectivity from their back-haul package. > Require they sell access to their private PoP at a cost which they also pay at a similar rate (to the PoP on the balance sheet). * Things equivalent to 'right of way' grants (poles on public land, trenches, etc) must be owned by the citizens. > Also leased at FRAND rates to any entities > Serviceable with 'one touch' by any entity licensed with the owners (citizens via the City/County/State/etc that owns it) Possibly other things I can't think of in a short break.

VectorLock 8y ago

Clever concept. I wonder if they'll publish any more information on how their service works. Things like what this 'route shader' is that people can read without having to go through Email-Our-Sales-Team shit.

politician 8y ago
It's a custom bidding script that tells their real-time auction platform how much you want to pay for route improvements for a particular user session. Typically, shaders are used to move vertices in 3d meshes or color pixels in the GPU. Network Next is just using the term in a creative way to help connect to their audience, but it's nothing different than a bidding model.
- gafferongames 8y ago
  Route shaders select the best route out of multiple ones available. Right now they could be used to for example, specify basic things like: acceptable latency (eg. I don’t want to pay to optimize a player’s session from 20ms down to 19ms, 20 is good enough), minimum improvement vs. public internet before a session takes network next (eg. 5ms improvement or greater), and of course maximum price for the bid. If we don’t find a suitable route at the price specified, the session takes the public internet and is free. Right now it’s fixed function like OpenGL 1.0 (eg. Parameters), but long term, it could even be LUA bytecode. Hence, route shader is not just a marketing term, it’s a real thing :)
  - kbenson 8y ago
    The really interesting thing I see for this is a possible other route for game revenue in the future. Instead of paying up front (traditional) pricing or advertising, or cosmetic (or not so cosmetic) in game purchases, you could have two tiers, free and optimized latency for a cost. Or some combination. Cosmetic in-game purchases aren't necessarily bad, but the amount my son has spent and tries to continue to spend (I strictly limit what of his money he can waste on it now) on fortnite stuff is ridiculous and concerning. A sane middle ground would be helpful.

NelsonMinar 8y ago

See also Haste, which has been doing this for a couple of years now: https://haste.net/ https://haste.net/ One thing that's interesting is how different these services need to be in different countries.

grenoire 8y ago
I think Network Next is developer-targeted, as opposed to Haste which offers a subscription service for gamers.
- gafferongames 8y ago
  This is correct. We work with game developers to sponsor the best route for their players. If we can’t significantly beat the public internet for that player (according to what the game developer specifies in their route shader), then we take the public internet and that player is free.

AlimJaffer 8y ago

As someone who works in the games space, I find this to be an interesting proposition. My primary concern revolves around similar 'net neutrality' aspects - more and more of the internet is becoming pay to play, this just further entrenches that theory if you want the best networking for your games as a studio.

politician 8y ago
That's a good observation. Given the alternative between bidding for performance from Network Next vs paying some flat fee to Comcast, et al. directly, I'd go with NN and let them deal with Comcast. ... and then maybe add "faster gameplay" power ups to our loot boxes (kidding).
- gafferongames 8y ago
  When we work with game developers, they find this aspect of the system incredibly compelling. Does it really make sense for every game company to negotiate with every transit provider and ISP to make sure traffic is delivered properly for their game? We think not. Plus, how could they get a fair price if they’re the small guy. Suppliers on Network Next cannot identify buyers and cannot have one price for one buyer and one for another. So now the small guys get the same price the big guys do and competition should drive the price down over time.

stcredzero 8y ago

The Internet is optimized for throughput at lowest cost. No amount of good netcode that you write can compensate for this. The problem is the internet itself. The internet doesn’t care about your game. The Internet did change, such that streaming movies became viable. Given that games are now bigger than movies, could it be argued that that the network lag we're seeing is the industry lagging behind the culture? (Which it generally does, at least a little.) A part of this might well involve special deals between large companies involving specialized infrastructure. There are such things now involved with movie streaming. Google's plans involve the infrastructure part of that already. What happens to net neutrality? Once game streaming becomes mature, then it's only a matter of time for App streaming to follow. We'll be back in the mainframe days, just with much larger companies with much larger multi-box cloud "mainframes", commanding the attention of a much larger portion of the population and the culture. "The computer is your friend. Trust the computer!" https://en.wikipedia.org/wiki/Paranoia_(role-playing_game) https://en.wikipedia.org/wiki/Paranoia_(role-playing_game)

politician 8y ago
> What happens to net neutrality? Counterintuitively, the market that Network Next is making shifts us closer to actual net neutrality than relying on the benevolence of FCC or Comcast by putting power into the hands of content providers where previously they were at the mercy of last-mile ISPs. If businesses can collectively bid for performance from private networks to offset actions by last-mile ISPs to cut costs and reduce service, then they also have the power to organize a general strike by refusing to bid for performance from traffic from specific ISPs until they work out better rates throughout the value chain. So I see this as a mechanism to restore the spirit of net neutrality despite the best wishes of the FCC.
- mrosett 8y ago
  > If businesses can collectively bid for performance from private networks You realize this is exactly what net neutrality prohibits?
  - 8y ago

AntiRush 8y ago

Valve recently announced partner access to their network backbone which addresses a lot of the problems mentioned here: * Access to our network, giving your players protection from attack, 100% reliable NAT traversal, and improved connectivity. * Tools for instantly estimating the ping between two arbitrary hosts without sending any packets. * A high quality end-to-end encrypted reliable-over-UDP protocol. >> Second, clients can select a route that gets off of the public Internet and onto our dedicated links as early as possible. On our backbone we can ensure that the routing is optimal, since we have peered with over 2,500 ISPs. We also prioritize the latency-sensitive game traffic over HTTP content downloads, which we can afford to do because game traffic makes up a relatively small percentage of our overall bandwidth utilization. And on our backbone, a sudden surge of traffic unrelated to gaming won’t degrade the experience. https://steamcommunity.com/groups/steamworks#announcements/detail/1791775741704351698 https://steamcommunity.com/groups/steamworks#announcements/d... They started using it in some of their games a couple of years ago (Counterstrike Global Offensive and Dota2 if I recall correctly).

Impossible 8y ago
Which is great if Steam is your only platform and you want to be tied to Valve forever, but as soon as you want your game on mobile, console or even another PC store like Epic or GOG you need another solution.
- soup10 8y ago
  i hate these underhanded "use this cool feature that we made to lock you in to our platform" tactics
  - bilbo0s 8y ago
    ??? Isn't the entire point of the original article to lock you into the Network Next platform by forcing you to implement their SDK? I think every network provider is going to work to lock you into their platform.
    - deleted 8y ago

the_mitsuhiko 8y ago

I'm just glad to see someone implements a real time bidding market without dropping the word blockchain. I really like the idea of it.

gafferongames 8y ago
I really hate blockchain :)

tinco 8y ago

This is epic. Great to see that someone who's been such a good support to the gaming community with his excellent learning resources is also being successful in his business. It's also interesting to see these "private internet" endeavours have popped up to solve centralization issues on the internet. Streaming sites like Youtube and Netflix have solved it in similar way, striking deals with ISPs, as have the big CDN's, simply putting their machines at the ISP's datacenters. Network.Next is solving this for the gaming domain, but there is a more general preferential routing on the internet problem this idea could be expanded to.

gafferongames 8y ago
Thank you very much.
- jay_kyburz 8y ago
  How about some Tribes Jetpack action!

jauer 8y ago

This reads like a traffic targeting system ala FB's cartographer (c10r) bolted onto an ad exchange (continuous bidding etc.) Interesting to see if they can keep the 10-second update intervals while scaling to more users. Curious to see if the market actually wants to pay for marginal improvements in gaming latency. I would have expected something like this targeting VoIP to be more lucrative.

robterrell 8y ago

Glenn, if you're still hanging around -- can the shader be written to account for the flow of P2P network traffic? Let's say I'm using an old-school configuration of a centralized lobby server that simply connects players to each other and the players connect directly over UDP?

gafferongames 8y ago
We currently only support client/server topology with dedicated servers.

petermcneeley 8y ago

The primary issue with the internet networking is the lack of Quality of Service (QoS) controls at the application layer.

gafferongames 8y ago
This is true, but without a marketplace and somebody paying, how can we tell whether the QoS requests are legitimate? I mean, “yeah, my traffic is always real-time priority”. People just will cheat it... so Network Next is a way of doing this, while saying, OK, for the truly real-time traffic, applications are willing to sponsor it. Since it’s paid, it won’t be exploited.

mkj 8y ago

This could be useful for long-distance remote desktop and SSH too. Curious how to build on top of it for diverse apps though (use it as a VPN somehow?)

gafferongames 8y ago
Right now we are an SDK that embeds in the client, server and backend for applications. It is open source too (BSD license), but not yet publicly released. Will be released soon, I look forward to seeing what people do with it, outside of games.

tarikjn 8y ago

Great reading this, I just want to drop a line to say that I have been working to solve precisely this problem -- for gaming but also other applications -- at the root. I have been working on a paper on this, here is the pre-proposal I published a whiled ago: https://docs.google.com/document/d/1xWaSB-3VSOMaiyxfjDqTalDCein0V46EbVrO6mFGPQs/edit?usp=sharing https://docs.google.com/document/d/1xWaSB-3VSOMaiyxfjDqTalDC... -- a lot of progress has been made since and I am looking for funding/grant before moving to publishing the actual paper and moving to implementation. The root of the issue is that the IP protocol is incomplete, in that is relies on private agreement and ISP servicing that cannot react to a real-time, real-world demand. IPv4 had some attempts at public QoS features that were mostly dropped in IPv6 because people realized it didn't work in practice. The only way to fix this for real is to merge ISP servicing and peering into a per-packet peer-to-peer network market. This solves a great many issues at the same time and essentially achieves the promise of mesh networking and more. If you would like to help, please drop me a line: tarik@graphnet.org (sorry no website at the moment).

openloop 8y ago

Get rid of TCP/IP. I’m sure you’re right sort of.

openloop 8y ago

Non uniform Data Protocol.

Animats 8y ago

Games need some low latency data, but not much. Player actions, character movement, and maybe bullets need low latency. Asset downloading does not. If you had a dial up modem running over a non-packet phone network, and didn't overload the buffering, you'd have better latency than the Internet. Many games make this distinction already. The latency-critical stuff goes over UDP, and it's limited to data where a missing packet is superseded by the next packet. Nothing is retransmitted. Bulk data goes over TCP, with reliable retransmission but more delay. It's too bad ISDN died off. 64Kb/s end to end, not packetization delay, and no jitter. The ideal gamer network would be an ISDN connection and a IP connection in parallel. If we had QoS systems where 1% of your maximum data rate could be at high priority, this, and the VoIP problem, would be mostly solved.

gsich 8y ago
Mostly it's just UDP with some TCP like reliable transfer built on top of it.
- gafferongames 8y ago
  This is correct. Most FPS game protocols are 90%+ unreliable UDP acting as a carrier wave for a very small subset of reliable/ordered data. The key here is to get this unreliable time series data across with the lowest latency and the greatest consistency (least jitter), and packet loss. That’s what we do at Network Next.
  - gafferongames 8y ago
    And also, we only accelerate the part of the game that needs the low latency. Game downloads and everything else... ehh, that’s a solved problem for a while now. We accelerate only the game traffic between client and server!

peteretep 8y ago

Where can I read more about League of Legends private internet as mentioned in the article?

gafferongames 8y ago
https://technology.riotgames.com/news/fixing-internet-real-time-applications-part-i https://technology.riotgames.com/news/fixing-internet-real-t... https://technology.riotgames.com/news/fixing-internet-real-time-applications-part-ii https://technology.riotgames.com/news/fixing-internet-real-t... https://technology.riotgames.com/news/fixing-internet-real-time-applications-part-iii https://technology.riotgames.com/news/fixing-internet-real-t...

Insanity 8y ago

Modern games have a problem that I believe is a direct consequence of consoles. The lack of a server browser. Just let me choose a server to which I have a good connection and I will be much happier. Instead of dropping me in games with random people all the time with random connection issues. One of the main reasons I don't play fps on the consoles is missing this. (And there are social advantages of playing with the same 'random' people as well). Having a server browser does not solve all issues mentioned in his presentation, but it does at least inform the player of the connection before they join a game.

gsich 8y ago
Random players can work with proper servers too, and many games just do that. The problem with server browsers, or user hosted servers are/were arbitrary rules enforced by the admin of that server. It's what killed Black Ops 1 on the PC.
- cjslep 8y ago
  The benefit of the server browsers is that there were tons of positive gaming communities that competed around making the best and friendliest server possible in the early 2000's. I fondly remember playing Call of Duty as a tween with other community members that were adults and learned how to have fun, manage disagreements, and deal with problematic people without screaming profanities and raging about other peoples' mother. The death of self-hosted servers means the death of admin powers which means the death of building a gaming community one wants to see come to fruition. Those admin powers are key to protecting vulnerable people from general assholes. And the "modern" online "report a person" is woefully laughable in comparison.
  - gsich 8y ago
    Often times admins are the assholes here (kick for using weapon X, perk Y, attachment Z) and you have no way prior to connecting to know which stupid rules are now in place. (that and the fact that dedicated servers in BO1 meant you could rent them just from one or two places, no files for you)

simonsays2 8y ago

This sort of stuff has been around for a while, it helps some, but realistically your game needs to be designed from the ground up with latency and stutter in mind. If you do this, you wont need the complexity and expense of this sort of thing. First you need client side prediction and smoothing. Next, you must design your game so it follows certain very strict rules about player interaction. These rules allow the players to feel the have real time synchronization when in fact synchronization is lazy. The specific rules required are different for each game, and must be nailed down before game design begins.

kaoD 8y ago
Are you teaching game networking to Glenn Fiedler? You probably learnt it from him, or someone who read his articles! Now seriously: client side prediction is not magic and it always introduces its own problems, like getting killed after getting into cover (especially in low TTK shooters) or the opposite: bullets not registering when if they hit in your screen. Client side prediction is sleight of hand. It will never replace actually perfect network conditions and will still feel better the lower your latency is. It's necessary, but not sufficient.
- gafferongames 8y ago
  Sure you can client side predict, but since the real game lives on the server, you want the minimal latency to that server so you see the most up to date version of what’s really happening, and your actions take effect as quickly as possible. You still want the lowest latency here.
- simonsays2 8y ago
  Fuck yeah, I'm trying to warn people to go it right and not wait for a magic bullet that will never come.