16 ms·
Why PS4 downloads are so slow
- ckorhonen 9y agoInteresting - definitely a problem I've encountered, though I had assumed the issues fell more on the CDN side of things. Anecdotally, when I switched DNS servers to Google vs. my ISP, PS4 download speeds improved significantly (20 minutes vs. 20 hours to download a a typical game).
- menacingly 9y agoI have slow downloads too (with google DNS), but I would think this wouldn't impact it much. It's not like it's performing a DNS lookup for each chunk, right?
- philsnow 9y agoWell no, the issue is that the ps4 is constraining the tcp receive window (which is how much data is in flight at once), so if your ISP DNS gives you a CDN node that's 200ms away but google's DNS gives you a CDN node that's 10ms away, you get receive_window bytes every 10ms instead of every 200ms. There's a similar thing happening (different DNS giving vastly different experience) with Netflix as well: Netflix has local nodes _inside_ many ISP's networks, close to ISP subscribers. If you use your ISP's DNS (and if they're doing their job properly), they'll send you to one of the local nodes and it'll take much less time to buffer, your stream will start faster, you'll drop less etc. Whereas if you use Google's DNS, they don't (necessarily) know the inner workings of your ISP so they send you to a generic Netflix CDN node that's near-ish but still probably 10-100x "farther" as the photon flies. If you want to have a mix of DNS settings per domain name, you can set up a dnsmasq inside your network and configure it to recurse to google's DNS normally but recurse to your ISP's DNS for just netflix.com, nflxvideo.com etc. see for instance [0] for one way to set that up. they have it running on a dd-wrt router but I have it running just on some rando linux machine somewhere in the house and configure my DHCP to tell everything to use that machine's IP as the primary DNS. [0] https://github.com/ab77/netflix-proxy/wiki/setting-up-netflix-proxy-with-dd-wrt https://github.com/ab77/netflix-proxy/wiki/setting-up-netfli...
- menacingly 9y agopretty helpful comment, thanks
- stephen_g 9y agoHmmm, does that actually improve speeds for Netflix, or was that somebody just doing it because they thought it would? It's been a while since I've looked into it, but I thought that Netflix's devices peer with the provider's routers and intercept traffic through that way, so the DNS server shouldn't matter (because they would share the same IPs as the publicly accessible CDN load balancers). I thought most CDNs worked that way now - using DNS never really worked well.
- hug 9y agoAnecdata: Office 365 will put you on an APAC CDN instead of an AU CDN if you use Google Public DNS from Australia, but not so if you use local ISP DNS (like, say, Telstra's).
- voltagex_ 9y agoI'm using Internode's DNS here and I get servers from Hong Kong and Singapore mostly.
- philsnow 9y agoI have only my own anecdata, which strongly support the hypothesis.
- jsnell 9y agoYeah, there are definitely other possible reasons. E.g. one workaround I found had Australian users block a specific IP in their firewall. Turns out the DNS configuration was bad, and they were getting an overseas CDN server every now and then. Due to the way the PS4 schedules the downloads (chunks downloaded serially, chunk boundaries at static byte offsets rather than based on time) having even one bad server in the rotation can really ruin performance. But CDN configuration errors can happen to anyone. I think this client-side mess is a much more creative way to screw up :)
- FridgeSeal 9y agoI don't suppose you'd know what that ip is and if this fix still works do you?
- tyfon 9y agoPersonally I've never had any issues but I see people complaining about it. I guess it's location more than anything. I have a fiber connection in Norway and it always downloads games at maximum speed.
- frik 9y agoPS4 and Switch have at least no peer-to-peer download. Win10 and XboxOne have peer-to-peer download - who would want that, bad for users, wasting upload bandwidth and counts against your monthly internet consumption. https://www.reddit.com/r/xboxone/comments/3rhs4s/xbox_updates_can_we_disable_p2p/ https://www.reddit.com/r/xboxone/comments/3rhs4s/xbox_update...
- derefr 9y agoIt works well if both devices are on the same LAN. Like inside a corporate office. What organization has multiple XBOs, though? Game dev studios? Hospitals? It must be much rarer than the Win10 case.
- tortasaur 9y agoI want that, as it normalizes the need for large upload bandwidth. The only reason it's such a scarce commodity now is that households aren't perceived as needing that bandwidth. The proliferation of legal peer-to-peer systems will hopefully drive the spread of symmetric connections.
- MBCook 9y agoOne of the reasons I switched off of AT&T uVerse is they seemed to give you the absolute minimum upload bandwidth needed to achieve the advertised download bandwidth. So if you were downloading or streaming and uploaded anything more than a small image your download speed would plummet as you ran out of upstream bandwidth. Online backups/syncing? Hope you're not trying to do anything else that requires bandwidth.
- jumpkickhit 9y agoI normally warm boot mine, saw the speed increase with nothing running before, so guess I was on the right track. I hope this is addressed by Sony in the future, or at least let us select if a download is a high priority or not.
- jcastro 9y agoLancache says it caches PS4 and XBox, anyone using this? https://github.com/multiplay/lancache https://github.com/multiplay/lancache (I use steamcache/generic myself, but should probably move to caching my 2 consoles as well).
- erikrothoff 9y agoTotally unrelated but: Dang it must be awesome to have a service that people dissect at this level. This analysis is more in depth and knowledgable than anything I've ever seen while employed at large companies, where people are literally paid to spend time on the product.
- PaulHoule 9y agoIt is astonishing how oblivious people who are "paid to spend time on the product" can be about the products they ship. Case in point, Microsoft shipped a EXPLORER.EXE with Windows 10 that is 20% whitespace because of an Adobe invented metadata format that I bet nobody involved with Windows ever thought about it. See http://ontology2.com/essays/LookingForMetadataInAllTheWrongPlaces.html http://ontology2.com/essays/LookingForMetadataInAllTheWrongP...
- Sukera 9y ago> Quick note: The situation is not different on Linux, MacOS, or other operating systems because all executable formats have some way to embed images. What publishers should do is remove unncessary metadata before publishing, which is easy to do in the case of the PNG because we can simply omit the iTXt chunk. From your link - so I guess the situation on linux isn't different? Or am I missing something, other than that it depends on the embedded PNG?
- PaulHoule 9y agoI don't expect it to be different on Linux or any other operating system.
- jwilk 9y agoI'm not quite sure what exactly the author's point is. Yes, you could embed images in ELFs, too, if you really wanted, but why? There's no standardized way to do it, and putting the images in separate files is easier.
- 9y ago
- ploxiln 9y agoReminds me of how Windows Vista's "Multimedia Class Scheduler Service" would put a low cap on network throughput if any sound was playing: http://www.zdnet.com/article/follow-up-playing-music-severely-degrades-network-transfer-performance-in-vista/ http://www.zdnet.com/article/follow-up-playing-music-severel... Mark Russinovich justified it by explaining that the network interrupt routine was just too expensive to be able to guarantee no glitches in media playback, so it was limited to 10 packets per millisecond when any media was playing: https://blogs.technet.microsoft.com/markrussinovich/2007/08/26/vista-multimedia-playback-and-network-throughput/ https://blogs.technet.microsoft.com/markrussinovich/2007/08/... but obviously this is a pretty crappy one-size-fits-all prioritization scheme for something marketed as a most-sophisticated best-ever OS at the time: https://blogs.technet.microsoft.com/markrussinovich/2008/02/04/inside-vista-sp1-file-copy-improvements/ https://blogs.technet.microsoft.com/markrussinovich/2008/02/... Many people had perfectly consistent mp3 playback when copying files over the network 10 times as fast in other OSes (including Win XP!) Often a company will have a "sophisticated best-ever algorithm" and then put in a hacky lazy work-around for some problem, and obviously don't tell anyone about it. Sometimes the simpler less-sophisticated solution just works better in practice.
- ShabbosGoy 9y agoThanks for the links and reading material. It's amazing how important the scheduler is in an OS. I'm no expert on OS-level programming, but couldn't they have just used a simple round robin scheduler?
- zeta0134 9y agoIn this case the scheduler isn't as much of a problem as the interrupt service routine, which by its very nature interrupts the normal thread-level scheduler to do some work with the hardware. Both the Network Interface Card (or wireless chip) and the Audio hardware generate interrupts for a variety of reasons. Network cards generate interrupts when data is finished sending or receiving. Audio hardware generally generates its own interrupts when its buffers have emptied out, to signal the OS to provide it more audio to play. In this case, it sounds like Vista's network interrupt was running for so long that it was taking priority over other interrupts (audio) and causing them to not be serviced quickly enough. This is definitely a problem, but the fix should not have been to limit the amount of work done per interrupt, but instead to delegate that work to some other process with lower priority. It's weird to think of audio as higher priority than networking traffic, but in this case that actually makes a ton of sense. Audio generation only requires ~88.2 KB of data per second, but latency is much lower and less tolerable by a human listener. Modern network connections are working with multiple MB per second, and are (relatively) more tolerant of slight delays. Same with mouse and keyboard input, it's why your mouse tends to move smoothly on most Windows machines even when the rest of the OS is obviously struggling with some heavy task.
- mbrd 9y agoThis Reddit thread also has an interesting analysis of slow PS4 downloads: https://www.reddit.com/r/PS4/comments/522ttn/ps4_downloads_are_notoriously_slow_i_might_have/ https://www.reddit.com/r/PS4/comments/522ttn/ps4_downloads_a...
- lossolo 9y agoDNS based GEO load balancing/CDN's are wrong idea today. For example if you use DNS that has bad configuration or one that is not supplied by your ISP, then you could be routed to servers thousands km/miles from your location. Last time I've checked akamai used that flawed dns based system. What you want to use now is what for example cloudflare uses which is anycast IP. You just announce same IP class on multiple routers/locations and all traffic is routed to the nearest locations thanks to how BGP routing works.
- toast0 9y agoAs with all things, there are tradeoffs. As you say, GEO DNS doesn't work very well when the authoritative server doesn't get enough information to give a good answer; for example, if all of a nationwide ISP uses the same pool of recursive DNS, and doesn't proved EDNS client subnet, the authoritative server will not get any signal about where in the country the users are. Or, if you use DNS servers in another country for some reason, you may be assumed to be nearby to that country. Anycast isn't perfect either though. It may be more likely to get you a route with fewer hops, but that's not necessarily the lowest latency link. And if there's a congested link in the path, it's much harder to steer a portion of users, but not all to alternate servers. You also have to deal with the potential of routing changing from packet to packet, and potentially different locations getting different parts of the same connection. It's not 'just announce the same IPs from multiple locations' and take a nap. Anyway, unless you have thousands of datacenters well distributed throughout the globe, a lot of people are going to have high latency to whatever server is chosen, and high latency and very small windows == poor throughput, regardless of how you get to a server.
- londons_explore 9y agoDo any big networks do anycast tcp stream reassembly when a connection from a user gets some packets split between different datacenters? One could imagine a system where one datacenter forwards the packets (encapsulated) to the other so that the stream isn't broken.
- mjevans 9y agoThe DNS server is the wrong place to make that decision. If there is a set of choices and a decision to make, letting the client choose would be the best option. I think I'd start with a client shuffling the list of results and asking for small fragments of the dataset from each. Based on the latency, bandwidth detected, and the size of the requested data it would then pick the most optimal set of servers. This also allows for fallback in case servers are unresponsive.
- ambar123 9y agoI dont want to know...
- Tloewald 9y agoIt's not just four years into launch since the PS3 was at least as bad.
- foobarbazetc 9y agoThe CDN thing is an issue too. Using a local DNS resolver instead of Google DNS helped my PS4 speeds. The other "trick" if a download is getting slow is to run the in built "network test". This seems to reset all the windows back even if other things are running.
- jsnell 9y ago> The other "trick" if a download is getting slow is to run the in built "network test". This seems to reset all the windows back even if other things are running. I did not see that in my testing. The built in speedtest runs with a large receive window, but the store downloads are not affected. (You can see an example in the last graph; there's a speedtest early on that has a receive window 100x larger than the PSN downloads that are crawling along). It's probably just a placebo.
- matwood 9y ago> It's probably just a placebo. Possibly, but I think the OP above was onto something about it 'resetting' the windows. I just tested downloading a large game while I had another game running. The PS4 showed it was going to take 6 hours. I closed the other game I had running, and it still showed 6 hours. I ran the speed test, and now it shows it's going to take 50 minutes. BTW, thanks for running these tests as I have some stuff to try instead of waiting hours for a game to download :)
- mlonkibjuyhv 9y agoBut how long did it end up actually taking?
- jsnell 9y agoMy experience is that the finish time prediction isn't reliable. It's based average speed over the whole connection, not the current speed. So the projection will lag behind if the speed changes dramatically. Maybe the speedtest somehow resets the projection? The measurement really needs to be done on the wire.
- 9y ago
- g09980 9y agoWant to see something like this for (Apple's) App Store. Downloads are fast, but the App Store experience itself is so, so slow. Takes maybe five seconds to load search results or reviews even on a wi-fi connection.
- dfee 9y agoI’m glad I’m not the only one who’s noticed this.
- wingworks 9y agoMe too, it's painfully slow
- matthewbauer 9y agoI've noticed this too but it seems to be the same thing on both macOS and iOS, maybe it's on the server end?
- ex3ndr 9y agoSame with Google Play since day 0.
- ripdog 9y agoReally? On my galaxy note 4, Google Play pages open within ~400ms consistently.
- dingo_bat 9y agoHave you used Apple store? Google play is much faster. There is no comparison.
- pier25 9y agoI use both iOS and Android on a daily basis. In my experience the Play store is consistently faster. I'm not surprised, Google is better at these kind of things.
- Tanegashima 9y ago
- hgdsraj 9y agoWhat download speeds do you get? I usually average 8-10 MB/s
- lqdc13 9y agoI've gotten anywhere between 100KB/s and 60MB/s on the same ISP.
- tgb 9y agoSorry for the newbie question, but can someone explain why the round trip time is so important for transfer speeds? From the formula I'm guessing something like this happens: server sends DATA to client, client receives DATA then sends ACK to server, server receives ACK and then finally goes ahead and sends DATA2 to the client. But TCP numbers their packets and so I would expect them to continue sending new packets while waiting for ACKs of old packets, and my reading of Wikipedia agrees. So what causes the RTT dependence in the transfer rate?
- mburns 9y ago>But TCP numbers their packets and so I would expect them to continue sending new packets while waiting for ACKs of old packets, and my reading of Wikipedia agrees. TCP keeps a sliding congestion window that relies on RTT. https://en.wikipedia.org/wiki/TCP_congestion_control#Fast_retransmit https://en.wikipedia.org/wiki/TCP_congestion_control#Fast_re...
- londons_explore 9y agoThere is something called the receive window. That is the size of data that can be sent before getting more ACK's back. If the delay is longer, and the receive window is the same size, then total throughput is lower.
- preinheimer 9y agoI think this is a good question, I did some reading myself! The blog post talks about "receive windows" which comes into play here. The server will send up to the number of bytes specified in the receive window before needing to start seeing some ACKs. So the shorter the round trip time, the less likely the server is to spend time twiddling it's thumbs waiting for an ack, because it's exhausted its receive window. Of course, increasing the size of the receive window would also help. (I read this answer to get here, I could be way off: https://stackoverflow.com/questions/9613197/what-determines-number-of-data-packets-before-an-ack-is-sent-net-sockets-on-se https://stackoverflow.com/questions/9613197/what-determines-... )
- syncsynchalt 9y ago
- deleted 9y ago[deleted]
- Reedx 9y agoPS3 was even worse in my experience - PS4 was a big improvement, although still a lot slower than Xbox. However, with both PS4 and Xbox One it's amazingly slow to browse the stores and much of the dashboard. Anyone else experience that? It's so bad I feel like it must just be me... I avoid it as much as possible and definitely decreases the number of games I buy.
- s_kilk 9y ago> it's amazingly slow to browse the stores and much of the dashboard Same, my PS4 can take up to a few minutes to load store pages. And sometimes it just hangs, or times out, and I need to quit the app and try again. It's really not a good experience at all.
- spike021 9y agoThe PS4's PSN store's search function is terrifically awful. Why on earth do they think "typing" is easier using a spin dial for each character in the search query? Not only that but the dial moves so slowly.
- rado 9y agoNot network related (?), but the slow and choppy PS4 UI/dashboard is one of my "favourite" features. What were they thinking?
- jhasse 9y ago"JavaScript is just as fast as C++ nowadays."
- sydney6 9y agoIs it possible that lacking TCP Timestamps in the Traffic from the CDN is causing the TCP Window Size Auto Scaling Mechanism to fail? See this commit: https://svnweb.freebsd.org/base?view=revision&revision=316676 https://svnweb.freebsd.org/base?view=revision&revision=31667...
- andrewstuart 9y agoIts bizarre because I bought something from the PlayStation store on my PS4 and it took DAYS to download. The strange part of the story is that it took so long to download that the next day I went and bought the game (Battlefield 4) from the shop and brought it back home and installed it and started playing it, all whilst the original purchase from the PlayStation store was still downloading. I ask Sony if they would refund the game that I bought from the PlayStation store given that I had gone and bought it elsewhere from a physical store during the download and they said "no". So I never want to buy from the PlayStation store again. Why would Sony not care about this above just about everything else?
- ripdog 9y agoBecause they are in a dominant market position. Honestly, it's your fault for buying a console - you knew you'd be at sony's mercy due to the monopolistic nature of consoles. The cost of switching to the one comparible console option is so great, most people will not bother unless sony's issues become too great - and network speed is presumably not a deal-breaker for enough people for sony to care. So you will continue to suffer abominable download speeds on your PS4, because the cost of switching is too great for most. Basically, suck it up. I hate to get all PCMR on you, but this is why everyone should game on PC. If my steam purchase downloaded super slow, I could simply get a refund on steam (an automated process) and purchase the same game on GOG, or on Humble Store. Sometimes Origin or Uplay. PC means freedom, or not being at the mercy of a single uncaring corporate entity.
- andrewstuart 9y agoI left PC gaming behind years ago. I like to sit on the couch and I want my gaming device to be a black box that does its job and is dedicated to couch based entertainment (I use it for TV too). PC gaming requires a...... PC... usually sitting at a desk, much more powerful than I need, lots of configuration etc etc to hard.... I'm a couch gamer in the evening after the programming is done. I don't care much about the PlayStation download thing. I generally prefer to buy my games on DVD anyway. Sony should care however.... despite not wanting to spend money on digital downloads I find I spend ever more with Apple App Store because despite my hesitations they make it easy and it works, whereas I don't want to buy digital with Sony and they make it easy not to do that. >>>PC means freedom, or not being at the mercy of a single uncaring corporate entity I just don't care about that stuff any more.. life's too short to worry about who our corporate overlords are.
- bitwize 9y agoThis is so that there's plenty of bandwidth available for networked play. The Switch firmware even states that it will halt downloads if a game attempts to connect to the network.
- jonny_eh 9y agoEven if the game is suspended? It seems reasonable that you're right, but it also seems like an oversight to not lift the throttling when games are suspended, or if the game's not being played online.
- galonk 9y agoI always assumed the answer was "because Sony is a hardware company that has never understood the first thing about software." Turns out I was right.
- tenryuu 9y agoI remember someone hacking at this issue a while ago. They blocked Sony Japan's server, of which the download was coming from. The Playstation the fetched the file from a more local server, of which the speed was considerable faster. Really strange
- motoboi 9y agoCloser server means lower rtt. All things equal, this translates to a faster transmission.
- cdevs 9y agoAs a developer people seemed surprised I don't have some massive gaming rig at home but there's something about it that feels like work. I don't want to sit up and be fully alert - I did that all day at work I want 30 mins to veg out on a console jumping between Netflix and some quick multiplayer game with less hackers glitchin out on the game. It seems impressive what PS4 attempts to accomplish while you're playing a game and yet try and download a 40gig game and some how tip toe in the background not screwing up the gaming experience. I couldn't imaging trying to deal with cranking up the speed here and there while keeping the game experience playable in a online game. Chrome is slow? Close you're 50 tabs, want faster PS4 downloads, close your games/apps. Got it.
- dcow 9y agoBut it's just one app and boom 7kb receive window. The author goes into how terrible the impl is. I'm sure it might seem crazy if you're not familiar with kernel networking. Also I hope you get more than 30 min of free time each day (=
- cdevs 9y agoI know my comment came off as putting the article down but I always have praise for anyone on the hunt for issues of why Firefox does this or why PS4 does this. I shoot for 30 mins of game time but we all know how that turns into 2-3 hours when I ment to start some programming side project. Games are the devil.
- lokedhs 9y agoAs one piece of information I offer my own experience with PSN downloads on the PS4. I'm in Singapore and my normal download speed is around 250 Mb/s, sometimes getting closer to 300. However, I sometimes download from the Swedish store as well, and those download speeds are always very slow. I don't think I've ever gone above one tenth of what I get with local downloads. That said, bandwidth between Asia and Singapore are naturally more unpredictable, so I don't know if I can blame Sony here. My point is that PS4 downloads can be very fast, and the Singapore example is evidence of this fact.
- rohmish 9y agoI've experienced similar thing with PS3. That said, unlike the author I am too lazy to inspect traffic on that.
- deafcalculus 9y agoWhy doesn't PS4 use LEDBAT for background downloads? Wouldn't this address the latency problem without sacrificing download speeds? AFAIK, Macs do this at least for OS updates.
- Cryptoboss 9y agoI rented Final Fantasy XV last night to try it out. I put it in and had a 13 Gig update. So of course I had to download it all night to even get a chance to play it. Then I finally open it and it's takes another hour to install it. Now I'm not even sure I'll get to play it much before I even have to return it. It's a problem.
- Companion 9y agoI actually dread downloading patches and whatnot from PSN for this reason. I have a 500 mbit connection, that works perfectly well on all my other devices, but my Ps4 Pro, well, it's incredibly fickle. There'll be days where download speeds are good, and then there'll be days where even downloading 200mb is a challenge. It's all wired, so it's not a wifi related problem. I went through different routers and even changed ISP's once and the problem still persisted, so I think I have ruled it out as being on my end. It seems to be some weird QoS feature of the PS4, or possibly the PSN not being up to scratch - I don't know. Stuff like closing all background apps, or changing DNS, they don't really seem to do anything for me. Sometimes pausing/unpausing does help, though..