7 ms·

Solving the 100GB Game Download

118 points by monkeywork 3y ago

monkeywork 3y ago
Not affiliated with this project but it looks to be incredibly cool - very small team. I hope this shows up as a Linux / Steamdeck option in the near future as well. Demo: https://www.youtube.com/watch?v=e5wAn-4e5hQ https://www.youtube.com/watch?v=e5wAn-4e5hQ TL;DW: Mounts a virtual filesystem that on-demands downloads game content to your local drive in the form of a cache. Start playing games instantly without downloading the entire game.
- PaulHoule 3y ago
  I am little skeptical since a game needs to load one file and then another file and then another file and this could add up to an awful lot of waiting in an uncontrolled manner. Although big downloads are slow in my rural area I feel included by them because I can play the game in the end whereas streaming is not a good option at all.
  - monkeywork 3y ago
    https://www.youtube.com/watch?v=e5wAn-4e5hQ https://www.youtube.com/watch?v=e5wAn-4e5hQ This video shows a good demo of it with Halo on Steam.
    - interroboink 3y ago
      Thanks for the link. As they say in the video though, latency issues depend on the game's implementation. From [1]: "If the game does like a read-wait-read-wait pattern, then it's bottlenecked by the latency... and that can become a problem." Halo presumably has a solid streaming foundation, so it's a good fit for the service. [1] https://www.youtube.com/watch?v=e5wAn-4e5hQ&t=591s https://www.youtube.com/watch?v=e5wAn-4e5hQ&t=591s
      - teaearlgraycold 3y ago
        You could just push files to the client in advanced of request based on basic statistics.
        chevas 3y ago
        "Server Side Prediction: Via servers are constantly learning from repeat access patterns of players. This let's us make high quality predictions for what content you'll need, before you need it."
        teaearlgraycold 3y ago
        Nice
  - vvanders 3y ago
    Yep, you even had to account for this on spinning media(i.e. DVDs) where you'd want "seek-free" loading since you'd hitch/stall of you ended up bouncing all over the disk. I worked on a title where this was particularly problematic from a game design perspective(limited ram, large assets), we ended up designing in "slow" sequences to mask the disk load.
  - foota 3y ago
    They say they're planning to implement prediction, so learning from previous play of a game to learn what's going to be requested next and fetch it. Not sure how well it works.
  - imtringued 3y ago
    I wouldn't be surprised if the demo didn't increase latency and only throttled the download speed.
- oehpr 3y ago
  You know if it also took the time to utilize the bandwidth while it has it. Like via shouldn't stop downloading if a game doesn't request the files. They might even be able to integrated a bit of brains into it and keep a record of which files tend to be historically requested next for a game. In the background you're slowly grabbing those, and those threads pause for direct requests. I guess this is simpler but it feels like using via is costing you by inducing longer load times for the whole game.
  - 0xCMP 3y ago
    The page implies they learn from user's download patterns to do this and predict which files the user will need before they need them.
lpa22 3y ago
“Stuck with 120Mbps down” made me spit out my drink
- thomastjeffery 3y ago
  It's absurd and real at the same time. My apartment contract forces me to pay $99/mo. for roughly 120mbit, while most residences on my block can get 1gbit symmetric from Google Fiber for $70/mo.
  - pcchristie 3y ago
    I assume the parent comment is making the opposite point? I'm in Australia and I can get 90-100mbps which is very fast in this country - I'd say the average house gets 50-60mbps (guessing, haven't looked it up).
    - thomastjeffery 3y ago
      That's what I mean by absurd: 120mbit/sec is fantastic! It's also quite slow, depending on what you compare it to.
    - cookie_monsta 3y ago
      > Melbourne here. We just switched to the 5G modem and reliably get 300 down, 20 up. The nanobots I can live with
  - permo-w 3y ago
    those prices are crazy. it never ceases to amaze me how expensive life is in the US. that aside, I'd be delighted with 120mbi/s. that's just under 2 hours to download 100gb. it may well take you longer to actually go to the shop and buy a physical copy
    - thomastjeffery 3y ago
      Don't get me wrong, it's a wonderful thing, and I'm happy to have it. I've lived a lot of my life on much slower connections, usually 10-20mbit.. Over that same span of time, I've seen my local storage throughout go from ~100mb/sec to ~6000mb/sec. The improvement of internet bandwidth for the average person is moving at a glacial pace. I'm happy to live near the middle of that spectrum; and I'm frustrated to be so close, yet so far away, from the bleeding edge.
rkunde 3y ago
Related, have any major games using Binomial’s texture compression come out yet?
- sp332 3y ago
  Sure, it's available in Unity, Unreal, and Blender at least. The KTX2 format has been a Khonos group standard since 2021.
  - rkunde 3y ago
    I did notice that download sizes for the PS5 version of a game are smaller compared to the PS4 version. Maybe that’s why!
bbstats 3y ago
this is incredible.
Waterluvian 3y ago
I felt the urge to play Deadfire today so I got my gaming PC out of the closet. Shortly thereafter it went down a flight of stairs (don’t ask) and lost both wifi antennas. Downloading 20GB at about 5mbit was painfully retro, but I got the idea to Ethernet it to my MacBook and in a few clicks was borrowing its quite decent wifi connection. But my MacBook doesn’t have an Ethernet port or a normal usb port. So I got a dongle with Ethernet and plugged it into a USB to mini and then a mini to C (didn’t have a USB to USB C cable). Probably the most “LAN party experience” I’ve had in 15 years.
- deleted 3y ago
  [deleted]
- boomboomsubban 3y ago
  Obviously I don't know your setup, but that sounds harder than "move the computer next to the router, plug it in."
  - Waterluvian 3y ago
    I was thinking that but I didn’t want to have to get the ladders out.
    - boomboomsubban 3y ago
      That does sound like a scenario where digging through converters is easier.
  - lozenge 3y ago
    That's much less satisfying though. The excitement when you revive an old device by using a USB stick to pass a file to Bluetooth and then send the file via infrared can't be beat.
gregod 3y ago
Related: MS Office 2010 was/is available as "click-to-run" edition, based on a virtual streaming file system. https://learn.microsoft.com/en-us/office/troubleshoot/office-suite-issues/office-click-to-run-installation https://learn.microsoft.com/en-us/office/troubleshoot/office... The tech/product behind it has its roots in game distribution: https://en.wikipedia.org/wiki/Microsoft_App-V https://en.wikipedia.org/wiki/Microsoft_App-V
- permo-w 3y ago
  is that the same "Office Click-to-Run" that I've had to disable running from start-up 15 times?
  - pierrefermat1 3y ago
    My god is that thing genuinely cancerous, the autoruns software helped me get rid of it for good
    - permo-w 3y ago
      the real problem here isn't office ctr, it's the fact that there are 10 different ways to run a third-party program from start-up, many of which are not only opt-out but opt-out somewhere deep in the mysteries of the regisphere
7373737373 3y ago
Does this work well with M$ Flight Simulator? It's notorious for its long installation and update times and huge (100GB+) storage requirements
- monkeywork 3y ago
  oh man that's a great idea - i wish i could test but I get mine via gamepass not steam.
- deleted 3y ago
  [deleted]
- daveoc64 3y ago
  I would imagine not as Flight Simulator uses its own game launcher to download most of the game assets.
sorbusherra 3y ago
I kinda did this when i got my steam deck. I mounted my nas to steam deck, added it as a steam storage and moved my games there. It works rather well. No storage required on steam deck. It is slower than playing from SD, but not that bad on smaller games.
pcchristie 3y ago
This seems an incredible technical accomplishment. My one question is, is it supposed to be solving for disk space constraints or connection speed constraints? I'm guessing both, but the former is pretty cheaply solved these days with a new drive (or by finally admitting you only need to have 2-3 games installed at once and probably aren't playing a dozen games in parallel), and I'm not sure it seems to be solving the latter. The site says "stuck with 120mbps", and the demo on YouTube artificially caps download speeds to 200mbps, presumably as if to simulate "poor" connections. I'm in Australia and get 90-100 mbps, which is half of the demo speed, and I would be an upper outlier in this country. I can only imagine the people really suffering from 100GB downloads are on circa 5, 10, 20, 40 mbps. I get 10MBPs (megabytes) down reliably from Steam, which means even a 100GB game would "only" take ~ 2 and a half hours to download. The speed used in the demo would take ~ 75 minutes to download a 100GB game! Is that really an outrageously inconvenient wait? I sound more negative than I am. I guess what I'm saying is I'd love to know how slow your speed can get before this stops working well. That said, truly an impressive piece of tech.
- NikolaNovak 3y ago
  Right. 2.5 hours initial wait to then play a game normally is... fine. It's a non issue for most folks:) I noticed Battle.net from BLizzard tracks couple of thresholds in the progress bar - when the game will be fully downloaded, but also when the game is playable. For many updates, those two thresholds are very close together; but occasionally, the "playable" is half the distance or less of "completed", which makes me think somebody there is still putting in the effort :)
  - MBCook 3y ago
    PS5 has a similar system to let you start playing (properly designed) games before downloading is complete. I know iOS has something like that, though o don’t think I’ve ever seen it used. And I’d expect Xbox does as well, but I’m not in that ecosystem so I don’t know as much about their features.
    - fbdab103 3y ago
brucethemoose2 3y ago
This, and the apparent need for 4TB+ SSDs for Steam libraries is at odds with how I play games. I tend to bounce between 2 or 3 AAA games at most before I "finish" and delete them. Maybe I will come back some day, but I certainly don't switch every day to warrant keeping dozens at once or insta installing them, and if its an intricate mod setup or something I just archive it.
- deleted 3y ago
  [deleted]
forrestthewoods 3y ago
Neat. Looks like they’ve made a virtual file system that fetches files as the game tries to load them. This is exactly how I think tools like Git should work. I hadn’t actually consider it for game downloads! Neat.
foota 3y ago
Wow, that's a really nice looking UI.
candleknight 3y ago
I got the following error when I opened up a game page in my library (Hyper Light Drifter, if it matters): Check failed: g.steam_receive_prefix_bytes[1] == 0x31305456 File: client_steam.cpp Line: 2620 Last Error: 0x00000000. The operation completed successfully.
foota 3y ago
I wonder where they're hosting this, since most cloud providers would cost a fortune to download all the data.
- scosman 3y ago
  I don’t think they are hosting it? Seems like client side cache layer calling steam DL APIs?
  - CyberDildonics 3y ago
    Are you asking if that's what they are doing?
  - foota 3y ago
    Hm, I think they talked about the server some... It's possible there's a server coordinating things and the client talks to steam for the actual data, but that sounds (even more) complicated.
- monkeywork 3y ago
  Sounds like he is running a steam cache out of Chicago and Germany so likely running it off of a VPS service for now. Honestly if he gets this working really well (especially on linux / steamdeck) this could be an excellent aquire + hire by valve.
sva_ 3y ago
> Via hooks directly into Steam, making setup easy. But this doesn't hook into the game itself, causing issues with VAC like the AMD AntiLag+ recently did?
- rottencupcakes 3y ago
  It sounds like it’s just a file system, backed by a local cache and then the network. They can probably monitor file access to figure out what they should cache locally without actually touching any game files.
  - chaboud 3y ago
    Yeah. The probabilistic access patterns for games are going to be comically repeatables for prefetching.
dmitrygr 3y ago
“Solving the 100GB Game…” That’s been solved already. Play older games. Better story, less fluff. As an extra credit, they are smaller, don’t require top of the line hardware, do not need to insert any kernel anti-cheat rootkits, and do not spy on you.
laserDinosaur 3y ago
Ok the video on the home page is very confusing. I couldn't understand why it was demoing that you can play a game without waiting for the full install..when the video only runs games that are already installed. Turns out that's just a visual side affect of the app itself - everything appears as if it's already installed. So when he opens Destiny in the demo it's not actually installed yet (even though it looks like it is). I feel like that could have been made clearer.
zX41ZdbW 3y ago
We've recently made a similar thing for databases: https://github.com/ClickHouse/web-tables-demo https://github.com/ClickHouse/web-tables-demo It downloads the database on demand and optionally caches on the client side: https://presentations.clickhouse.com/meetup73/new_features/#11 https://presentations.clickhouse.com/meetup73/new_features/#... But it is not a new idea; for example, it was already implemented for SQLite a long time ago.
readyplayernull 3y ago
Slightly offtopic, is there a way to setup a virtual file system that caches most used files and removes least used of a project in internal SSD from an external HDD?
- yydcool 3y ago
  i recommend primocache on Windows
  - berbec 3y ago
    seconded. Primocache is wonderful, and doesn't suffer from the glaring issue using the powershell hack of Windows Storage Spaces [1]. You can set this up, with proper powershell scripts/commands, but Xbox games won't install to a Storage Space. It boggled my mind when I finally tracked this down. Primo Cache [2] has one disadvantage over storage spaces - the SSD cache space is dedicated as Cache, and does not add to the total space of the drive. 1TB SSD + 4TB hdd = 4TB available, vs storage spaces which will give you 5TB. This minor (to me) downside is outweighed by the fact it works for everything. It shows up as a normal drive, not a virtual disk, and it just works. I recommend it highly for anyone who has limited SSD space. 1: https://github.com/freemansoft/win10-storage-spaces https://github.com/freemansoft/win10-storage-spaces 2: https://www.romexsoftware.com/en-us/primo-cache/ https://www.romexsoftware.com/en-us/primo-cache/
- EricBurnett 3y ago
  Ooh, I'd love that. Or native steam functionality for the same. I never install games onto my SSD for that reason, so I'm at the mercy of whatever windows naturally does in that space. (Possibly nothing, as I don't remember manually configuring a SSD cache file on this computer, and last I checked that was required).
- tenebrisalietum 3y ago
  On Linux you have bcache and flashcache, but they work on the block layer and not file layer.
  - 3y ago
meristohm 3y ago
"To approximate average connections, demos were ran with a 120 Mbps download speed limit." Whoah. I know 3Mbps isn't blazing fast anymore, but it works fine for streaming video and downloading a 100Gb game from Steam overnight, back when I used to play games. Via seems like a cool idea, but the energy cost is high if I'm playing Vermintide 2 with only 10Gb available- that's a lot of continuous downloading, depending on how long I play. Not wanting to bother with downloading so much data contributed to to my preference for small games like Valheim and Deep Rock Galactic, and ultimately to being done with videogames.
sombragris 3y ago
I like it, but it does not "solve" the big downloads issue. You know what would solve the 100 GB game download problem? Reducing that to 30 or 40 GB, not inventing an (admittedly clever and nice) man in the middle streaming things to you in byte-sized chunks.
- dangus 3y ago
  That ain’t happening, and I don’t want it to happen, personally. All that data represents visual fidelity. I don’t want the of type games that boast amazing graphics and endless content to get smaller and more crude. I think this product is actually a big dead end because all you need is fast Internet to make this whole thing obsolete. Storage is also stupidly cheap even for blazing fast SSDs, for the price of a typical AAA title you can get a 2TB pcie nvme tlc SSD. I also see cable companies like Xfinity offering speeds above 1Gbps over plain copper. They’re selling speeds that consumer WiFi can’t even handle. Even the laziest monopoly cable companies have rolled out new docsis versions at minimal cost that deliver really impressive speeds on their ancient copper networks. If you saturate WiFi 6 at 600mbps a 100GB download is only going to take around 30 minutes if not less.
  - hakfoo 3y ago
    That assumes that the data is being used responsibly. Leftover assets that weren't properly cleaned before shipping the build? Uncompressed media that could be? (I recall that even well into the 2000s games often came with basically uncompressed CD audio tracks)
    - Guvante 3y ago
      To be fair audio compression wasn't exactly great in early 2000 and if the CD was required to play you could just not install the audio.
    - dangus 3y ago
      It just doesn’t bother me. If a game wasted something like 500GB of storage space that means it’s temporarily borrowing $20 worth of SSD storage. (I think the uncompressed audio was usually because of CPU decoding restraints, and the fact that a typical PlayStation game had such a crazy amount of “free” storage to work with on a disc compared to the amount of data needed to create the game and its world - might as well toss in voice acting and FMV video)
oldbbsnickname 3y ago
If this isn't it, the way this has been done before: the SnapVolumes/AppVolumes & Softricity method. 0. Write a custom storage driver that presents an read-only overlay volume where the app should be. 1. If the OS doesn't support it, redirect conflicting writes, deletes, and moves to another volume or cache locally. 2. Cache the app gradually on-demand from the network. 3. Profit!

118 points by monkeywork 3y ago

Not affiliated with this project but it looks to be incredibly cool - very small team. I hope this shows up as a Linux / Steamdeck option in the near future as well. Demo: https://www.youtube.com/watch?v=e5wAn-4e5hQ https://www.youtube.com/watch?v=e5wAn-4e5hQ TL;DW: Mounts a virtual filesystem that on-demands downloads game content to your local drive in the form of a cache. Start playing games instantly without downloading the entire game.

PaulHoule 3y ago
I am little skeptical since a game needs to load one file and then another file and then another file and this could add up to an awful lot of waiting in an uncontrolled manner. Although big downloads are slow in my rural area I feel included by them because I can play the game in the end whereas streaming is not a good option at all.
- monkeywork 3y ago
  https://www.youtube.com/watch?v=e5wAn-4e5hQ https://www.youtube.com/watch?v=e5wAn-4e5hQ This video shows a good demo of it with Halo on Steam.
  - interroboink 3y ago
    Thanks for the link. As they say in the video though, latency issues depend on the game's implementation. From [1]: "If the game does like a read-wait-read-wait pattern, then it's bottlenecked by the latency... and that can become a problem." Halo presumably has a solid streaming foundation, so it's a good fit for the service. [1] https://www.youtube.com/watch?v=e5wAn-4e5hQ&t=591s https://www.youtube.com/watch?v=e5wAn-4e5hQ&t=591s
    - teaearlgraycold 3y ago
      You could just push files to the client in advanced of request based on basic statistics.
      - chevas 3y ago
        "Server Side Prediction: Via servers are constantly learning from repeat access patterns of players. This let's us make high quality predictions for what content you'll need, before you need it."
        teaearlgraycold 3y ago
        Nice
- vvanders 3y ago
  Yep, you even had to account for this on spinning media(i.e. DVDs) where you'd want "seek-free" loading since you'd hitch/stall of you ended up bouncing all over the disk. I worked on a title where this was particularly problematic from a game design perspective(limited ram, large assets), we ended up designing in "slow" sequences to mask the disk load.
- foota 3y ago
  They say they're planning to implement prediction, so learning from previous play of a game to learn what's going to be requested next and fetch it. Not sure how well it works.
- imtringued 3y ago
  I wouldn't be surprised if the demo didn't increase latency and only throttled the download speed.
oehpr 3y ago
You know if it also took the time to utilize the bandwidth while it has it. Like via shouldn't stop downloading if a game doesn't request the files. They might even be able to integrated a bit of brains into it and keep a record of which files tend to be historically requested next for a game. In the background you're slowly grabbing those, and those threads pause for direct requests. I guess this is simpler but it feels like using via is costing you by inducing longer load times for the whole game.
- 0xCMP 3y ago
  The page implies they learn from user's download patterns to do this and predict which files the user will need before they need them.

lpa22 3y ago

“Stuck with 120Mbps down” made me spit out my drink

thomastjeffery 3y ago
It's absurd and real at the same time. My apartment contract forces me to pay $99/mo. for roughly 120mbit, while most residences on my block can get 1gbit symmetric from Google Fiber for $70/mo.
- pcchristie 3y ago
  I assume the parent comment is making the opposite point? I'm in Australia and I can get 90-100mbps which is very fast in this country - I'd say the average house gets 50-60mbps (guessing, haven't looked it up).
  - thomastjeffery 3y ago
    That's what I mean by absurd: 120mbit/sec is fantastic! It's also quite slow, depending on what you compare it to.
  - cookie_monsta 3y ago
    > Melbourne here. We just switched to the 5G modem and reliably get 300 down, 20 up. The nanobots I can live with
- permo-w 3y ago
  those prices are crazy. it never ceases to amaze me how expensive life is in the US. that aside, I'd be delighted with 120mbi/s. that's just under 2 hours to download 100gb. it may well take you longer to actually go to the shop and buy a physical copy
  - thomastjeffery 3y ago
    Don't get me wrong, it's a wonderful thing, and I'm happy to have it. I've lived a lot of my life on much slower connections, usually 10-20mbit.. Over that same span of time, I've seen my local storage throughout go from ~100mb/sec to ~6000mb/sec. The improvement of internet bandwidth for the average person is moving at a glacial pace. I'm happy to live near the middle of that spectrum; and I'm frustrated to be so close, yet so far away, from the bleeding edge.

rkunde 3y ago

Related, have any major games using Binomial’s texture compression come out yet?

sp332 3y ago
Sure, it's available in Unity, Unreal, and Blender at least. The KTX2 format has been a Khonos group standard since 2021.
- rkunde 3y ago
  I did notice that download sizes for the PS5 version of a game are smaller compared to the PS4 version. Maybe that’s why!

bbstats 3y ago

this is incredible.

Waterluvian 3y ago

I felt the urge to play Deadfire today so I got my gaming PC out of the closet. Shortly thereafter it went down a flight of stairs (don’t ask) and lost both wifi antennas. Downloading 20GB at about 5mbit was painfully retro, but I got the idea to Ethernet it to my MacBook and in a few clicks was borrowing its quite decent wifi connection. But my MacBook doesn’t have an Ethernet port or a normal usb port. So I got a dongle with Ethernet and plugged it into a USB to mini and then a mini to C (didn’t have a USB to USB C cable). Probably the most “LAN party experience” I’ve had in 15 years.

deleted 3y ago
[deleted]
boomboomsubban 3y ago
Obviously I don't know your setup, but that sounds harder than "move the computer next to the router, plug it in."
- Waterluvian 3y ago
  I was thinking that but I didn’t want to have to get the ladders out.
  - boomboomsubban 3y ago
    That does sound like a scenario where digging through converters is easier.
- lozenge 3y ago
  That's much less satisfying though. The excitement when you revive an old device by using a USB stick to pass a file to Bluetooth and then send the file via infrared can't be beat.

gregod 3y ago

Related: MS Office 2010 was/is available as "click-to-run" edition, based on a virtual streaming file system. https://learn.microsoft.com/en-us/office/troubleshoot/office-suite-issues/office-click-to-run-installation https://learn.microsoft.com/en-us/office/troubleshoot/office... The tech/product behind it has its roots in game distribution: https://en.wikipedia.org/wiki/Microsoft_App-V https://en.wikipedia.org/wiki/Microsoft_App-V

permo-w 3y ago
is that the same "Office Click-to-Run" that I've had to disable running from start-up 15 times?
- pierrefermat1 3y ago
  My god is that thing genuinely cancerous, the autoruns software helped me get rid of it for good
  - permo-w 3y ago
    the real problem here isn't office ctr, it's the fact that there are 10 different ways to run a third-party program from start-up, many of which are not only opt-out but opt-out somewhere deep in the mysteries of the regisphere

7373737373 3y ago

Does this work well with M$ Flight Simulator? It's notorious for its long installation and update times and huge (100GB+) storage requirements

monkeywork 3y ago
oh man that's a great idea - i wish i could test but I get mine via gamepass not steam.
deleted 3y ago
[deleted]
daveoc64 3y ago
I would imagine not as Flight Simulator uses its own game launcher to download most of the game assets.

sorbusherra 3y ago

I kinda did this when i got my steam deck. I mounted my nas to steam deck, added it as a steam storage and moved my games there. It works rather well. No storage required on steam deck. It is slower than playing from SD, but not that bad on smaller games.

pcchristie 3y ago

This seems an incredible technical accomplishment. My one question is, is it supposed to be solving for disk space constraints or connection speed constraints? I'm guessing both, but the former is pretty cheaply solved these days with a new drive (or by finally admitting you only need to have 2-3 games installed at once and probably aren't playing a dozen games in parallel), and I'm not sure it seems to be solving the latter. The site says "stuck with 120mbps", and the demo on YouTube artificially caps download speeds to 200mbps, presumably as if to simulate "poor" connections. I'm in Australia and get 90-100 mbps, which is half of the demo speed, and I would be an upper outlier in this country. I can only imagine the people really suffering from 100GB downloads are on circa 5, 10, 20, 40 mbps. I get 10MBPs (megabytes) down reliably from Steam, which means even a 100GB game would "only" take ~ 2 and a half hours to download. The speed used in the demo would take ~ 75 minutes to download a 100GB game! Is that really an outrageously inconvenient wait? I sound more negative than I am. I guess what I'm saying is I'd love to know how slow your speed can get before this stops working well. That said, truly an impressive piece of tech.

NikolaNovak 3y ago
Right. 2.5 hours initial wait to then play a game normally is... fine. It's a non issue for most folks:) I noticed Battle.net from BLizzard tracks couple of thresholds in the progress bar - when the game will be fully downloaded, but also when the game is playable. For many updates, those two thresholds are very close together; but occasionally, the "playable" is half the distance or less of "completed", which makes me think somebody there is still putting in the effort :)
- MBCook 3y ago
  PS5 has a similar system to let you start playing (properly designed) games before downloading is complete. I know iOS has something like that, though o don’t think I’ve ever seen it used. And I’d expect Xbox does as well, but I’m not in that ecosystem so I don’t know as much about their features.
  - fbdab103 3y ago

brucethemoose2 3y ago

This, and the apparent need for 4TB+ SSDs for Steam libraries is at odds with how I play games. I tend to bounce between 2 or 3 AAA games at most before I "finish" and delete them. Maybe I will come back some day, but I certainly don't switch every day to warrant keeping dozens at once or insta installing them, and if its an intricate mod setup or something I just archive it.

deleted 3y ago
[deleted]

forrestthewoods 3y ago

Neat. Looks like they’ve made a virtual file system that fetches files as the game tries to load them. This is exactly how I think tools like Git should work. I hadn’t actually consider it for game downloads! Neat.

foota 3y ago

Wow, that's a really nice looking UI.

candleknight 3y ago

I got the following error when I opened up a game page in my library (Hyper Light Drifter, if it matters): Check failed: g.steam_receive_prefix_bytes[1] == 0x31305456 File: client_steam.cpp Line: 2620 Last Error: 0x00000000. The operation completed successfully.

foota 3y ago

I wonder where they're hosting this, since most cloud providers would cost a fortune to download all the data.

scosman 3y ago
I don’t think they are hosting it? Seems like client side cache layer calling steam DL APIs?
- CyberDildonics 3y ago
  Are you asking if that's what they are doing?
- foota 3y ago
  Hm, I think they talked about the server some... It's possible there's a server coordinating things and the client talks to steam for the actual data, but that sounds (even more) complicated.
monkeywork 3y ago
Sounds like he is running a steam cache out of Chicago and Germany so likely running it off of a VPS service for now. Honestly if he gets this working really well (especially on linux / steamdeck) this could be an excellent aquire + hire by valve.

sva_ 3y ago

> Via hooks directly into Steam, making setup easy. But this doesn't hook into the game itself, causing issues with VAC like the AMD AntiLag+ recently did?

rottencupcakes 3y ago
It sounds like it’s just a file system, backed by a local cache and then the network. They can probably monitor file access to figure out what they should cache locally without actually touching any game files.
- chaboud 3y ago
  Yeah. The probabilistic access patterns for games are going to be comically repeatables for prefetching.

dmitrygr 3y ago

“Solving the 100GB Game…” That’s been solved already. Play older games. Better story, less fluff. As an extra credit, they are smaller, don’t require top of the line hardware, do not need to insert any kernel anti-cheat rootkits, and do not spy on you.

laserDinosaur 3y ago

Ok the video on the home page is very confusing. I couldn't understand why it was demoing that you can play a game without waiting for the full install..when the video only runs games that are already installed. Turns out that's just a visual side affect of the app itself - everything appears as if it's already installed. So when he opens Destiny in the demo it's not actually installed yet (even though it looks like it is). I feel like that could have been made clearer.

zX41ZdbW 3y ago

We've recently made a similar thing for databases: https://github.com/ClickHouse/web-tables-demo https://github.com/ClickHouse/web-tables-demo It downloads the database on demand and optionally caches on the client side: https://presentations.clickhouse.com/meetup73/new_features/#11 https://presentations.clickhouse.com/meetup73/new_features/#... But it is not a new idea; for example, it was already implemented for SQLite a long time ago.

readyplayernull 3y ago

Slightly offtopic, is there a way to setup a virtual file system that caches most used files and removes least used of a project in internal SSD from an external HDD?

yydcool 3y ago
i recommend primocache on Windows
- berbec 3y ago
  seconded. Primocache is wonderful, and doesn't suffer from the glaring issue using the powershell hack of Windows Storage Spaces [1]. You can set this up, with proper powershell scripts/commands, but Xbox games won't install to a Storage Space. It boggled my mind when I finally tracked this down. Primo Cache [2] has one disadvantage over storage spaces - the SSD cache space is dedicated as Cache, and does not add to the total space of the drive. 1TB SSD + 4TB hdd = 4TB available, vs storage spaces which will give you 5TB. This minor (to me) downside is outweighed by the fact it works for everything. It shows up as a normal drive, not a virtual disk, and it just works. I recommend it highly for anyone who has limited SSD space. 1: https://github.com/freemansoft/win10-storage-spaces https://github.com/freemansoft/win10-storage-spaces 2: https://www.romexsoftware.com/en-us/primo-cache/ https://www.romexsoftware.com/en-us/primo-cache/
EricBurnett 3y ago
Ooh, I'd love that. Or native steam functionality for the same. I never install games onto my SSD for that reason, so I'm at the mercy of whatever windows naturally does in that space. (Possibly nothing, as I don't remember manually configuring a SSD cache file on this computer, and last I checked that was required).
tenebrisalietum 3y ago
On Linux you have bcache and flashcache, but they work on the block layer and not file layer.
- 3y ago

meristohm 3y ago

"To approximate average connections, demos were ran with a 120 Mbps download speed limit." Whoah. I know 3Mbps isn't blazing fast anymore, but it works fine for streaming video and downloading a 100Gb game from Steam overnight, back when I used to play games. Via seems like a cool idea, but the energy cost is high if I'm playing Vermintide 2 with only 10Gb available- that's a lot of continuous downloading, depending on how long I play. Not wanting to bother with downloading so much data contributed to to my preference for small games like Valheim and Deep Rock Galactic, and ultimately to being done with videogames.

sombragris 3y ago

I like it, but it does not "solve" the big downloads issue. You know what would solve the 100 GB game download problem? Reducing that to 30 or 40 GB, not inventing an (admittedly clever and nice) man in the middle streaming things to you in byte-sized chunks.

dangus 3y ago
That ain’t happening, and I don’t want it to happen, personally. All that data represents visual fidelity. I don’t want the of type games that boast amazing graphics and endless content to get smaller and more crude. I think this product is actually a big dead end because all you need is fast Internet to make this whole thing obsolete. Storage is also stupidly cheap even for blazing fast SSDs, for the price of a typical AAA title you can get a 2TB pcie nvme tlc SSD. I also see cable companies like Xfinity offering speeds above 1Gbps over plain copper. They’re selling speeds that consumer WiFi can’t even handle. Even the laziest monopoly cable companies have rolled out new docsis versions at minimal cost that deliver really impressive speeds on their ancient copper networks. If you saturate WiFi 6 at 600mbps a 100GB download is only going to take around 30 minutes if not less.
- hakfoo 3y ago
  That assumes that the data is being used responsibly. Leftover assets that weren't properly cleaned before shipping the build? Uncompressed media that could be? (I recall that even well into the 2000s games often came with basically uncompressed CD audio tracks)
  - Guvante 3y ago
    To be fair audio compression wasn't exactly great in early 2000 and if the CD was required to play you could just not install the audio.
  - dangus 3y ago
    It just doesn’t bother me. If a game wasted something like 500GB of storage space that means it’s temporarily borrowing $20 worth of SSD storage. (I think the uncompressed audio was usually because of CPU decoding restraints, and the fact that a typical PlayStation game had such a crazy amount of “free” storage to work with on a disc compared to the amount of data needed to create the game and its world - might as well toss in voice acting and FMV video)

oldbbsnickname 3y ago

If this isn't it, the way this has been done before: the SnapVolumes/AppVolumes & Softricity method. 0. Write a custom storage driver that presents an read-only overlay volume where the app should be. 1. If the OS doesn't support it, redirect conflicting writes, deletes, and moves to another volume or cache locally. 2. Cache the app gradually on-demand from the network. 3. Profit!