6 ms·

Foveated streaming! That's a great idea. Foveated rendering is complicated to implement with current rendering APIs in a way that actually improves performance,

• xeonmc 11mo ago
  > Roughly equivalent resolution to Quest 3 and less than Vision Pro. This won't be suitable as a monitor replacement for general desktop use. But the price is hopefully low. Question, what is the criteria for deciding this to be the case? Could you not just move your face closer to the virtual screen to see finer details?
  • modeless 11mo ago
    Not only would it be a chore to constantly lean in closer to different parts of your monitor to see full detail, but looking at close-up objects in VR exacerbates the vergence-accommodation mismatch issue, which causes eye strain. You would need varifocal lenses to fix this, which have only been demonstrated in prototypes so far.
    • Fernicia 11mo ago
      Couldn't you get around that by having a "zoom" feature on a very large but distant monitor?
      • wongarsu 11mo ago
        Yes. You can make a low-resolution monitor (like 800x600px, once upon a time a usable resolution) and/or provide zoom and panning controls I've tried that combination in an earlier iteration of Lenovo's smart glasses, and it technically works. But the experience you get is not fun or productive. If you need to do it (say to work on confidential documents in public) you can do it, but it's not something you'd do in a normal setup
      • rtkwe 11mo ago
        For a small taste of what using that might be like turn on screen magnification on your existing computers. It's technically usable but not particularly productive or pleasant to use if you don't /have/ to use it.
      • potatolicious 11mo ago
        Yes but that can create major motion sickness issues - motion that does not correspond top the user's actual physical movements create a dissonance that is expressed as motion sickness for a large portion of the population. This is the main reason many VR games don't let you just walk around and opt for teleportation-based movement systems - your avatar moving while your body doesn't can be quite physically uncomfortable. There are ways of minimizing this - for example some VR games give you "tunnel vision" by blacking out peripheral vision while the movement is happening. But overall there's a lot of ergo considerations here and no perfect solution. The equivalent for a virtual desktop might be to limit the size of the window while the user is zooming/panning.
  • potatolicious 11mo ago
    There's no precise criteria but the usual measure is ppd (pixels per degree) and it needs to be high enough such that detailed content (such as text) displayed at a reasonable size is clearly legible without eye strain. > "Could you not just move your face closer to the virtual screen to see finer details?" Sure, but then you have the problem of, say, using an IMAX screen as your computer monitor. The level of head motion required to consume screen content (i.e., a ton of large head movements) would make the device very uncomfortable quite quickly. The Vision Pro has about ~35ppd and generally people seems to think it hits the bar for monitor replacement. Meta Quest 3 has ~25ppd and generally people seem to think it does not. The Steam Frame is specs-wise much closer to Quest 3 than Vision Pro. There are some software things you can do to increase legibility of details like text, but ultimately you do need physical pixels.
    • giobox 11mo ago
      Even the vision pro at 35ppd simply isn't close to the PPD you can get from a good desktop monitor (we can calculate PPD for desktop monitors too, using size and viewing distance). Apple's "retina" HiDPI monitors typically have PPD well beyond 35 at ordinary viewing distances, even a 1080p 24 inch monitor on your desk can exceed this. For me personally, 35ppd feels about the minimum I would accept for emulating a monitor for text work in a VR headset, but it's still not good enough for me to even begin thinking about using it to replace any of my monitors. > https://phrogz.net/tmp/ScreenDensityCalculator.html https://phrogz.net/tmp/ScreenDensityCalculator.html
      • potatolicious 11mo ago
        Oh yeah for sure. Most people seem to accept that 35ppd is "good enough" but not actually at-par with a high quality high-dpi monitor. I agree with you - I would personally consider 35ppd to be the floor for usability for this purpose. It's good in a pinch (need a nice workstation setup in a hotel room?) but I would not currently consider any extant hardware as full-time replacements for a good monitor.
        • andybak 11mo ago
  • jayd16 11mo ago
    It's just about what pixel per degree will get you close to the modern irl setup. Obviously it's enough for 80 char consoles but you'd need to dip into large fonts for a desktop.
    • rtkwe 11mo ago
      I did the math on this site and I'd have to hunch less than a foot from the screen to hit 35 PPD on my work provided Thinkpad X1 Carbon with a 14" 1920x1200 screen. My usual distance is nearly double that so my ppd normally is more like 70 ppd, roughly. https://phrogz.net/tmp/ScreenDensityCalculator.html#find:distance,pxW:1920,pxH:1200,size:14,sizeUnit:in,axis:diag,distUnit:in,density:35 https://phrogz.net/tmp/ScreenDensityCalculator.html#find:dis...
• monocasa 11mo ago
  Foveated streaming is wild to me. Saccades are commonly as low as 20-30ms when reading text, so guaranteeing that latency over 2.4Ghz seems Sisyphean. I wonder if they have an ML model doing partial upscaling until the eyetracking state is propagated and the full resolution image under the new fovea position is available. It also makes me wonder if there's some way to do neural compression of the peripheral vision optimized for a nice balance between peripheral vision and hints in the embedding to allow for nicer upscaling.
  • cube2222 11mo ago
    They're doing it over 6GHz, if I understand correctly, which with a dedicated router gets you to a reasonable latency with reasonable quality even without foveated rendering (with e.g. a Quest 3). With foveated rendering I expect this to be a breeze.
    • monocasa 11mo ago
      Even 5.8Ghz is getting congested. There's a dedicated router in this case (a USB fob), but you still have to share spectrum with the other devices. And at the 160Mhz symbol rate mode on WiFi6, you only have one channel in the 5.8GHz spectrum that needs to be shared.
      • esseph 11mo ago
        Not so much of an issue when neighbors with paper thin walls see that 6ghz as a -87 signal That said, in the US it is 1200MHz aka 5.925 GHz to 7.125 GHz.
        • monocasa 11mo ago
          More of an issue when your phone's wifi or your partner watching a show while you game is eating into that one channel in bursts, particularly since the dedicated fob means that it's essentially another network conflicting with the regular WiFI rather than deeply collaborating for better real time guarantees (not that arbitrary wifi routers would even support real time scheduling). MIMO helps here to separate the spectrum use by targeted physical location, but it's not perfect by any means.
• cedws 11mo ago
  Why hasn't Meta tried this given the huge amount of R&D they've put into VR and they had literally John Carmack on the team in the past?
  • modeless 11mo ago
    They prioritized cost, so they omitted eye tracking hardware. They've also bet more on standalone apps rather than streaming from a PC. These are reasonable tradeoffs. The next Quest may add eye tracking, who knows. Quest Pro had it but was discontinued for being too expensive. We'll have to wait on pricing for Steam Frame, but I don't expect them to match Meta's subsidies, so I'm betting on this being more expensive than Quest. I also think that streaming from a gaming PC will remain more of a niche thing despite Valve's focus on it here, and people will find a lot of use for the x86/Windows emulation feature to play games from their Steam library directly on the headset.
    • robotnikman 11mo ago
      It will be interesting to see how the X86 emulation plays out. In the Verge review of the headset they mentioned stutters when playing on the headset due to having to 'recompile x86 game code on the fly', but they may offer precompiled versions which can be downloaded ahead of time, similar to the precompiled shaders the Steam Deck downloads. If they get everything working well I'm guessing we could see an ARM powered Steam Deck in the future. Despite the fact it uses a Qualcomm chip, I'm curious on whether it retains the ability to load alternative OS's like other Steam hardware.
      • girvo 11mo ago
        > Despite the fact it uses a Qualcomm chip, I'm curious on whether it retains the ability to load alternative OS's like other Steam hardware. I think it should: we have Linux support/custom operating systems on Snapdragon 8 Gen 2 devices right now today, and the 8 Gen 3 has upstream support already AFAIK https://rocknix.org/devices/ayn/odin2/ https://rocknix.org/devices/ayn/odin2/
  • jayd16 11mo ago
• MetaWhirledPeas 11mo ago
  > Roughly equivalent resolution to Quest 3 and less than Vision Pro. This won't be suitable as a monitor replacement for general desktop use. The real limiting factor is more likely to be having a large headset on your face for an extended period of time, combined with a battery that isn't meant for all-day use. The resolution is fine. We went decades with low resolution monitors. Just zoom in or bring it closer.
  • krzyk 11mo ago
    2k X 2k doesn't sound low res it is like full HD, but with twice vertical. My monitor is 1080p. Never tried VR set, so I don't know if that translates similarly.
    • deleted 11mo ago
      [deleted]
    • potatolicious 11mo ago
      Your 2K monitor occupies something like a 20-degree field of view from a normal sitting position/distance. The 2K resolution in a VR headset covers the entire field of view. So effectively your 1080p monitor has ~6x the pixel density of the VR headset.
      • krzyk 11mo ago
        Thank you for explaining, it makes sense now.
    • rtkwe 11mo ago
      The problem is that 2k square is spread across the whole FOV of the headset so when it's replicating a monitor unless it's ridiculously close to your face a lot of those pixels are 'wasted' in comparison to a monitor with similar stats.
      • MetaWhirledPeas 11mo ago
        Totally true, but unlike a real monitor you can drag a virtual monitor close to your face without changing the focal distance, meaning it's no harder on your eyes. (Although it is harder on your neck.)
• dagmx 11mo ago
  Foveated streaming is cool. FWIW the Vision Pro does that for their Mac virtual display as well, and it works really well to pump a lot more pixels through.
  • eptcyka 11mo ago
    I think it works really well to pump the same amount of pixels, just focusing them on the more important parts.
    • Psillisp 11mo ago
      Always PIP, Pump Important Pixels
    • dagmx 11mo ago
      It lets you pump more pixels in a given bandwidth window. People are conflating rendering (which is not what I’m talking about) with transmission (which is what I’m talking about). Lowering the quality outside the in focus sections lets them reduce the encoding time and bandwidth required to transmit the frame over.
  • anvuong 11mo ago
    It's the same amount of pixels though, just with reduced bitrate for unfocused regions so you save time in encoding, transmitting, and decoding, essentially reducing latency. For foveated rendering, the amount of rendered pixels are actually reduced.
    • entropicdrifter 11mo ago
      That depends on the specifics of the encode/decode pipeline for the streamed frames. Could be the blurry part actually is lower res and lower bitrate until it's decoded, then upscaled and put together with the high res part. I'm not saying they do that, but it's an option.
    • dagmx 11mo ago
      It’s the same number of pixels rendered but it lets you reduce the amount of data sent , thereby allowing you to send more pixels than you would have been able to otherwise
• ch4s3 11mo ago
  > Mouth tracking? What a vile thought in the context of the steam… catalogue.
  • rtkwe 11mo ago
    They're probably thinking of it in comparison to the Apple Pro which attempts to do some facial tracking of the bottom of your face to inform their 'Personas', it notably still fails quite badly on bearded people where it can't see the bottom half of the face well.
    • ch4s3 11mo ago
      I gathered as much, but still.
  • SchemaLoad 11mo ago
    I'm guessing it's main use case will be VR chat syncing mouths to avatars.
    • riskable 11mo ago
      The porn industry disagrees.
      • Andrex 11mo ago
        If the porn industry likes it, it's bad? Guess we have to get rid of physical home media. And the internet. It was a good run I guess.
  • willis936 11mo ago
    Funny enough the Digital Foundry folks put a Gabe quote about tongue input in their most recent podcast. https://www.youtube.com/watch?v=c9zfExb5vCU&t=1h32m44s https://www.youtube.com/watch?v=c9zfExb5vCU&t=1h32m44s
• nabakin 11mo ago
  And foveated streaming has a 1-2ms wireless latency on modern GPUs according to LTT. Insane.
  • CobrastanJorji 11mo ago
    Wow, that's just 1 frame of latency at 60 fps. Edit: Nevermind, I'm dumb. 1/60th of a second is 16 milliseconds, not 1.6 milliseconds.
    • bspammer 11mo ago
      Much less than, 1 frame is 16ms
    • sph 11mo ago
      60 fps is 16.67 ms per frame.
    • redrblackr 11mo ago
      No, thats between 0.06 and 0.12 frame latency on 60fps. It's not even a frame on 144Hz (1s/144≈7ms)
  • tshaddox 11mo ago
    That's pretty quick. I've heard that in ideal circumstances Wi-Fi 6 can get close to 5ms and Wi-Fi 7 can get down to 2ms. I's impressive if they're really able to get below 2ms motion-to-photon latency, given that modern consumer headsets with on-device compute are also right at that same 2ms mark.
• archon810 11mo ago
  Have a look at this video by Dave2D. In his hands-on, he was very impressed with foveated streaming https://youtu.be/356rZ8IBCps https://youtu.be/356rZ8IBCps.
• JeremyNT 11mo ago
  I guess there's a market for this but I'm personally disappointed that they've gone with the "cram a computer into the headset" route. I'd much rather have a simpler, more compact dumb device like the Bigscreen Beyond 2, which in exchange should prove much lighter and more comfortable to wear for long time periods. The bulk and added component cost of the "all in one" PC/headset models is just unnecessary if you already have a gaming PC.
  • modeless 11mo ago
    You can get a Beyond if that's what you want. It's an amazing device, and will be far more comfortable and higher resolution than this one. Valve has supported Bigscreen in integrating Lighthouse tracking, and I hope that they continue that support by somehow allowing them to integrate the inside-out tracking they've developed for this device in the next version of the Beyond.
    • preisschild 11mo ago
      That would probably add a lot of extra weight and it would need to make the device bigger.
      • modeless 11mo ago
        I don't think it would be too bad. Cameras are tiny. The processing would still happen on the PC, and you could delete the lighthouse tracking sensors. I guess the hardest part would be sending that much camera data back to the PC over the cable.
        • zonkerdonker 11mo ago
          Its worse than that. Cameras (plus DToF if you want tracking in the dark), imus, gyros, and necessarily onboard compute/SOC to handle processing that data. Shipping it all off to a remote computer and then making the round trip creates an untenable amount of lag. Thats not even accounting for controller and hand tracking. And once you have the pipeline and computation power to enable inside out tracking all on device, adding an OS is essentially free.
• bigiain 11mo ago
  > Foveated streaming! That's a great idea. Back when I was in Uni, so late 80s or early 90s, my dad was Project Manager on an Air Force project for a new F-111 flight simulator, when Australia upgraded the avionics on their F-111 fighter/bombers. The sim cockpit had a spherical dome screen and a pair of Silicon Graphics Reality Engines. One of them projected an image across the entire screen at a relatively low resolution. The other projector was on a turret that pan/tilted with the pilot's helmet, and projected a high resolution image but only in a perhaps 1.5m circle directly in from of where the helmet was aimed. It was super fun being the project manager's kid, and getting to "play with it" on weekends sometimes. You could see what was happening while wearing the helmet and sitting in the seat if you tried - mostly ny intentionally pointing your eyes in a different direction to your head - but when you were "flying around" it was totally believable, and it _looked_ like everything was high resolution. It was also fun watching other people fly it, and being able to see where they were looking, and where they weren't looking and the enemy was speaking up on them.
  • zeroq 11mo ago
    I'll share a childhood story as well. Somewhere between '93 and '95 my father took me abroad to Germany and we visited a gaming venue. It was packed with typical arcade machines, games where you sit in a cart holding a pistol and you shoot things on the screen while cart was moving all over the place simulating bumpy ride, etc. But the highlight was a full 3D experience shooter. You got yourself into a tiny ring, 3D headset and a single puck hold in hand. Rotate the puck and you move. Push the button and you shoot. Look around with your head. Most memorable part - you could duck to avoid shots! Game itself, as I remember it, was full wireframe, akin to Q3DM17 (the longest yard) minus jump pads, but the layout was kind of similar. Player was holding a dart gun - you had a single shot and you had to wait until the projectile decayed or connected with other player. I'm not entirely sure if the game was multiplayer or not. I often come back to that memory because shortly after within that time frame my father took me to a computer fair where I had the opportunity to play doom/hexen with VFX1 (or whatever it was called) and it was supposed to revolutionize the world the way AI is suppose to do it now. Then there was a P5 glove with jaw dropping demo videos of endless possibilities of 3D modelling with your hands, navigating a mech like you were actually inside, etc. It never came.
• regularfry 11mo ago
  I use a 1920x1080 headset as a monitor replacement. It's absolutely fine. 2160x2160 will be more than workable as long as the tracking is on point.
• nixpulvis 11mo ago
  I once lived in a place that had a bathroom with mirrors that faced each other. I think I convinced myself that not only is my attention to detail more concentrated at the center, but that my response time was also fastest there (can anyone confirm that?). So this gets me thinking. What would it feel like to correct for that effect? Could you use the same technique to essentially play the further parts early, so it all comes in at once? Kinda a hair brained idea, I know, but we have the technology, and I'm curious.
  • TheOtherHobbes 11mo ago
    Peripheral vision is extremely good at spotting movement at low resolution and moving the eye to look at it. I don't know if it's faster, but it's a non-trivial part of the experience.
    • nixpulvis 11mo ago
      Yea, I've heard and noticed that as well (thought about adding a note about it to my original comment). But what I'm curious about is the timing. What I suspect is that peripherals are more sensitive to motion, but still lag slightly behind the center of focus. I'm not sure if it's dependent on how actively you are trying to focus. I'd love to learn more about this, but I didn't find anything when I looked online a bit.
    • consp 11mo ago
      It's good enough to see flickering on crt monitors at 50-60hz for some people.
      • yencabulator 11mo ago
        I can see the spinning color wheels inside cheaper projectors as rapidly-changing rainbow lights leaking out of their ventilation grilles, but only with peripheral vision and mostly only if I'm moving my head at the same time.
• dyauspitr 11mo ago
  How the hell would foveated streaming even work, it seems physically impossible. Tracking where your eye is looking then sending that information to a server, it processing it and then streaming that back seems impossible.
  • utopiah 11mo ago
    Eye tracking hardware and software specifically focus on low latency, e.g. FPGA close to the sensor. The resulting packets they send is also ridiculously small (e.g 2 numbers as x,y positions of the pupils) so ... I can see that happening. Sure eyes move very VERY fast but if you do relatively small compute on dedicated hardware it can also go quite fast while remaining affordable.
  • imtringued 11mo ago
    It just needs to be less impossible than not doing it. I.e. sending a full frame of information must be an even more impossible problem.
  • LarsDu88 11mo ago
    The data you're sending out is just the position and motion vectors of the pupils. And you probably only need about 16 bits for each of these numbers for 2 eyes. So the equivalent of two floating point numbers along a particular channel or 32 bits at minimum. Any lag can be compensated for by simply interpolating the motion vectors. It actually makes a lot of sense!
• KolibriFly 11mo ago
  Yet this is shaping up to be one of the most interesting VR releases
• dspillett 11mo ago
  > Foveated streaming! That's a great idea. It would be interesting to see⁰ how that behaves when presented with weird eyes like mine or worse. Mine often don't always point the same way and which one I'm actually looking through can be somewhat arbitrary from one moment to the next… Though the flapping between eyes is usually in the presence of changes, however minor, in required focal distance, so maybe it wouldn't happen as much inside a VR headset. ---- [0] Sorry not sorry.
• jagermo 11mo ago
  > Very cool, should be very hackable. Very unlike every other standalone VR headset. That might be the reason I'm going to buy it. I want to support this and Steam has done a lot to get gaming on linux going.
• bob1029 11mo ago
  > 2160 x 2160 LCD (per eye) 72-144Hz refresh rate I question that we could not create a special purpose video codec that handles this without trickery. The "per eye" part sounds spooky at first, but how much information is typically different between these frames? The mutual information is probably 90%+ in most VR games. If we were to enhance something like x264 to encode the 2nd display as a residual of the 1st display, this could become much more feasible from a channel capacity standpoint. Video codecs already employ a lot of tricks to make adjacent frames that are nearly identical occupy negligible space. This seems very similar (identical?) to the problem of efficiently encoding a 3d movie: https://en.wikipedia.org/wiki/2D_plus_Delta https://en.wikipedia.org/wiki/2D_plus_Delta https://en.wikipedia.org/wiki/Multiview_Video_Coding https://en.wikipedia.org/wiki/Multiview_Video_Coding
  • starkrights 11mo ago
    I'm entirely unfamiliar with the vr rendering space, so all I have to go on is what (I think) your comment implies. Is the current state of VR rendering really just rendering and transporting two videostreams independent of eachother? Surely there has to be at least some academic prior-art on the subject, no?
• chaostheory 11mo ago
  > But the price is hopefully low. The main value of Meta VR and AR products is the massive price subsidy which is needed because the brand has been destroyed for all generations older than Alpha. The current price estimate for the Steam Frame is $1200 vs Quest 3 at $600 which is still a very reasonable price given the technology, tariffs, and lack of ad invading privacy
  • coolspot 11mo ago
    Quest 3 is $499 and Quest 3S is $299 in the US