38 ms·

Behind the Tech with John Carmack: 5k Immersive Video

395 points by ot 8y ago

MikusR 8y ago
Not mentioned in article, but Exynos versions of s8/9 don't need these tricks.
- JohnCarmack 8y ago
  No, Exynos has the same block limit as the snapdragon chips -- 4k60. The difference is that Exynos doesn't have the same 4096 maximum dimension limit, so it can do 5120x2560 (monoscopic) at 30 fps, while snapdragon can only decode 4096x2560 at 30 fps. The view dependent player is about playing 5120x5120 (stereo) at 60 fps.
  - MikusR 8y ago
    It can play a HEVC file that is 6144x3072 at 60fps.
    - JohnCarmack 8y ago
      I just tried, and while it does decode a 6kx3k 60 fps video, which is very admirable, it doesn't hold 60 fps while doing it. There are probably encoding options to minimize work on the decoder that could let you push it a bit more. MediaExtractor seems to be arbitrarily limited to a lower resolution, but that can be bypassed.
      - MikusR 8y ago
        Can you try with skybox vr? At least on my note 8 Oculus video really doesn't like h265 video that's more than 4k.
- jbk 8y ago
  I agree with you. VLC on the Note 8 (and the S8) can do 8K decoding at 48fps, as is demoed here: https://vimeo.com/254723180 https://vimeo.com/254723180 We've spent quite a bit of time to get that right, though, and those are rare devices (and does not work with the Qcom version). Maybe that's one video, with low bitrate (5Mbps), but it worked.
Retric 8y ago
"5120 x 5120 at 60 fps" here is why that's overkill. If you start putting dots on a sphere you can go in one direction turn 90 degrees, put 5120 another axis. However, if you tile like that with 5120 x 5120 you get a lot of wasted pixels at the poles. If a 5120 pixel Radius is fine... then Radius of sphere = 2 pi * r, and Surface of sphere is 4 pi * r^2. So 5120 / 2pi =r substitute for r > 4 * (5120/2pi)^2 simplify > (5120)^2 /(pi) ~= we need ~1/3 aka 1/pi of 5280^2. However, I suspect you actually want more than 5120 pixel radius.
- cma 8y ago
  > If you start putting dots on a sphere you can go in one direction turn 90 degrees, Starting where? Turning where? > However, if you tile like that with 5120 x 5120 you get a lot of wasted pixels at the poles. You didn't mention tiling. > If a 5120 pixel Radius is fine... then Radius of sphere = 2 pi * r, and Surface of sphere is 4/3 pi * r^2. so 5120 / 2pi =r => 4/3 * (5120/2pi)^2 => (5120)^2 /(3 pi) ~= we need ~10% of 5280^2. More details on what you are trying to say at each step please.
  - Retric 8y ago
    Quick edit, it's 4pi r I was thinking volume for a second there. As to tiling, I am saying if cut a sphere into 5120 slices, and the middle slice is fine with 5120 pixels. Then the poles are going to have ~1 pixel on them. How you tile them is really a question of tradeoffs.
    - cma 8y ago
      I think you should draw a diagram and link it. > As to tiling, I am saying if cut a sphere into 5120 slices, and the middle slice is fine with 5120 pixels. What kind of slices? Like sections of an orange? Like longitude?
      - Retric 8y ago
        I was thinking lines of latitude.
seanalltogether 8y ago
I always wondered if HTM spheres are a worthwhile solution for lower sized textures. You start with 8 equilateral triangles that make up the sphere and subdivide as needed. You could possibly use a square texture made of 8 right triangles that each need minimal distortion (as opposed to an equirectangular texture). https://arxiv.org/ftp/cs/papers/0701/0701164.pdf https://arxiv.org/ftp/cs/papers/0701/0701164.pdf
- psurge 8y ago
  The WorldWide Telescope uses this exact idea: http://www.worldwidetelescope.org/docs/worldwidetelescopeprojectionreference.html#TOASTProjection http://www.worldwidetelescope.org/docs/worldwidetelescopepro... There are also other interesting spherical tessellations that could be used - one is implemented in Google's S2 geometry library, the other, HEALPix, is used for astronomical datasets: http://s2geometry.io/resources/earthcube http://s2geometry.io/resources/earthcube http://iopscience.iop.org/article/10.1086/427976/pdf http://iopscience.iop.org/article/10.1086/427976/pdf
- Asooka 8y ago
  And you could use ptex textures to have a regular pixel density.
- FreeFull 8y ago
  Alternatively, you could maybe use a cube map. You would still have a bit of distortion, but that could be handled during the video generation anyway.
  - samplatt 8y ago
    Distortion visible in cube maps goes doubly so when viewed in VR. Cube maps are easy but not optimal.
    - slavik81 8y ago
      What's the source of the distortion?
mrybczyn 8y ago
Retina has 100 million photoreceptors per eye. Also, need higher FPS. Maybe 120 is good enough?
- anonytrary 8y ago
  Correct me if I'm wrong, but isn't the difference between 60Hz and 120Hz refresh more or less imperceptible to the average human? I'm sure there's a distribution, but I'd be hard pressed to find a person who could differentiate between 100Hz and 120Hz refresh. It seems like a waste to push rendering beyond the point at which we can even tell there's a difference. Edit: Thanks for the feedback, I guess it is perceptible. Nevertheless, I think my argument becomes valid at some N. Sure, N !== 60, but N = 144 or 120 may be more reasonable. I'm not too concerned with what N is, more so with the fact that "doubling the refresh rate" eventually becomes an act of futility.
  - seanmcdirmid 8y ago
    It is definitely perceptible. Compare a normal iPad (60 Hz) to an iPad Pro (120 Hz), the fluidity of movement is very apparent in just playing around with the home screens.
    - deleted 8y ago
      [deleted]
    - conradev 8y ago
      Yeah, it's night and day.
    - kbenson 8y ago
      It may be perceptible, but the number of factors that could affect the different systems in that comparison means it's not exactly a good test. At the simplest level, there's no guarantee it's actually even rendering updated frames at the rates in question if it's limited by some other factor, and the differing hardware may change at what point that limit is hit.
pcnix 8y ago
With John Carmack at the problem, this is definitely something I'll be keeping a close eye on. For people that are unaware, he's one of the original creators of Doom, and his innovations in computer graphics are legendary.
- ubertakter 8y ago
  Not disparaging, but how likely is it that someone doesn't know who John Carmack and reads Hacker News? Surely the Venn diagram of those two sets doesn't have much overlap...
  - sp332 8y ago
    You're probably underestimating how popular HN is. And how old you are (ducks) But seriously it's 2018! It's the distant future where Doom is run in emulators in browser windows.
- dokem 8y ago
  What's a doom?
  - egypturnash 8y ago
    One of the first games to do real time texture mapping on fairly arbitrary 3D surfaces, back in 1993 when you had to do it all on the CPU because 3D graphics cards didn’t exist except as super-expensive prototypes found in military flight simulators and the like. Full of clever tricks to make this something you could do in real time on a 486. It was influential enough that for many years, the genre we now call “first person shooters” was known as “Doom clones”.
    - kleiba 8y ago
      What exactly do you mean by "games"?
      - egypturnash 8y ago
Daemon404 8y ago
The slice approach should be pretty readily doable today with e.g. libx264 (which you can force exactly N rectangular (full-row) slices with, using i_slice_count), and calculating N based on the resolution of the eleven vertically and horziontally stitched clips and their boundaries. (And with VP9 using some craft tile-column/tile-rows setup, maybe...) This is assuming the videos are stitched pre-encode, of course... From the post, it almost sounds as if the idea would be to stitch independent H.264 streams into a new unified one using slice mangling + slices... which would be pretty crazy stuff. (As a side note, it's a shame flexible macroblock ordering is only in the baseline and extended profiles... I still don't understand that decision at all.) EDIT: Dawned on me that the hard part is on the client/decode side. D'oh.
v8engine 8y ago
It mentions "...5120 pixels across 360 degrees is a good approximation. Anyone over this on current VR headsets is simply wasting resources and inviting aliasing." How does increasing the resolution cause aliasing? Shouldn't it be the opposite?
- AboutTheWhisles 8y ago
  He did not say it causes aliasing, he said it is inviting aliasing. More resolution is going to be fine if the pixels are filtered well. If they aren't, that could certainly invite aliasing. Pixel filtering, especially in a scenario like this, is going to have a wide spectrum of techniques that can work to varying degrees, with trade offs of quality and speed.
  - gmueckl 8y ago
    A pixel filter that takes into account the highly nonregular shape of the projected screen pixels and the warped projected video pixels in the source at the same time would be quite an interesting task.
  - jpambrun 8y ago
    I think the point was mostly if you are filtering to downsample, you should probably not have downloaded the bytes or have wasted the energy decoding.
    - AboutTheWhisles 8y ago
      The question I was replying to was about aliasing.
- gmueckl 8y ago
  Aliasing happens only when the signal that you are sampling has higher frequencies than what you can represent in your sampled data. So he is absolutely correct here: if the source material has higher resolution, the projection onto the lower resolution display would lead to the same kind of undersampling. Basically, you'd miss in-between pixels and if these have significant content, it becomes very noticeable (e.g. parts of very thin features missing from the image).
rasz 8y ago
I am somewhat shocked Oculus is wasting Carmacks time and potential on something nobody wants (watching 360 panorama movies).
- tener 8y ago
  I want it. Also looks like prerequisite for other fancy stuff.
- sturmen 8y ago
  I like 360 panorama movies. Done right, the feeling of presence is excellent. VR is a niche to begin with, so I would hesitate to say that any one application (ex. CAD, gaming, 360 panoramas, etc) is the "killer app" so far since the audience is so small. As it grows (which I pray it will), we'll see definitive trends emerge.
  - GordonS 8y ago
    > VR is a niche to begin with I would prefix that with "in its current state".
    - criddell 8y ago
      It's been a niche thing for 30 years or so. But any day now...
  - white-flame 8y ago
    The feeling of presence is so limited compared to realtime-rendered 3d spaces where you can move your head around from the central camera point even just a bit. Adding some artificial parallax shift to the movie frames might be enough to give it that extra oomph to feel truly immersive.
    - sp332 8y ago
      The Oculus Go doesn't have 6 DoF, and he's been working on that and phone-based VR for the last few years. Because they're cheaper, they move a lot more units! And because they're more constrained, they need more optimization attention from someone like Carmack.
jokoon 8y ago
He talked about equirectangular projection. I wonder how he is achieving that, with what kind of shader, I asked the question once, and it seems you cannot do that with a vertex shader, but maybe a geometry shader? I already read articles about the pannini projection, which is quite cool, but I guess occulus is facing the same issues and has similar solutions...
- Const-me 8y ago
  Any shader. You can render a single quad and calculate UV in the pixel shader. Geometrically, this is the most accurate method. You can do without any shaders, write a CPU code that builds a reasonably dense (about 8x8 pixels/triangle) static geosphere mesh with position and texcoord channels. You can also build a reasonably dense grid mesh occupying just the screen with just the positions in clip space, and calculate UV channel in the VS based on the rotation. GS will work too but I’m not sure it makes sense performance-wise. People normally use GS-es to implement small dynamic objects. For static geometry I won’t be surprised an immutable buffer will be faster, just because that’s what hardware & drivers are optimized for. See e.g. this http://www.joshbarczak.com/blog/?p=667 http://www.joshbarczak.com/blog/?p=667
- undershirt 8y ago
  These are the steps for transformation that may answer your questions: 1. Cubemap (source) - 360° videos are often sourced from six different images stitched together. 2. Equirect (video) - the cubemap is transformed to "equirect" trivially with a fragment shader—which maps every output pixel to some input pixel. 3. Perspective (output) - Mapping equirect video to a perspective projection is also done with a fragment shader, just using a different transformation depending on the focal point of the user. 4. Pannini (alternative output) - You mentioned this projection, and it's an alternative to the "perspective" projection that allows a wider output FOV that minimizes distortion in the periphery.
  - slavik81 8y ago
    The transformation from cubemap to equirectangular projection seems unnecessary. Why convert? There's support for cubemap textures on modern hardware, and having a seperate video stream per face would make culling them easy. It wouldn't be as fine-grained as Carmack's end result, but it would be simple and avoids resampling.
masonicb00m 8y ago
John Carmack's attention to quality and detail is incredibly inspiring.
vesche 8y ago
Finally a John Carmack post that isn't on Facebook! Great read too, can't wait to see more.
- goldenSilence 8y ago
  I think the distiction is between personal experiences, and activity on projects, requiring review, to determine the disclosure of information which may or may not reveal trade secrets.
modeless 8y ago
Do you feather the edges of the high resolution videos to blend with the low resolution background? I found this necessary when I implemented something similar because otherwise the borders are far too obvious, though it feels bad to waste those pixels at the edge of the video. How are the videos stored? Is each gop a separate file or do you seek around in larger files? Is it feasible to stream over HTTP or is it only possible to play local videos for now?
- JohnCarmack 8y ago
  I was prepared to blend the edges, but it turned out not to be necessary. If the compression ration was increased enough that there were lots of artifacts in the low res version it might be more important. I was originally going to put it into an mp4 file with the base stored first, so normal video players could at least play the low res version, but the Android MediaExtractor fails when presented with more than 10 tracks, so I just rolled my own trivial container file. Peak bitrate for Henry is around 40 Mbps, so it wouldn't stream for most people. With some rearrangement of the file so each strip has a full gop continuous, instead of time interleaving all 11, the bitrate wold be cut in half, but it would still be a lot of fairly small requests, so it would call for pipelined HTTP2.
  - modeless 8y ago
    Ah, so all the frames for every strip are interleaved in your container, and you just read sequentially and ignore the frames you don't need? That's probably the right thing for local videos. I had each gop in a separate file like HLS or DASH (except for the background which was a single file also containing the audio track). It's unwieldy but makes HTTP streaming a little simpler because you don't need range requests or an index. Also, instead of bitstream hacking to stitch three strips into one, I encoded multiple strips into "pre-stitched" views. This means that every strip is encoded redundantly in multiple views, bloating the on-disk video size. But for streaming that only affects the server, not the client, and it's nice for the client to only download/decode one view at a time (plus the background) instead of three strips. Bitstream hacking to join the strips would definitely be better if it can work, though.
sigi45 8y ago
If you have the keyframe always at the seek time, would that be enough? Because you said the delay is too long? If yes, would it be possible to render, for every second (half second or whatever resolution you like) a snippet which only contains the seek time until the next keyframe and than tell the webserver 'if someone seeks to time x, take this snippet and when done, jump to the original video'? You could do that transparent with some fuse fs.
iamleppert 8y ago
Another approach to space efficiency would be to do away with the need for dual video streams entirely and just average the stereo images together to form a single monocular image. Then, send a disparity map along with the monocular video. Decode the mono video and use the disparity map to interpolate the view of either eye. You’ll have all the information you need for reconstruction and the disparity map can be efficiently compressed via normalization and perhaps even by sending just the vectors of the contours. Another idea is to take advantage of the fact head motions are really just a translation vector of the camera. There’s no need to send pixels that have just transposed locations unless they have changed in time. If I was designing such a system I’d try to take advantage of the fact there isn’t a lot changing fundamentally in the scene when you move your head, and maintain some sort of state and only request chunks of pixels that are actually needed. You wouldn’t even have to use a traditional video codec as the preservation of state would be far more efficient than thinking about things in terms of flat pixels and video.
- munificent 8y ago
  > Decode the mono video and use the disparity map to interpolate the view of either eye. By "disparity map" are you thinking something like a heightmap applied to the scene facing the viewer and then you use that to skew things for each eye? If so, how would that handle parts of the scene that are occluded/revealed to one eye but not the other?
  - iamleppert 8y ago
    True, occlusions would be a problem but we’re taking about fake autostereoscopic 3D here, where most of the stereo rigs used for capture have but a modest baseline. Almost all of the depth perception comes from disparity, occlusions would still in fact be very visible by the averaging method I described and be at whatever depth plane of the occluder which is probably a good guess anyway. Not like your other eye would receive a correspondence from an occlusion in the real world.
    - romwell 8y ago
DanielBMarkham 8y ago
(Much hand-waving here) I wonder if the problem isn't a misaligned paradigm for just what a codec does. Right now, codecs exist to take bytestreams and fill frames at a certain rate. They're a bitstream-to-framerate device. What if instead of delivering to a frame during a certain time period, the codec had to instantly deliver whatever it could, but only geared to the actual acuity of the users retina. Seems like there would be less information, you would never have lag, and all optimization would be around the detail of data delivered, not framerate or screensize. I also birthday parties and Bar Mitzvahs, folks. I'm here all week. (I figure it couldn't hurt to throw crazy ideas out there. Every now and then a crazy idea actually amounts to something. Coding is cool because it not only let you solve problems, sometimes it lets you change the universe the problem lives in. Good luck, John!)
- undershirt 8y ago
  only delivering the visible portion of a 360° video assumes that the video is ready to be decoded at any angle, which is the entire problem.
  - DanielBMarkham 8y ago
    But the problem is to display something for a moving angle, not display something for everything. As best I can tell, it is a different problem. I want to restate what I'm hearing so you can correct me. You are saying something like "Gee, if we could show any retinally-matched conic the user's pointing their eyeball at? To do that we'd have to have the entire scene rendered anyway" I'm saying you can't do it now. It hurts when I do that. So stop doing that. Instead of trying to render stereo scenes quicker, try to render a moving conic real-time quicker. When the codec runs, it's optimizing areas of a frame changing over time. If instead it optimized possible vision movement paths, you'd end up solving the framerate and resolution problems. Then you could concentrate on optimizing the codec in a different way than people are currently trying to optimize it. It couldn't hurt, and there may be opportunities for consolidation if you look at the problem as visual-movement-path-rendering instead of frame rendering. I don't know. I just know it's a problem now. Set your constraints differently and optimize along a different line. Sometimes that works. Does that explain the differences here? Or do you want me to start spec'ing out what I mean by retinal-path codec? At some point this will a bit over the top. ADD: I'll say it a little differently. Our constraint now is "how fast can you render this frame". Codecs are built for rendering frames on screens where people watch movies and play games. But that's not the world we're trying to solve a problem in. They may be configured the wrong way. Instead, require that anything the eyeball is looking at should render consistently in, say 5ms. This sounds like the same thing, until you realize that the eyeball can't look at everything at the same time. Different parts of the image are temporally separated. So if I update the image in back of your head once every 200ms? You're not going to know. As far you're concerned, it's all instant. It becomes a different kind of problem.
mattbierner 8y ago
Fascinating. Ammeture question though: are there other ways to encode the images besides pixels that will ultimately be better suited to 360 content? Holograms came to my mind because they degrade gracefully as you toss out bits of the signal
- samplatt 8y ago
  Lytro's light-field camera technologies have huge potential for 360 content and the photogrammetry industry, but they effectively sat on the technology doing very little with it (compared to their competitors), until Google hired some of the employees as the company shut down earlier this year.
Animats 8y ago
I dunno. Looking at canned content with VR goggles seems like a dead end. It's 3D TV on steroids. Remember 3D TV? Works fine. Too much trouble just to watch TV. Market failure. It's been five years since the Oculus Rift DK1 appeared, and there's still no killer app. Game developers are pulling back from VR.[1][2] The VR virtual worlds are a disaster. Sansar has about 50 (fifty) concurrent users. SineSpace and High Fidelity have similar numbers. VR goggles are cool for about an hour, and in the closet in a month. The technology is coming along just fine, but that's not the problem. [1] https://www.gq.com/story/is-vr-gaming-over-before-it-even-started https://www.gq.com/story/is-vr-gaming-over-before-it-even-st... [2] https://mashable.com/2018/01/24/virtual-reality-gaming-loser-gdc-2018-survey/#WG.n6Sqwp5qm https://mashable.com/2018/01/24/virtual-reality-gaming-loser...
- samplatt 8y ago
  You can keep believing that, sure. I'll just be over here, continuing to earn my living delivering interactive 360 video and VR demonstrations for our clients :-)
  - wyldfire 8y ago
    Is the video still produced as ordinary 2d/planar video? It's just projected as 360? Or is there some kind of depth information? What makes it so compelling?
    - samplatt 8y ago
      Most of our video is rendered down to ordinary video that gets reprojected, driven mostly by a requirement to be able to be played back on lowest-common-denominator hardware and platforms; nothing near as cool as what Carmack's article details. But there's other problems that require fun solutions to, too.
      - brennebeck 8y ago
        I’m curious: what company/industry are you with and what tech are you using specifically?
Njkl 8y ago
This might not be possible if the "system software" doesn't cooperate, but it's possible to encode videos such that you can keep them "warm" without decoding all frames, for example in a IBBPBBPBBPBB structure where all B-frames are not referenced by any other frame (other arrangements are possible). Forcing this structure has a cost, but it's much smaller than having more I-frames. You can then alternate decoding 3 such streams (each one offset by 1 frame, including the I-frames - this is not a problem, is just means you'll not be ready to output anything for 2 frames after a seek) for the cost of 1. Switching to 60fps is then "instantaneous". Old iTunes used to code h.264 video like this (with a PBPBPB structure, so it could play at half-rate, which it did if the CPU couldn't keep up). Note unreferenced does not imply B-frame nor the other way around. Another (admittedly crazy) idea, for a setup with a lower-res version and a higher-res overlay, is trying to store the difference only, affording a (significant?) bitrate reduction for the high res "patches". This is very tricky to do in practice, though (needs larger range or losing the lsb; the codecs aren't really designed for this). I don't think it has ever been done.
Ono-Sendai 8y ago
> Directly texture mapping a video in VR gives you linear filtering, not the sRGB filtering you want. Isn't this back-to-front? Generally filtering is considered to work better in a linear colour space. Using a sRGB texture will convert each pixel to a linear colour space, before the reconstruction filtering is done, AFAIK.
Maurice_Ribble 8y ago
I find it interesting that you went in a different direction than the earlier ideas discussed for this problem. I can see where this is simpler in many ways and probably more importantly a better fit for a lot of content than some of those early ideas. You mentioned the overhead for handling many streams being a problem. Can you go into more detail on the type of overhead you saw and do you feel this is something can can be addressed with better drivers or some application changes? Or do you think the only solution is packing multiple stripes into a single stream using the encoding bit manipulation you mentioned? It seems a shame for hardware that can support more streams leaves them inaccessible to real world applications.
nomercy400 8y ago
You don't need 5120 (for 360 degrees on X-axis) x 5120 (for 360 degrees on Y-axis) do you? If you cover 360 degrees in the X direction, then your Y-axis only has to be 180 degrees, because you don't need 'behind you' on the Y-axis as that is already covered by the X-axis' wide 360 degrees. Or am I making an error here?
- michaelt 8y ago
  It's stereo, so Carmack means 360°x180°x2 for two eyes. One would assume there would be a lot in common between stereo cameras' views, so presumably there are compression efficiencies to be found.
holografix 8y ago
Legitimate question: are there “untouched nature” type simulations for one of the leading VR platforms? Sitting at the top of a mountain, different times of the day and weather. Exploring a huge, very realistic forest. Being a pigeon observing a busy street from above as you fly around and perch yourself on different spots.

395 points by ot 8y ago

MikusR 8y ago

Not mentioned in article, but Exynos versions of s8/9 don't need these tricks.

JohnCarmack 8y ago
No, Exynos has the same block limit as the snapdragon chips -- 4k60. The difference is that Exynos doesn't have the same 4096 maximum dimension limit, so it can do 5120x2560 (monoscopic) at 30 fps, while snapdragon can only decode 4096x2560 at 30 fps. The view dependent player is about playing 5120x5120 (stereo) at 60 fps.
- MikusR 8y ago
  It can play a HEVC file that is 6144x3072 at 60fps.
  - JohnCarmack 8y ago
    I just tried, and while it does decode a 6kx3k 60 fps video, which is very admirable, it doesn't hold 60 fps while doing it. There are probably encoding options to minimize work on the decoder that could let you push it a bit more. MediaExtractor seems to be arbitrarily limited to a lower resolution, but that can be bypassed.
    - MikusR 8y ago
      Can you try with skybox vr? At least on my note 8 Oculus video really doesn't like h265 video that's more than 4k.
jbk 8y ago
I agree with you. VLC on the Note 8 (and the S8) can do 8K decoding at 48fps, as is demoed here: https://vimeo.com/254723180 https://vimeo.com/254723180 We've spent quite a bit of time to get that right, though, and those are rare devices (and does not work with the Qcom version). Maybe that's one video, with low bitrate (5Mbps), but it worked.

Retric 8y ago

"5120 x 5120 at 60 fps" here is why that's overkill. If you start putting dots on a sphere you can go in one direction turn 90 degrees, put 5120 another axis. However, if you tile like that with 5120 x 5120 you get a lot of wasted pixels at the poles. If a 5120 pixel Radius is fine... then Radius of sphere = 2 pi * r, and Surface of sphere is 4 pi * r^2. So 5120 / 2pi =r substitute for r > 4 * (5120/2pi)^2 simplify > (5120)^2 /(pi) ~= we need ~1/3 aka 1/pi of 5280^2. However, I suspect you actually want more than 5120 pixel radius.

cma 8y ago
> If you start putting dots on a sphere you can go in one direction turn 90 degrees, Starting where? Turning where? > However, if you tile like that with 5120 x 5120 you get a lot of wasted pixels at the poles. You didn't mention tiling. > If a 5120 pixel Radius is fine... then Radius of sphere = 2 pi * r, and Surface of sphere is 4/3 pi * r^2. so 5120 / 2pi =r => 4/3 * (5120/2pi)^2 => (5120)^2 /(3 pi) ~= we need ~10% of 5280^2. More details on what you are trying to say at each step please.
- Retric 8y ago
  Quick edit, it's 4pi r I was thinking volume for a second there. As to tiling, I am saying if cut a sphere into 5120 slices, and the middle slice is fine with 5120 pixels. Then the poles are going to have ~1 pixel on them. How you tile them is really a question of tradeoffs.
  - cma 8y ago
    I think you should draw a diagram and link it. > As to tiling, I am saying if cut a sphere into 5120 slices, and the middle slice is fine with 5120 pixels. What kind of slices? Like sections of an orange? Like longitude?
    - Retric 8y ago
      I was thinking lines of latitude.

seanalltogether 8y ago

I always wondered if HTM spheres are a worthwhile solution for lower sized textures. You start with 8 equilateral triangles that make up the sphere and subdivide as needed. You could possibly use a square texture made of 8 right triangles that each need minimal distortion (as opposed to an equirectangular texture). https://arxiv.org/ftp/cs/papers/0701/0701164.pdf https://arxiv.org/ftp/cs/papers/0701/0701164.pdf

psurge 8y ago
The WorldWide Telescope uses this exact idea: http://www.worldwidetelescope.org/docs/worldwidetelescopeprojectionreference.html#TOASTProjection http://www.worldwidetelescope.org/docs/worldwidetelescopepro... There are also other interesting spherical tessellations that could be used - one is implemented in Google's S2 geometry library, the other, HEALPix, is used for astronomical datasets: http://s2geometry.io/resources/earthcube http://s2geometry.io/resources/earthcube http://iopscience.iop.org/article/10.1086/427976/pdf http://iopscience.iop.org/article/10.1086/427976/pdf
Asooka 8y ago
And you could use ptex textures to have a regular pixel density.
FreeFull 8y ago
Alternatively, you could maybe use a cube map. You would still have a bit of distortion, but that could be handled during the video generation anyway.
- samplatt 8y ago
  Distortion visible in cube maps goes doubly so when viewed in VR. Cube maps are easy but not optimal.
  - slavik81 8y ago
    What's the source of the distortion?

mrybczyn 8y ago

Retina has 100 million photoreceptors per eye. Also, need higher FPS. Maybe 120 is good enough?

anonytrary 8y ago
Correct me if I'm wrong, but isn't the difference between 60Hz and 120Hz refresh more or less imperceptible to the average human? I'm sure there's a distribution, but I'd be hard pressed to find a person who could differentiate between 100Hz and 120Hz refresh. It seems like a waste to push rendering beyond the point at which we can even tell there's a difference. Edit: Thanks for the feedback, I guess it is perceptible. Nevertheless, I think my argument becomes valid at some N. Sure, N !== 60, but N = 144 or 120 may be more reasonable. I'm not too concerned with what N is, more so with the fact that "doubling the refresh rate" eventually becomes an act of futility.
- seanmcdirmid 8y ago
  It is definitely perceptible. Compare a normal iPad (60 Hz) to an iPad Pro (120 Hz), the fluidity of movement is very apparent in just playing around with the home screens.
  - deleted 8y ago
    [deleted]
  - conradev 8y ago
    Yeah, it's night and day.
  - kbenson 8y ago
    It may be perceptible, but the number of factors that could affect the different systems in that comparison means it's not exactly a good test. At the simplest level, there's no guarantee it's actually even rendering updated frames at the rates in question if it's limited by some other factor, and the differing hardware may change at what point that limit is hit.

pcnix 8y ago

With John Carmack at the problem, this is definitely something I'll be keeping a close eye on. For people that are unaware, he's one of the original creators of Doom, and his innovations in computer graphics are legendary.

ubertakter 8y ago
Not disparaging, but how likely is it that someone doesn't know who John Carmack and reads Hacker News? Surely the Venn diagram of those two sets doesn't have much overlap...
- sp332 8y ago
  You're probably underestimating how popular HN is. And how old you are (ducks) But seriously it's 2018! It's the distant future where Doom is run in emulators in browser windows.
dokem 8y ago
What's a doom?
- egypturnash 8y ago
  One of the first games to do real time texture mapping on fairly arbitrary 3D surfaces, back in 1993 when you had to do it all on the CPU because 3D graphics cards didn’t exist except as super-expensive prototypes found in military flight simulators and the like. Full of clever tricks to make this something you could do in real time on a 486. It was influential enough that for many years, the genre we now call “first person shooters” was known as “Doom clones”.
  - kleiba 8y ago
    What exactly do you mean by "games"?
    - egypturnash 8y ago

Daemon404 8y ago

The slice approach should be pretty readily doable today with e.g. libx264 (which you can force exactly N rectangular (full-row) slices with, using i_slice_count), and calculating N based on the resolution of the eleven vertically and horziontally stitched clips and their boundaries. (And with VP9 using some craft tile-column/tile-rows setup, maybe...) This is assuming the videos are stitched pre-encode, of course... From the post, it almost sounds as if the idea would be to stitch independent H.264 streams into a new unified one using slice mangling + slices... which would be pretty crazy stuff. (As a side note, it's a shame flexible macroblock ordering is only in the baseline and extended profiles... I still don't understand that decision at all.) EDIT: Dawned on me that the hard part is on the client/decode side. D'oh.

v8engine 8y ago

It mentions "...5120 pixels across 360 degrees is a good approximation. Anyone over this on current VR headsets is simply wasting resources and inviting aliasing." How does increasing the resolution cause aliasing? Shouldn't it be the opposite?

AboutTheWhisles 8y ago
He did not say it causes aliasing, he said it is inviting aliasing. More resolution is going to be fine if the pixels are filtered well. If they aren't, that could certainly invite aliasing. Pixel filtering, especially in a scenario like this, is going to have a wide spectrum of techniques that can work to varying degrees, with trade offs of quality and speed.
- gmueckl 8y ago
  A pixel filter that takes into account the highly nonregular shape of the projected screen pixels and the warped projected video pixels in the source at the same time would be quite an interesting task.
- jpambrun 8y ago
  I think the point was mostly if you are filtering to downsample, you should probably not have downloaded the bytes or have wasted the energy decoding.
  - AboutTheWhisles 8y ago
    The question I was replying to was about aliasing.
gmueckl 8y ago
Aliasing happens only when the signal that you are sampling has higher frequencies than what you can represent in your sampled data. So he is absolutely correct here: if the source material has higher resolution, the projection onto the lower resolution display would lead to the same kind of undersampling. Basically, you'd miss in-between pixels and if these have significant content, it becomes very noticeable (e.g. parts of very thin features missing from the image).

rasz 8y ago

I am somewhat shocked Oculus is wasting Carmacks time and potential on something nobody wants (watching 360 panorama movies).

tener 8y ago
I want it. Also looks like prerequisite for other fancy stuff.
sturmen 8y ago
I like 360 panorama movies. Done right, the feeling of presence is excellent. VR is a niche to begin with, so I would hesitate to say that any one application (ex. CAD, gaming, 360 panoramas, etc) is the "killer app" so far since the audience is so small. As it grows (which I pray it will), we'll see definitive trends emerge.
- GordonS 8y ago
  > VR is a niche to begin with I would prefix that with "in its current state".
  - criddell 8y ago
    It's been a niche thing for 30 years or so. But any day now...
- white-flame 8y ago
  The feeling of presence is so limited compared to realtime-rendered 3d spaces where you can move your head around from the central camera point even just a bit. Adding some artificial parallax shift to the movie frames might be enough to give it that extra oomph to feel truly immersive.
  - sp332 8y ago
    The Oculus Go doesn't have 6 DoF, and he's been working on that and phone-based VR for the last few years. Because they're cheaper, they move a lot more units! And because they're more constrained, they need more optimization attention from someone like Carmack.

jokoon 8y ago

He talked about equirectangular projection. I wonder how he is achieving that, with what kind of shader, I asked the question once, and it seems you cannot do that with a vertex shader, but maybe a geometry shader? I already read articles about the pannini projection, which is quite cool, but I guess occulus is facing the same issues and has similar solutions...

Const-me 8y ago
Any shader. You can render a single quad and calculate UV in the pixel shader. Geometrically, this is the most accurate method. You can do without any shaders, write a CPU code that builds a reasonably dense (about 8x8 pixels/triangle) static geosphere mesh with position and texcoord channels. You can also build a reasonably dense grid mesh occupying just the screen with just the positions in clip space, and calculate UV channel in the VS based on the rotation. GS will work too but I’m not sure it makes sense performance-wise. People normally use GS-es to implement small dynamic objects. For static geometry I won’t be surprised an immutable buffer will be faster, just because that’s what hardware & drivers are optimized for. See e.g. this http://www.joshbarczak.com/blog/?p=667 http://www.joshbarczak.com/blog/?p=667
undershirt 8y ago
These are the steps for transformation that may answer your questions: 1. Cubemap (source) - 360° videos are often sourced from six different images stitched together. 2. Equirect (video) - the cubemap is transformed to "equirect" trivially with a fragment shader—which maps every output pixel to some input pixel. 3. Perspective (output) - Mapping equirect video to a perspective projection is also done with a fragment shader, just using a different transformation depending on the focal point of the user. 4. Pannini (alternative output) - You mentioned this projection, and it's an alternative to the "perspective" projection that allows a wider output FOV that minimizes distortion in the periphery.
- slavik81 8y ago
  The transformation from cubemap to equirectangular projection seems unnecessary. Why convert? There's support for cubemap textures on modern hardware, and having a seperate video stream per face would make culling them easy. It wouldn't be as fine-grained as Carmack's end result, but it would be simple and avoids resampling.

masonicb00m 8y ago

John Carmack's attention to quality and detail is incredibly inspiring.

vesche 8y ago

Finally a John Carmack post that isn't on Facebook! Great read too, can't wait to see more.

goldenSilence 8y ago
I think the distiction is between personal experiences, and activity on projects, requiring review, to determine the disclosure of information which may or may not reveal trade secrets.

modeless 8y ago

Do you feather the edges of the high resolution videos to blend with the low resolution background? I found this necessary when I implemented something similar because otherwise the borders are far too obvious, though it feels bad to waste those pixels at the edge of the video. How are the videos stored? Is each gop a separate file or do you seek around in larger files? Is it feasible to stream over HTTP or is it only possible to play local videos for now?

JohnCarmack 8y ago
I was prepared to blend the edges, but it turned out not to be necessary. If the compression ration was increased enough that there were lots of artifacts in the low res version it might be more important. I was originally going to put it into an mp4 file with the base stored first, so normal video players could at least play the low res version, but the Android MediaExtractor fails when presented with more than 10 tracks, so I just rolled my own trivial container file. Peak bitrate for Henry is around 40 Mbps, so it wouldn't stream for most people. With some rearrangement of the file so each strip has a full gop continuous, instead of time interleaving all 11, the bitrate wold be cut in half, but it would still be a lot of fairly small requests, so it would call for pipelined HTTP2.
- modeless 8y ago
  Ah, so all the frames for every strip are interleaved in your container, and you just read sequentially and ignore the frames you don't need? That's probably the right thing for local videos. I had each gop in a separate file like HLS or DASH (except for the background which was a single file also containing the audio track). It's unwieldy but makes HTTP streaming a little simpler because you don't need range requests or an index. Also, instead of bitstream hacking to stitch three strips into one, I encoded multiple strips into "pre-stitched" views. This means that every strip is encoded redundantly in multiple views, bloating the on-disk video size. But for streaming that only affects the server, not the client, and it's nice for the client to only download/decode one view at a time (plus the background) instead of three strips. Bitstream hacking to join the strips would definitely be better if it can work, though.

sigi45 8y ago

If you have the keyframe always at the seek time, would that be enough? Because you said the delay is too long? If yes, would it be possible to render, for every second (half second or whatever resolution you like) a snippet which only contains the seek time until the next keyframe and than tell the webserver 'if someone seeks to time x, take this snippet and when done, jump to the original video'? You could do that transparent with some fuse fs.

iamleppert 8y ago

Another approach to space efficiency would be to do away with the need for dual video streams entirely and just average the stereo images together to form a single monocular image. Then, send a disparity map along with the monocular video. Decode the mono video and use the disparity map to interpolate the view of either eye. You’ll have all the information you need for reconstruction and the disparity map can be efficiently compressed via normalization and perhaps even by sending just the vectors of the contours. Another idea is to take advantage of the fact head motions are really just a translation vector of the camera. There’s no need to send pixels that have just transposed locations unless they have changed in time. If I was designing such a system I’d try to take advantage of the fact there isn’t a lot changing fundamentally in the scene when you move your head, and maintain some sort of state and only request chunks of pixels that are actually needed. You wouldn’t even have to use a traditional video codec as the preservation of state would be far more efficient than thinking about things in terms of flat pixels and video.

munificent 8y ago
> Decode the mono video and use the disparity map to interpolate the view of either eye. By "disparity map" are you thinking something like a heightmap applied to the scene facing the viewer and then you use that to skew things for each eye? If so, how would that handle parts of the scene that are occluded/revealed to one eye but not the other?
- iamleppert 8y ago
  True, occlusions would be a problem but we’re taking about fake autostereoscopic 3D here, where most of the stereo rigs used for capture have but a modest baseline. Almost all of the depth perception comes from disparity, occlusions would still in fact be very visible by the averaging method I described and be at whatever depth plane of the occluder which is probably a good guess anyway. Not like your other eye would receive a correspondence from an occlusion in the real world.
  - romwell 8y ago

DanielBMarkham 8y ago

(Much hand-waving here) I wonder if the problem isn't a misaligned paradigm for just what a codec does. Right now, codecs exist to take bytestreams and fill frames at a certain rate. They're a bitstream-to-framerate device. What if instead of delivering to a frame during a certain time period, the codec had to instantly deliver whatever it could, but only geared to the actual acuity of the users retina. Seems like there would be less information, you would never have lag, and all optimization would be around the detail of data delivered, not framerate or screensize. I also birthday parties and Bar Mitzvahs, folks. I'm here all week. (I figure it couldn't hurt to throw crazy ideas out there. Every now and then a crazy idea actually amounts to something. Coding is cool because it not only let you solve problems, sometimes it lets you change the universe the problem lives in. Good luck, John!)

undershirt 8y ago
only delivering the visible portion of a 360° video assumes that the video is ready to be decoded at any angle, which is the entire problem.
- DanielBMarkham 8y ago
  But the problem is to display something for a moving angle, not display something for everything. As best I can tell, it is a different problem. I want to restate what I'm hearing so you can correct me. You are saying something like "Gee, if we could show any retinally-matched conic the user's pointing their eyeball at? To do that we'd have to have the entire scene rendered anyway" I'm saying you can't do it now. It hurts when I do that. So stop doing that. Instead of trying to render stereo scenes quicker, try to render a moving conic real-time quicker. When the codec runs, it's optimizing areas of a frame changing over time. If instead it optimized possible vision movement paths, you'd end up solving the framerate and resolution problems. Then you could concentrate on optimizing the codec in a different way than people are currently trying to optimize it. It couldn't hurt, and there may be opportunities for consolidation if you look at the problem as visual-movement-path-rendering instead of frame rendering. I don't know. I just know it's a problem now. Set your constraints differently and optimize along a different line. Sometimes that works. Does that explain the differences here? Or do you want me to start spec'ing out what I mean by retinal-path codec? At some point this will a bit over the top. ADD: I'll say it a little differently. Our constraint now is "how fast can you render this frame". Codecs are built for rendering frames on screens where people watch movies and play games. But that's not the world we're trying to solve a problem in. They may be configured the wrong way. Instead, require that anything the eyeball is looking at should render consistently in, say 5ms. This sounds like the same thing, until you realize that the eyeball can't look at everything at the same time. Different parts of the image are temporally separated. So if I update the image in back of your head once every 200ms? You're not going to know. As far you're concerned, it's all instant. It becomes a different kind of problem.

mattbierner 8y ago

Fascinating. Ammeture question though: are there other ways to encode the images besides pixels that will ultimately be better suited to 360 content? Holograms came to my mind because they degrade gracefully as you toss out bits of the signal

samplatt 8y ago
Lytro's light-field camera technologies have huge potential for 360 content and the photogrammetry industry, but they effectively sat on the technology doing very little with it (compared to their competitors), until Google hired some of the employees as the company shut down earlier this year.

Animats 8y ago

I dunno. Looking at canned content with VR goggles seems like a dead end. It's 3D TV on steroids. Remember 3D TV? Works fine. Too much trouble just to watch TV. Market failure. It's been five years since the Oculus Rift DK1 appeared, and there's still no killer app. Game developers are pulling back from VR.[1][2] The VR virtual worlds are a disaster. Sansar has about 50 (fifty) concurrent users. SineSpace and High Fidelity have similar numbers. VR goggles are cool for about an hour, and in the closet in a month. The technology is coming along just fine, but that's not the problem. [1] https://www.gq.com/story/is-vr-gaming-over-before-it-even-started https://www.gq.com/story/is-vr-gaming-over-before-it-even-st... [2] https://mashable.com/2018/01/24/virtual-reality-gaming-loser-gdc-2018-survey/#WG.n6Sqwp5qm https://mashable.com/2018/01/24/virtual-reality-gaming-loser...

samplatt 8y ago
You can keep believing that, sure. I'll just be over here, continuing to earn my living delivering interactive 360 video and VR demonstrations for our clients :-)
- wyldfire 8y ago
  Is the video still produced as ordinary 2d/planar video? It's just projected as 360? Or is there some kind of depth information? What makes it so compelling?
  - samplatt 8y ago
    Most of our video is rendered down to ordinary video that gets reprojected, driven mostly by a requirement to be able to be played back on lowest-common-denominator hardware and platforms; nothing near as cool as what Carmack's article details. But there's other problems that require fun solutions to, too.
    - brennebeck 8y ago
      I’m curious: what company/industry are you with and what tech are you using specifically?

Njkl 8y ago

This might not be possible if the "system software" doesn't cooperate, but it's possible to encode videos such that you can keep them "warm" without decoding all frames, for example in a IBBPBBPBBPBB structure where all B-frames are not referenced by any other frame (other arrangements are possible). Forcing this structure has a cost, but it's much smaller than having more I-frames. You can then alternate decoding 3 such streams (each one offset by 1 frame, including the I-frames - this is not a problem, is just means you'll not be ready to output anything for 2 frames after a seek) for the cost of 1. Switching to 60fps is then "instantaneous". Old iTunes used to code h.264 video like this (with a PBPBPB structure, so it could play at half-rate, which it did if the CPU couldn't keep up). Note unreferenced does not imply B-frame nor the other way around. Another (admittedly crazy) idea, for a setup with a lower-res version and a higher-res overlay, is trying to store the difference only, affording a (significant?) bitrate reduction for the high res "patches". This is very tricky to do in practice, though (needs larger range or losing the lsb; the codecs aren't really designed for this). I don't think it has ever been done.

Ono-Sendai 8y ago

> Directly texture mapping a video in VR gives you linear filtering, not the sRGB filtering you want. Isn't this back-to-front? Generally filtering is considered to work better in a linear colour space. Using a sRGB texture will convert each pixel to a linear colour space, before the reconstruction filtering is done, AFAIK.

Maurice_Ribble 8y ago

I find it interesting that you went in a different direction than the earlier ideas discussed for this problem. I can see where this is simpler in many ways and probably more importantly a better fit for a lot of content than some of those early ideas. You mentioned the overhead for handling many streams being a problem. Can you go into more detail on the type of overhead you saw and do you feel this is something can can be addressed with better drivers or some application changes? Or do you think the only solution is packing multiple stripes into a single stream using the encoding bit manipulation you mentioned? It seems a shame for hardware that can support more streams leaves them inaccessible to real world applications.

nomercy400 8y ago

You don't need 5120 (for 360 degrees on X-axis) x 5120 (for 360 degrees on Y-axis) do you? If you cover 360 degrees in the X direction, then your Y-axis only has to be 180 degrees, because you don't need 'behind you' on the Y-axis as that is already covered by the X-axis' wide 360 degrees. Or am I making an error here?

michaelt 8y ago
It's stereo, so Carmack means 360°x180°x2 for two eyes. One would assume there would be a lot in common between stereo cameras' views, so presumably there are compression efficiencies to be found.

holografix 8y ago

Legitimate question: are there “untouched nature” type simulations for one of the leading VR platforms? Sitting at the top of a mountain, different times of the day and weather. Exploring a huge, very realistic forest. Being a pigeon observing a busy street from above as you fly around and perch yourself on different spots.