7 ms·

I've been working in AR/VR for a few years now and fairly convinced of this. In fact, beyond the OS interlace I'd say this is true of presenting most data. The

I've been working in AR/VR for a few years now and fairly convinced of this. In fact, beyond the OS interlace I'd say this is true of presenting most data. The usefulness of most interfaces is that they abstract something real. Part of the value of abstraction is the removal of most the extraneous aspects of a problem. Removing a whole dimension removes a lot of data that may be extraneous. Maps are one very good example of this. A map is partially so powerful because it removes a ton of information you probably don't care about. Occasionally would you like to know a house is up a steep hill before you drive there? Sure. But most of the time who cares. And if you really need elevation data, there's a special map for that. A map which actually shows you information about grade in much clearer way than even a 3D model would (being as humans are actually really bad at guessing a grade by looking at a real slope).

jerf 7y ago
I think it's more than that. As Carmack observes, you don't see in 3D. You see two 2D planes that your brain extracts a certain amount of depth information from. As a UI goes, you can't actually freely use that third dimension, because as soon as one element obscures another, either the front element is too opaque to see through, in which case the second might as well not be there, or the opacity is not 100% in which case it just gets confusing fast. You've really got something more like 2.1d to work with with humans; intefaces need to take that into account, and maybe sometimes use that .1d depth dimension to provide light hinting, but not depend on it. So you're not removing a dimension... you're acknowledging it doesn't exist. There isn't really a "third dimension" here to take advantage of. It may be visually impressive if something flies from far away to up close, but in interface terms, all that has happened is that the z-buffer values shifted, but the interface still has the same number of square radians available that it did before. The depth information is a lot more like an additional color than an additional spatial dimension. UIs are better with color hinting available when used well, but there's not much fundamental difference between a color 2D interface and a monochrome 2D interface. Proof that it isn't much is that there are some people who use settings to set their phone to monochrome, and the result is at worst occasionally inconvenient, not crippling. Compare with trying to go from a 2D interface to a 1D interface, which would be utterly crippling and destroy a huge swathe of current UI paradigms. To truly "see in 3D" would require a fourth-dimension perspective. A 4D person could use a 3D display arbitrarily, because they can freely see the entire 3D space, including seeing things inside opaque spheres, etc, just like we can look at a 2D display and see the inside of circles and boxes freely. They have access to all sorts of paradigms we don't, just like no 2D designer would ever come up with a "text box", even though it fits on the 2D plane, because no 2D user could see inside the text box to see what's in it. (This is one of those cases where the mere act of typing it out improved my understanding of the topic. I didn't have the "depth information is really just another color" idea going in, but the more I think about it the more sense it makes, in terms of the amount of information that channel can carry. Just like color, it's not quite 0, but you can't really stuff all that much in there.)
- vbuwivbiu 7y ago
  yes we see 2d surfaces in 3d. Dolphins can actually see in 3d as they can sense entire 3d volumes and project them back to eachother too.
  - snazz 7y ago
    I’m assuming you meant “yes we see 3d surfaces in 2d” to be consistent with the parent comment? Also, I’m impressed that dolphins have that ability. How does projecting them to each other work biologically? Do they somehow encode what they see as sound (less lossily than humans do)?
    - taneq 7y ago
      I'd imagine they work directly in the frequency domain. They don't "see with sound", they'll just hear nested spheres or whatever.
      - all2 7y ago
        This suggests to me that humans could harness the same audio phenomena.
        taneq 7y ago
        Some people can. There's a few cases of blind humans who can echolocate to some degree of accuracy (enough to ride a skateboard in one case).
- DubiousPusher 7y ago
  True, almost any interface that overlaps helpful data in layers or stacks is usually terrible. Hell, I don't even like desktop computer GUIs that allow you to have stacks of windows. I'd rather see one thing at a time and cycle between them. Or have a picker view. That said we do actually get quite a bit out of that ability to see depth. People who lose depth perception have quite a hard time adapting to the world. And our spatial understanding seems to go beyond our vision. That to me is where a 3D interface might be really powerful. Sometimes things which we struggle to decode in 2D are just intuitive in 3D like knots or the run of wires or pipes. As I said elsewhere in this thread I think the 3D interfaces that are really going to be powerful haven't occurred to us yet. And I believe that what we'll find in time is that there are things which 3D interfaces are tremendously advantagous for and using anything else will feel like a hinderance. But those will be things for which 2D interfaces don't already do an amazing job.
stcredzero 7y ago
Maps are one very good example of this. A map is partially so powerful because it removes a ton of information you probably don't care about...Sure. But most of the time who cares. This is usually true, but then again, there is the flip side. There are exceptions to this, and those exceptions may be interesting areas of inquiry. In particular, there are situations where one would like to go from a map representation to seeing relationships which can't be readily expressed in a map. One example of this would be evaluating views from a house. Another example, would be the layout of ducting within the ceiling or within crawl spaces. Another example would be evaluating lines of sight or lines of fire within a particular scene. Often, this can be handled as something like a relief map or a scale model which can be manipulated and zoomed into by the user in VR, but the exceptions to the "a map is usually enough" seem to coincide with areas where VR/AR might be particularly interesting. I think there might be some relationship here to Archy. https://en.wikipedia.org/wiki/Archy https://en.wikipedia.org/wiki/Archy
- DubiousPusher 7y ago
  I completely agree. A lot of what my team works on is finding those cases where a 3D view has a very high value. One example is a project we worked on that allows construction companies to use the 3D CAD data for their project as a setting to move equipment through. It's not unheard of for a company to bring a piece of equipment on site only to find it cannot move through a given space because material is in the way. Sometimes a partially complete structure which could've been built later is impassable. This is a situation in which a completely 3D representation is really powerful and the effort can save significant money in the long run. I think there are absolutely cases where 3D interfaces will blow away 2D ones. We're on the road to discovering those. But the industry has had a tendency to try to take what works in 2D and make a 3D version, often ending with a user experience that is less intuitive and less powerful. The 3D interfaces that are really going to make a difference will probably take decades to invent/discover. Our 2D interfaces have had literally thousands of years to develop.
  - stcredzero 7y ago
    It's not unheard of for a company to bring a piece of equipment on site only to find it cannot move through a given space because material is in the way. This is precisely the kind of situation I was thinking about! This is a situation in which a completely 3D representation is really powerful and the effort can save significant money in the long run. I think this is a key criteria for determining what problems should be looked at.
atoav 7y ago
I strongly agree with your point about abstraction. However I'd love if people who work on such interfaces would get a stronger grip on how the perception of depth works. Painters for example use saturation, colors, occlusion, details, abstraction and perspective to get the most out of depth. In a 3D scene with the same objects you can make or break the scene's readability, purely based on the placement of objects. What you can't do, is to come and slap 3D onto it and it will somehow work out. If you think 3D space itself will make it magically work, you are wrong. Like a graphic designer puts a lot of work into making 2D information readable, 3D information also would need a ton of work to become readable. Interestingly enough a lot of 2D designs actually emulate levels of depth by using occlusion, focus, shadows, etc. I can certainly imagine 3D interfaces that would work, you would only have to be very disciplined about your use of dimensions and how much to have on which plane and why.
- DubiousPusher 7y ago
  Agree. the really great 3D interfaces we create will be 3D first and not a clumsy adaption of 2D ones.
munchbunny 7y ago
On the one hand, I totally agree with this as my experience with AR/VR interfaces as well. However, I'm not convinced that this is due to any inherent limitations in our brain's ability to understand a third dimension, just that we are accustomed to only having to deal with two dimensions. I'm carefully wording this in terms of UI design in 3-D because I agree about the physical issues of eyes and focus. For example, in many spaceflight simulators, the radar includes vertical bars to indicate height above/below your plane. That's a 3-D interface rendered onto a 2-D screen, though it's 3-D in VR. Once you learn to read it, you become highly effective at mapping what you see to a spatial understanding of what's going on around you. The driver's controls in a car are also a good example of a 3-D interface demonstrating patterns that are inherently 3-D (twist knobs, paddles, H pattern gearbox, etc.), and it works mostly because you are expected to develop muscle memory.
- wpietri 7y ago
  Are car controls really a good example of a 3D interface in the sense Carmack is discussing? I think he's specifically talking about 3D visualization. In contrast car interfaces are specifically non-visual. You have to look to learn how to use them, but after that, anything you have to look at to use is bad, because it takes the driver's eyes off the road. But in the same way that the human eyes extract a 2D version of a 3D world, I think the a car's nobs are a proprioceptive projection of the 3D world.
  - munchbunny 7y ago
    I know Carmack was specifically talking about 3D visualization, but I was responding to the parent comment's point about the value of data simplification that 2D brings. I don't usually quote myself, but that's what I meant by: I'm carefully wording this in terms of UI design in 3-D because I agree about the physical issues of eyes and focus. I specifically brought up the car's knobs and wheels and buttons because it's an example of a fairly complex but widely adopted and successful non-2D interface. My point was that I don't think there is an inherent limitation in the human brain's ability to comprehend or think in 3D, and that I think it's more of a training and acclimatization phenomenon.
Terr_ 7y ago
> The usefulness of most interfaces is that they abstract something real. When I think someone is overenthusiastic about virtual reality applications (especially stuff involving "collaboration") I like to contrast it with an insanely-successful tool with broad adoption: E-mail. Who wants to take their message, drop it into a virtual envelope, affix a virtually-licked stamp and virtually-walk to the local virtual-post-office? Heck no, we just click the big "Send" button! We didn't model the cruft in 3D, we removed it, because it wasn't truly necessary to the job.
- amelius 7y ago
  You can still send emails in 3d. Except you do it from a virtual beach.
- andybak 7y ago
  Let's not throw the baby out with the bathwater here. Carmack is talking about good UI for VR applications - not why VR applications are a mistake.
lostmsu 7y ago
I completely disagree with your maps example here. Yes, 2D currently works ok, but I can't see how it can beat a full-surround AR experience, which simply overlays a shiny string along the path you have to follow. That is for real-time navigation. For positioning and planning, 3D will be required to navigate a multistory mall. But a 3D map movie-style could potentially be replaced with something better, if researched.

jerf 7y ago

I think it's more than that. As Carmack observes, you don't see in 3D. You see two 2D planes that your brain extracts a certain amount of depth information from. As a UI goes, you can't actually freely use that third dimension, because as soon as one element obscures another, either the front element is too opaque to see through, in which case the second might as well not be there, or the opacity is not 100% in which case it just gets confusing fast. You've really got something more like 2.1d to work with with humans; intefaces need to take that into account, and maybe sometimes use that .1d depth dimension to provide light hinting, but not depend on it. So you're not removing a dimension... you're acknowledging it doesn't exist. There isn't really a "third dimension" here to take advantage of. It may be visually impressive if something flies from far away to up close, but in interface terms, all that has happened is that the z-buffer values shifted, but the interface still has the same number of square radians available that it did before. The depth information is a lot more like an additional color than an additional spatial dimension. UIs are better with color hinting available when used well, but there's not much fundamental difference between a color 2D interface and a monochrome 2D interface. Proof that it isn't much is that there are some people who use settings to set their phone to monochrome, and the result is at worst occasionally inconvenient, not crippling. Compare with trying to go from a 2D interface to a 1D interface, which would be utterly crippling and destroy a huge swathe of current UI paradigms. To truly "see in 3D" would require a fourth-dimension perspective. A 4D person could use a 3D display arbitrarily, because they can freely see the entire 3D space, including seeing things inside opaque spheres, etc, just like we can look at a 2D display and see the inside of circles and boxes freely. They have access to all sorts of paradigms we don't, just like no 2D designer would ever come up with a "text box", even though it fits on the 2D plane, because no 2D user could see inside the text box to see what's in it. (This is one of those cases where the mere act of typing it out improved my understanding of the topic. I didn't have the "depth information is really just another color" idea going in, but the more I think about it the more sense it makes, in terms of the amount of information that channel can carry. Just like color, it's not quite 0, but you can't really stuff all that much in there.)

vbuwivbiu 7y ago
yes we see 2d surfaces in 3d. Dolphins can actually see in 3d as they can sense entire 3d volumes and project them back to eachother too.
- snazz 7y ago
  I’m assuming you meant “yes we see 3d surfaces in 2d” to be consistent with the parent comment? Also, I’m impressed that dolphins have that ability. How does projecting them to each other work biologically? Do they somehow encode what they see as sound (less lossily than humans do)?
  - taneq 7y ago
    I'd imagine they work directly in the frequency domain. They don't "see with sound", they'll just hear nested spheres or whatever.
    - all2 7y ago
      This suggests to me that humans could harness the same audio phenomena.
      - taneq 7y ago
        Some people can. There's a few cases of blind humans who can echolocate to some degree of accuracy (enough to ride a skateboard in one case).
DubiousPusher 7y ago
True, almost any interface that overlaps helpful data in layers or stacks is usually terrible. Hell, I don't even like desktop computer GUIs that allow you to have stacks of windows. I'd rather see one thing at a time and cycle between them. Or have a picker view. That said we do actually get quite a bit out of that ability to see depth. People who lose depth perception have quite a hard time adapting to the world. And our spatial understanding seems to go beyond our vision. That to me is where a 3D interface might be really powerful. Sometimes things which we struggle to decode in 2D are just intuitive in 3D like knots or the run of wires or pipes. As I said elsewhere in this thread I think the 3D interfaces that are really going to be powerful haven't occurred to us yet. And I believe that what we'll find in time is that there are things which 3D interfaces are tremendously advantagous for and using anything else will feel like a hinderance. But those will be things for which 2D interfaces don't already do an amazing job.

stcredzero 7y ago

Maps are one very good example of this. A map is partially so powerful because it removes a ton of information you probably don't care about...Sure. But most of the time who cares. This is usually true, but then again, there is the flip side. There are exceptions to this, and those exceptions may be interesting areas of inquiry. In particular, there are situations where one would like to go from a map representation to seeing relationships which can't be readily expressed in a map. One example of this would be evaluating views from a house. Another example, would be the layout of ducting within the ceiling or within crawl spaces. Another example would be evaluating lines of sight or lines of fire within a particular scene. Often, this can be handled as something like a relief map or a scale model which can be manipulated and zoomed into by the user in VR, but the exceptions to the "a map is usually enough" seem to coincide with areas where VR/AR might be particularly interesting. I think there might be some relationship here to Archy. https://en.wikipedia.org/wiki/Archy https://en.wikipedia.org/wiki/Archy

DubiousPusher 7y ago
I completely agree. A lot of what my team works on is finding those cases where a 3D view has a very high value. One example is a project we worked on that allows construction companies to use the 3D CAD data for their project as a setting to move equipment through. It's not unheard of for a company to bring a piece of equipment on site only to find it cannot move through a given space because material is in the way. Sometimes a partially complete structure which could've been built later is impassable. This is a situation in which a completely 3D representation is really powerful and the effort can save significant money in the long run. I think there are absolutely cases where 3D interfaces will blow away 2D ones. We're on the road to discovering those. But the industry has had a tendency to try to take what works in 2D and make a 3D version, often ending with a user experience that is less intuitive and less powerful. The 3D interfaces that are really going to make a difference will probably take decades to invent/discover. Our 2D interfaces have had literally thousands of years to develop.
- stcredzero 7y ago
  It's not unheard of for a company to bring a piece of equipment on site only to find it cannot move through a given space because material is in the way. This is precisely the kind of situation I was thinking about! This is a situation in which a completely 3D representation is really powerful and the effort can save significant money in the long run. I think this is a key criteria for determining what problems should be looked at.

atoav 7y ago

I strongly agree with your point about abstraction. However I'd love if people who work on such interfaces would get a stronger grip on how the perception of depth works. Painters for example use saturation, colors, occlusion, details, abstraction and perspective to get the most out of depth. In a 3D scene with the same objects you can make or break the scene's readability, purely based on the placement of objects. What you can't do, is to come and slap 3D onto it and it will somehow work out. If you think 3D space itself will make it magically work, you are wrong. Like a graphic designer puts a lot of work into making 2D information readable, 3D information also would need a ton of work to become readable. Interestingly enough a lot of 2D designs actually emulate levels of depth by using occlusion, focus, shadows, etc. I can certainly imagine 3D interfaces that would work, you would only have to be very disciplined about your use of dimensions and how much to have on which plane and why.

DubiousPusher 7y ago
Agree. the really great 3D interfaces we create will be 3D first and not a clumsy adaption of 2D ones.

munchbunny 7y ago

On the one hand, I totally agree with this as my experience with AR/VR interfaces as well. However, I'm not convinced that this is due to any inherent limitations in our brain's ability to understand a third dimension, just that we are accustomed to only having to deal with two dimensions. I'm carefully wording this in terms of UI design in 3-D because I agree about the physical issues of eyes and focus. For example, in many spaceflight simulators, the radar includes vertical bars to indicate height above/below your plane. That's a 3-D interface rendered onto a 2-D screen, though it's 3-D in VR. Once you learn to read it, you become highly effective at mapping what you see to a spatial understanding of what's going on around you. The driver's controls in a car are also a good example of a 3-D interface demonstrating patterns that are inherently 3-D (twist knobs, paddles, H pattern gearbox, etc.), and it works mostly because you are expected to develop muscle memory.

wpietri 7y ago
Are car controls really a good example of a 3D interface in the sense Carmack is discussing? I think he's specifically talking about 3D visualization. In contrast car interfaces are specifically non-visual. You have to look to learn how to use them, but after that, anything you have to look at to use is bad, because it takes the driver's eyes off the road. But in the same way that the human eyes extract a 2D version of a 3D world, I think the a car's nobs are a proprioceptive projection of the 3D world.
- munchbunny 7y ago
  I know Carmack was specifically talking about 3D visualization, but I was responding to the parent comment's point about the value of data simplification that 2D brings. I don't usually quote myself, but that's what I meant by: I'm carefully wording this in terms of UI design in 3-D because I agree about the physical issues of eyes and focus. I specifically brought up the car's knobs and wheels and buttons because it's an example of a fairly complex but widely adopted and successful non-2D interface. My point was that I don't think there is an inherent limitation in the human brain's ability to comprehend or think in 3D, and that I think it's more of a training and acclimatization phenomenon.

Terr_ 7y ago

> The usefulness of most interfaces is that they abstract something real. When I think someone is overenthusiastic about virtual reality applications (especially stuff involving "collaboration") I like to contrast it with an insanely-successful tool with broad adoption: E-mail. Who wants to take their message, drop it into a virtual envelope, affix a virtually-licked stamp and virtually-walk to the local virtual-post-office? Heck no, we just click the big "Send" button! We didn't model the cruft in 3D, we removed it, because it wasn't truly necessary to the job.

amelius 7y ago
You can still send emails in 3d. Except you do it from a virtual beach.
andybak 7y ago
Let's not throw the baby out with the bathwater here. Carmack is talking about good UI for VR applications - not why VR applications are a mistake.

lostmsu 7y ago

I completely disagree with your maps example here. Yes, 2D currently works ok, but I can't see how it can beat a full-surround AR experience, which simply overlays a shiny string along the path you have to follow. That is for real-time navigation. For positioning and planning, 3D will be required to navigate a multistory mall. But a 3D map movie-style could potentially be replaced with something better, if researched.