4 ms·

The big incoming UI challenge is XR. But as with ST/Alto, it's an integrated bootstrap problem of hardware, system design, foundational software, user software

The big incoming UI challenge is XR. But as with ST/Alto, it's an integrated bootstrap problem of hardware, system design, foundational software, user software, and UI. Absent research labs, we wait on market availability for hardware. For the modern analogues of mice and bitmapped screens. Eye tracking may be cheap next year, but now it's $2k. High-resolution HMDs may be $10k next year, but now they're simply unavailable. Hand tracking is flaky. Haptics are "it buzzes". For software, we're so used to the 2D windows interface, it's easy to forget how much foundational work was needed. Eg, around topography, and Smalltalk, and live coding. Hand gesture recognition, spoken dialog management, 3D constraint-based layout, are just a few modern analogues. And so on. So it looks like we're going to be doing broad-based innovation and infrastructure construction, gated on hardware availability. And plagued by patents. It's not clear to me what can usefully be done to explore UI in the meantime. You can do wireframing in TiltBrush, but not really. It doesn't give you the feel of say hand-attached controls, and of automated collaboration in general. The "oh, it feels good to present this graph as a sphere, and hold it you lap and peel off layers". So... Perhaps do UI design by paper improv? A crew scribbling on and holding up paper, hovering around a "user", who is managing a todo list? :) That could be fun.

scroot 9y ago
The eleVR people have some interesting demos in this regard, where you program from inside the VR system. That might be the best way to try things out and conduct the "foundational work"
- mncharity 9y ago
  > "foundational work" Hmm, I was ambiguous there. What I had in mind for "foundational" was, for example, constraint engines for GUI layout. Now we have CSS. But there was something like a decade, where we knew constraints were a good way to do UI layout, but there weren't available constraint engines. Each major language had odd bits of abandonware zombies, academic or hobby projects, with code which might or might not be online, or be buildable, or be licensed for use. So we had to wait for market incentives to change, and some non-trivial work to be done, before we had the foundation element of GUI constraint-based layout. An Alto analogy might be bitmapped fonts and display. A current analogy might be 3D dynamic structured graph layout. There's been a lot of research over the years on how to dynamically modify graphs without disorienting viewers. And on how to layout graphs within graphs, and with various spatial constraints. But last time I checked, the github cupboard was bare. So say you want to write a really wizzy todo app with complex dependencies. Well, first you read two decades of graph layout papers; then you write the graph layout library no one else has gotten to yet. Foundational work on our collective todo list.
- scroot 9y ago
  Ok I found what I was talking about: http://elevr.com/portfolio/future-programming-interfaces/ http://elevr.com/portfolio/future-programming-interfaces/
seanmcdirmid 9y ago
VR has a huge input problem, you can't touch any of the 3D stuff, you are basically left waving hands in the air. Bret Victor's lab is going in the other direction of tangible objects with digital projections.
- setr 9y ago
  There's really no reason we can't treat it like navigating a video game; movement and interation through the normal kb+m/controller model; the helmet is just the monitor, with a bigger pov (due to head turning) Im not sure why there's a belief that you need to pretend you physically exist in the vr world; you just need to be able to interact with, and navigate, for it to be sufficiently valuable. ie imagine using it as an infinite real-estate development environment, with possibly a visual layout of your data structures, and the ability to change location (ie a mountain top view). Its sufficiently valuable as a product with just this basic featureset. We don't need to wait for full-body VR, or AR, though no one seems to believe it
  - mncharity 9y ago
    Creating HMDs with an angular resolution sufficient for being a screen substitute, is still a work in progress. Otherwise, I image what you describe might have happened. The available market was gaming. With its emphasis on "presence", high frame rates, high polygon counts, dislike of visual artifacts and blur, and tolerance of low angular resolution. And even now, as parts become available to create a better screen substitute, the market isn't seen as worth pursuing yet. But 2018 should see a higher-resolution generation of gaming HMDs. And there's Varjo to watch/hope for, though it will be $10k-ish.
- mncharity 9y ago
  > tangible objects with digital projections You can also use tangible objects with XR. It might be useful for sorting things, like system subparts, and thinking with your hands. You can fit camera tracking markers on 1 cm wooden cubes. > you can't touch any of the 3D stuff The current controller "tock" impulse haptics can be surprisingly effective. Your brain fills in a lot. But no, no force feedback or individual finger haptics is mass market yet.
setr 9y ago
I wasn't really referring to how you physically interact with the system. More along the lines of: if you implement a 3D environment of your files, but you're still interacting with a directory/file model, then you've just constructed a less efficient interface. But if you can now move on to some kind of 3 dimensional, or n-dimensional model, maybe there's something to gain with the 3D interface. Theres little point representing 2D structures in a 3D environment. You have to change what you're representing as well to see any fundamental benefits.
- mncharity 9y ago
  Nod. Though just relaxing the severe real estate constraints of screens may be a big deal. It's much easier for tooling to provide speculative content when its "off to the side, just in case you care" rather than "in you face, covering what you were working on". So various existing "cute... but it's just not worth the cost of dealing with" ideas might become viable. Code analytics and documentation could be ever present, accessible at glance, just off to the side. That's one fun XR puzzle game. Given a domain where you deeply understand the constraints which determine some system shape, ponder what will change as XR alters the UI constraints. And what approaches already tried, might deserve another look.

scroot 9y ago

The eleVR people have some interesting demos in this regard, where you program from inside the VR system. That might be the best way to try things out and conduct the "foundational work"

mncharity 9y ago
> "foundational work" Hmm, I was ambiguous there. What I had in mind for "foundational" was, for example, constraint engines for GUI layout. Now we have CSS. But there was something like a decade, where we knew constraints were a good way to do UI layout, but there weren't available constraint engines. Each major language had odd bits of abandonware zombies, academic or hobby projects, with code which might or might not be online, or be buildable, or be licensed for use. So we had to wait for market incentives to change, and some non-trivial work to be done, before we had the foundation element of GUI constraint-based layout. An Alto analogy might be bitmapped fonts and display. A current analogy might be 3D dynamic structured graph layout. There's been a lot of research over the years on how to dynamically modify graphs without disorienting viewers. And on how to layout graphs within graphs, and with various spatial constraints. But last time I checked, the github cupboard was bare. So say you want to write a really wizzy todo app with complex dependencies. Well, first you read two decades of graph layout papers; then you write the graph layout library no one else has gotten to yet. Foundational work on our collective todo list.
scroot 9y ago
Ok I found what I was talking about: http://elevr.com/portfolio/future-programming-interfaces/ http://elevr.com/portfolio/future-programming-interfaces/

seanmcdirmid 9y ago

VR has a huge input problem, you can't touch any of the 3D stuff, you are basically left waving hands in the air. Bret Victor's lab is going in the other direction of tangible objects with digital projections.

setr 9y ago
There's really no reason we can't treat it like navigating a video game; movement and interation through the normal kb+m/controller model; the helmet is just the monitor, with a bigger pov (due to head turning) Im not sure why there's a belief that you need to pretend you physically exist in the vr world; you just need to be able to interact with, and navigate, for it to be sufficiently valuable. ie imagine using it as an infinite real-estate development environment, with possibly a visual layout of your data structures, and the ability to change location (ie a mountain top view). Its sufficiently valuable as a product with just this basic featureset. We don't need to wait for full-body VR, or AR, though no one seems to believe it
- mncharity 9y ago
  Creating HMDs with an angular resolution sufficient for being a screen substitute, is still a work in progress. Otherwise, I image what you describe might have happened. The available market was gaming. With its emphasis on "presence", high frame rates, high polygon counts, dislike of visual artifacts and blur, and tolerance of low angular resolution. And even now, as parts become available to create a better screen substitute, the market isn't seen as worth pursuing yet. But 2018 should see a higher-resolution generation of gaming HMDs. And there's Varjo to watch/hope for, though it will be $10k-ish.
mncharity 9y ago
> tangible objects with digital projections You can also use tangible objects with XR. It might be useful for sorting things, like system subparts, and thinking with your hands. You can fit camera tracking markers on 1 cm wooden cubes. > you can't touch any of the 3D stuff The current controller "tock" impulse haptics can be surprisingly effective. Your brain fills in a lot. But no, no force feedback or individual finger haptics is mass market yet.

setr 9y ago

I wasn't really referring to how you physically interact with the system. More along the lines of: if you implement a 3D environment of your files, but you're still interacting with a directory/file model, then you've just constructed a less efficient interface. But if you can now move on to some kind of 3 dimensional, or n-dimensional model, maybe there's something to gain with the 3D interface. Theres little point representing 2D structures in a 3D environment. You have to change what you're representing as well to see any fundamental benefits.

mncharity 9y ago
Nod. Though just relaxing the severe real estate constraints of screens may be a big deal. It's much easier for tooling to provide speculative content when its "off to the side, just in case you care" rather than "in you face, covering what you were working on". So various existing "cute... but it's just not worth the cost of dealing with" ideas might become viable. Code analytics and documentation could be ever present, accessible at glance, just off to the side. That's one fun XR puzzle game. Given a domain where you deeply understand the constraints which determine some system shape, ponder what will change as XR alters the UI constraints. And what approaches already tried, might deserve another look.