4 ms·

Can anyone suggest interesting work that is kind of the opposite of this? Vast real estate, rich HIDs, dynamic autonomous presentation? So focused on many/big

Can anyone suggest interesting work that is kind of the opposite of this? Vast real estate, rich HIDs, dynamic autonomous presentation? So focused on many/big screens, with an eye towards future high-res AR/VR 3D, rather than a single "space is precious" small screen. Rich human interface devices, like multitouch and 3D hand pose and stylus and head position and gaze, rather than a single mouse. Display as a dynamic ecosystem of presentations, under the influence of multiple agents, rather than detailed manual control. UI as collaborative improv dance, rather than as micromanagement. Punchcards and console toggles; teletype printer; VT100 terminal; small bitmapped monitor and mouse; larger monitor; largerer monitor; largererer; multiple monitors; with multitouch and stylus tablet; future high-res AR/VR... as tech so very slowly becomes less crippling, the UIs do too... Hmm, though for current VR, with its 1980's VGA-like non-blurry pixels are more precious than they've been for decades, fine-grained control and space sharing might be quite nice?

saulrh 8y ago
I don't think that I'd expect those interfaces would exist yet, nor are they going to be coming up relatively soon. You mostly see interfaces that have a ton of work put into them either when someone is smoothing off every possible rough edge they can find to cater to the widest possible audience (e.g. Windows Desktop), or when someone is optimizing the heck out of their interface to improve performance on a very specific use case. The second case clearly isn't what we're looking for; those interfaces will tend to be massively refined with all the cruft cut off, and in many cases the very first thing that goes is any trace of analog input or output. There's a reason that some of the most impressive feats of human bandwidth are from typists and players of rhythm games like Guitar Hero. The first case isn't going to happen until the necessary interface peripherals and interaction paradigms are common enough for large reserves of programmer effort to be expended on refining the experience for the common use case. Even then I don't know how rich the interface will be; at a certain point you run into issues of simple kinematics, things like gorilla arm and the fact that the mouse is actually incredibly precise. You can get approximations, but the only thing I'd want to use for real work is "like the setup I already have, but with monitors projected into a virtual space rather than hanging off arms on my desk".
- mncharity 8y ago
  > or when someone is optimizing the heck out of their interface [...] clearly isn't what we're looking for Why? For example, people doing video or audio production sometimes create extensive personalized HID setups, with knobs and sliders and joysticks and touch screens and drawing tablets and etc. Even with expensive and clunky tech. What if everyone could easily have better at need? > gorilla arm [..] mouse [...] the only thing I'd want to use for real work is I too like a standard of "use for real work" without compromising existing tooling. I prefer a well-tuned little red ThinkPad TrackPoint to mouse, in part because it requires less movement. But even with all of Point and touchpad and mouse... What if your entire keyboard and adjacent frame could serve as a multitouch surface, without compromising keyboarding experience? (Well, compromising it beyond nice ThinkPad chicklets.) My very limited experience suggests it could be nifty. I was playing with a completely-non-functional "sanity check" prototype, got distracted surfing, and then realized I'd become puzzled and distracted... because stroking the J keycap was bizarrely failing to scroll the page. :) Suggesting that adoption might have a low barrier. I think there's a startup hoping to do a plastic overlay, like a keyboard protector, that does multitouch. > monitors And those monitors could be 3D monitors. Perhaps adding subtle Z offsets to font color and weight and spacing and alignment. And semi-transparency becomes more powerful. All 1M NPM packages fit on a 1080p screen as color dots, but with high-res 3D, they might be little lines and shapes, with over and underlays and connections. 3D datavis infusion. Having full hand pose could permit much richer hand-barely-moving gesture vocabularies... than press-hold-and-release Ctrl-Meta-Mumble.
dustbuster 8y ago
3D/VR you say? https://arcan-fe.com/2018/03/29/safespaces-an-open-source-vr-desktop/ https://arcan-fe.com/2018/03/29/safespaces-an-open-source-vr...
- mncharity 8y ago
  :) Neat. Sadly, setting that youtube video to 720p-ish conveys my experience of reading text in a consumer VR HMD, which makes them hard to prototype with. Years ago there was a Java (research?) IDE (which I'm currently failing to quickly find), with a big window as workspace, and on it, assorted task-specific little windowlets, like an individual function and datastructure and bit of call stack. I wonder what a 3D/VR IDE might look like if the "windows" were similarly disaggregated and bitesized?
  - dustbuster 8y ago
    I tried it on my vive pro, the head / position tracking was not good enough (comes from openHMD reversing project) but I found the text good enough to use with emacs. My problem was the lenses and glare from high contrast color scheme.
    - mncharity 8y ago
      On PenTile-subpixeled Vive, where only green is full resolution, I was using a green emacs theme like base16-greenscreen-dark. But even on an RGB WMR, with custom subpixel rendering and tiny fonts... with centered circular regions of non-blurry pixels only say 500 px across, it just wasn't viable for me. I've heard of an ops person being happy, focused on terminals rather than on editing code. And as I'm on linux, and focused on programming in VR rather than games, I've been using custom browser-as-compositor stacks, with only low-level lighthouse drivers or optical tracking. Part waste of time, but part road less traveled. At VR a meetup yesterday, a well-informed speaker said "you still can't do X", and I was like, I've been trivially doing X for years now. There's been a lot of that.
icebraining 8y ago
Rather than virtual 3D, why not real, physical 3D? https://dynamicland.org/ https://dynamicland.org/
- mncharity 8y ago
  Neat project. I'd love to see it combined with something like Project North Star AR. But for expert use... When smartphones were new, the user population untrained, and onboarding was critical, there was the idea of skeuomorphic user interface design. The app should resemble the real-world object. So affordances would transfer from novice user experience. The calendar app should have lined pages, a spiral binding down the center to hint at horizontal flipping, and textured leather margins. :) We don't do that anymore. For expert users, what are the odds that the severe constraints of physical reality line up nicely with the domain constraints desired for maximum productivity? How long before people start complaining "I don't like reality - no stretchy arms - you actually have to walk over to something to grab it - yuck". I joke that I currently don't do V of VR, because I almost always have camera passthrough, don't do A of AR, because I'm usually ignoring the world other than for user input, and don't do R of either, because the UI is so often aphysical. Resemblance to physical reality as design smell. Though how to "think with your hands", when HIDs are still so crude? Tracked/instrumented manipulatives could be neat.
boomlinde 8y ago
As far as I'm concerned I already have a multi-touch hand pose interface, one that's seen more than a century of improvements. Not sure what 3D would add to it. Perhaps just a larger attack vector for tennis elbow, i.e. literally crippling. When I see movies with 3D projected interfaces and people interacting by waving their hands around them I can't help laughing about the ergonomics of it. I have similar thoughts about the general applicability of VR. For as long as VR means focusing on a screen a couple of centimeters from your eyes, it's not going to be a generally useful productivity tool because of the eye strain. With a monitor I can easily turn my chair and focus on something else for a few minutes. Overall, it's my opinion that the goal of designing human interfaces should be to minimize cognitive load, eye strain and arm strain. Window management "as collaborative improv dance" with "3D hand poses" sounds like the exact opposite of that. The future of human computer interfaces IMO lies in things like replacing the mouse with something that requires less hand movement, technology like e-ink to improve legibility and minimize eye strain, mouseless window management, ergonomic keyboards...at least generally. I can still see how VR and 3D gesturing can be immensely useful for things like CAD and video games, but in general I use my computer to read and write documents (maybe a specialized use case in itself). Whether those documents fly around in a 3D space or are projected in thin air in front of me is uninteresting to that end. That said, to sort of answer your question, here is an interesting paper: https://our.oakland.edu/bitstream/handle/10323/4785/An%20Infinite-Pane%2C%20Zooming%20User%20Interface%20Window%20Manager%20and%20Survey%20of%20X%20Window%20Managers.pdf?sequence=1&isAllowed=y https://our.oakland.edu/bitstream/handle/10323/4785/An%20Inf...
- mncharity 8y ago
  > one that's seen more than a century of improvements Ah, I have tried quills a couple of times. And dip pens. It was interesting. ;) > movies [...] can't help laughing about the ergonomics of it Yes. Though they do seem to have an "outreach" role, raising awareness and interest that improvement over the familiar is possible. And moving while working, like shifting from sitting to standing desk and back, and walking, can be nice. > minimize cognitive load, eye strain and arm strain Nod. > focusing on a screen a couple of centimeters [...] eye strain The focus distance is usually either around 1 m, or infinity. Downsides include being fixed, creating vergence-accommodation conflict. Prototypes, and the new Microsoft Hololens 2 (which paints the retina), variously improve on this. > minimize cognitive load One thing I've played with, is taking a downward keyboard-cam of hands on keyboard, and overlaying it on a 3D monitor. Raising it slightly in Z seemed to allow focusing past it, making it merely bothersome, rather than untenable. Upside is it reminded me of an old Lisp Machine, with the screen telling you what various keystrokes would do in your current context, so I don't have to remember them all. > less hand movement Yes. I really like the idea of keyboard as multitouch surface. Stroke a keycap, rather than moving to a touchpad. Touch a modifier key keycap, rather than having to actually press and hold it. And optical finger tracking allowing subtle hand gestures to be commands. > read and write documents [...] 3D space or are projected in thin air in front of me is uninteresting to that end Hmm. Sometimes it's useful to print out a document, and put it up on a wall, so you can see the big picture, or talk with someone about it. If one has a task which fills the available monitor space, it's easy and common to remember what it was like with a smaller monitor, and not want to go back, but harder to imagine whether the same might happen again with further expansion. More generally, one thing that cheaper real estate might do is lower the barrier for tooling to provide speculative assistance. You're writing for an audience including someone from Great Britain, using a phrase that has different meaning there than in the US... checking that with a keystroke, or as part of a "I'm done, check it now" report, seems less attractive than "comments are available off to your side - glance if you care". Grammarly meets VR. For myself, writing documents is often preceded by a lot of graph drawing, as I figure out the structure of the domain, concept space, and presentation alternatives. I very much look forward to moving that into direct-manipulation 3D.

saulrh 8y ago

I don't think that I'd expect those interfaces would exist yet, nor are they going to be coming up relatively soon. You mostly see interfaces that have a ton of work put into them either when someone is smoothing off every possible rough edge they can find to cater to the widest possible audience (e.g. Windows Desktop), or when someone is optimizing the heck out of their interface to improve performance on a very specific use case. The second case clearly isn't what we're looking for; those interfaces will tend to be massively refined with all the cruft cut off, and in many cases the very first thing that goes is any trace of analog input or output. There's a reason that some of the most impressive feats of human bandwidth are from typists and players of rhythm games like Guitar Hero. The first case isn't going to happen until the necessary interface peripherals and interaction paradigms are common enough for large reserves of programmer effort to be expended on refining the experience for the common use case. Even then I don't know how rich the interface will be; at a certain point you run into issues of simple kinematics, things like gorilla arm and the fact that the mouse is actually incredibly precise. You can get approximations, but the only thing I'd want to use for real work is "like the setup I already have, but with monitors projected into a virtual space rather than hanging off arms on my desk".

mncharity 8y ago
> or when someone is optimizing the heck out of their interface [...] clearly isn't what we're looking for Why? For example, people doing video or audio production sometimes create extensive personalized HID setups, with knobs and sliders and joysticks and touch screens and drawing tablets and etc. Even with expensive and clunky tech. What if everyone could easily have better at need? > gorilla arm [..] mouse [...] the only thing I'd want to use for real work is I too like a standard of "use for real work" without compromising existing tooling. I prefer a well-tuned little red ThinkPad TrackPoint to mouse, in part because it requires less movement. But even with all of Point and touchpad and mouse... What if your entire keyboard and adjacent frame could serve as a multitouch surface, without compromising keyboarding experience? (Well, compromising it beyond nice ThinkPad chicklets.) My very limited experience suggests it could be nifty. I was playing with a completely-non-functional "sanity check" prototype, got distracted surfing, and then realized I'd become puzzled and distracted... because stroking the J keycap was bizarrely failing to scroll the page. :) Suggesting that adoption might have a low barrier. I think there's a startup hoping to do a plastic overlay, like a keyboard protector, that does multitouch. > monitors And those monitors could be 3D monitors. Perhaps adding subtle Z offsets to font color and weight and spacing and alignment. And semi-transparency becomes more powerful. All 1M NPM packages fit on a 1080p screen as color dots, but with high-res 3D, they might be little lines and shapes, with over and underlays and connections. 3D datavis infusion. Having full hand pose could permit much richer hand-barely-moving gesture vocabularies... than press-hold-and-release Ctrl-Meta-Mumble.

dustbuster 8y ago

3D/VR you say? https://arcan-fe.com/2018/03/29/safespaces-an-open-source-vr-desktop/ https://arcan-fe.com/2018/03/29/safespaces-an-open-source-vr...

mncharity 8y ago
:) Neat. Sadly, setting that youtube video to 720p-ish conveys my experience of reading text in a consumer VR HMD, which makes them hard to prototype with. Years ago there was a Java (research?) IDE (which I'm currently failing to quickly find), with a big window as workspace, and on it, assorted task-specific little windowlets, like an individual function and datastructure and bit of call stack. I wonder what a 3D/VR IDE might look like if the "windows" were similarly disaggregated and bitesized?
- dustbuster 8y ago
  I tried it on my vive pro, the head / position tracking was not good enough (comes from openHMD reversing project) but I found the text good enough to use with emacs. My problem was the lenses and glare from high contrast color scheme.
  - mncharity 8y ago
    On PenTile-subpixeled Vive, where only green is full resolution, I was using a green emacs theme like base16-greenscreen-dark. But even on an RGB WMR, with custom subpixel rendering and tiny fonts... with centered circular regions of non-blurry pixels only say 500 px across, it just wasn't viable for me. I've heard of an ops person being happy, focused on terminals rather than on editing code. And as I'm on linux, and focused on programming in VR rather than games, I've been using custom browser-as-compositor stacks, with only low-level lighthouse drivers or optical tracking. Part waste of time, but part road less traveled. At VR a meetup yesterday, a well-informed speaker said "you still can't do X", and I was like, I've been trivially doing X for years now. There's been a lot of that.

icebraining 8y ago

Rather than virtual 3D, why not real, physical 3D? https://dynamicland.org/ https://dynamicland.org/

mncharity 8y ago
Neat project. I'd love to see it combined with something like Project North Star AR. But for expert use... When smartphones were new, the user population untrained, and onboarding was critical, there was the idea of skeuomorphic user interface design. The app should resemble the real-world object. So affordances would transfer from novice user experience. The calendar app should have lined pages, a spiral binding down the center to hint at horizontal flipping, and textured leather margins. :) We don't do that anymore. For expert users, what are the odds that the severe constraints of physical reality line up nicely with the domain constraints desired for maximum productivity? How long before people start complaining "I don't like reality - no stretchy arms - you actually have to walk over to something to grab it - yuck". I joke that I currently don't do V of VR, because I almost always have camera passthrough, don't do A of AR, because I'm usually ignoring the world other than for user input, and don't do R of either, because the UI is so often aphysical. Resemblance to physical reality as design smell. Though how to "think with your hands", when HIDs are still so crude? Tracked/instrumented manipulatives could be neat.

boomlinde 8y ago

As far as I'm concerned I already have a multi-touch hand pose interface, one that's seen more than a century of improvements. Not sure what 3D would add to it. Perhaps just a larger attack vector for tennis elbow, i.e. literally crippling. When I see movies with 3D projected interfaces and people interacting by waving their hands around them I can't help laughing about the ergonomics of it. I have similar thoughts about the general applicability of VR. For as long as VR means focusing on a screen a couple of centimeters from your eyes, it's not going to be a generally useful productivity tool because of the eye strain. With a monitor I can easily turn my chair and focus on something else for a few minutes. Overall, it's my opinion that the goal of designing human interfaces should be to minimize cognitive load, eye strain and arm strain. Window management "as collaborative improv dance" with "3D hand poses" sounds like the exact opposite of that. The future of human computer interfaces IMO lies in things like replacing the mouse with something that requires less hand movement, technology like e-ink to improve legibility and minimize eye strain, mouseless window management, ergonomic keyboards...at least generally. I can still see how VR and 3D gesturing can be immensely useful for things like CAD and video games, but in general I use my computer to read and write documents (maybe a specialized use case in itself). Whether those documents fly around in a 3D space or are projected in thin air in front of me is uninteresting to that end. That said, to sort of answer your question, here is an interesting paper: https://our.oakland.edu/bitstream/handle/10323/4785/An%20Infinite-Pane%2C%20Zooming%20User%20Interface%20Window%20Manager%20and%20Survey%20of%20X%20Window%20Managers.pdf?sequence=1&isAllowed=y https://our.oakland.edu/bitstream/handle/10323/4785/An%20Inf...

mncharity 8y ago
> one that's seen more than a century of improvements Ah, I have tried quills a couple of times. And dip pens. It was interesting. ;) > movies [...] can't help laughing about the ergonomics of it Yes. Though they do seem to have an "outreach" role, raising awareness and interest that improvement over the familiar is possible. And moving while working, like shifting from sitting to standing desk and back, and walking, can be nice. > minimize cognitive load, eye strain and arm strain Nod. > focusing on a screen a couple of centimeters [...] eye strain The focus distance is usually either around 1 m, or infinity. Downsides include being fixed, creating vergence-accommodation conflict. Prototypes, and the new Microsoft Hololens 2 (which paints the retina), variously improve on this. > minimize cognitive load One thing I've played with, is taking a downward keyboard-cam of hands on keyboard, and overlaying it on a 3D monitor. Raising it slightly in Z seemed to allow focusing past it, making it merely bothersome, rather than untenable. Upside is it reminded me of an old Lisp Machine, with the screen telling you what various keystrokes would do in your current context, so I don't have to remember them all. > less hand movement Yes. I really like the idea of keyboard as multitouch surface. Stroke a keycap, rather than moving to a touchpad. Touch a modifier key keycap, rather than having to actually press and hold it. And optical finger tracking allowing subtle hand gestures to be commands. > read and write documents [...] 3D space or are projected in thin air in front of me is uninteresting to that end Hmm. Sometimes it's useful to print out a document, and put it up on a wall, so you can see the big picture, or talk with someone about it. If one has a task which fills the available monitor space, it's easy and common to remember what it was like with a smaller monitor, and not want to go back, but harder to imagine whether the same might happen again with further expansion. More generally, one thing that cheaper real estate might do is lower the barrier for tooling to provide speculative assistance. You're writing for an audience including someone from Great Britain, using a phrase that has different meaning there than in the US... checking that with a keystroke, or as part of a "I'm done, check it now" report, seems less attractive than "comments are available off to your side - glance if you care". Grammarly meets VR. For myself, writing documents is often preceded by a lot of graph drawing, as I figure out the structure of the domain, concept space, and presentation alternatives. I very much look forward to moving that into direct-manipulation 3D.