15 ms·

A brief rant on the future of interaction design (2011)

335 points by li4ick 7y ago

tambourine_man 7y ago
2011
cvaidya1986 7y ago
This is great! Really shows what we have today is a reductionist approach to only using our finger on a 2D surface as opposed to utilizing all our motions, muscles and senses to interact with technology in 3D space.
- zdragnar 7y ago
  Personally, I think it's a bit asinine. Rather than decry anything that resembles what we have today as old-fashioned and not futuristic, it should first be asked "why are screens so common today?" We've been using our hands for years- keyboards, trackballs, mice, joysticks, custom wheels for driving games, the nintendo light gun for duck hunt, and so forth. Take that- and combine it with something else that should be jumping out at you from the article: every picture with a hand shows it holding a different object. Screens are multi-purpose. They are perfect for scenarios where one control must meet many needs (compared to the pictured hammer, which at best meets two). Let's say that we could imagine some multi-purpose, tactile, 3 dimensional interface... say, like a hologram from Iron Man or Star Trek. Now compare that to the article's mention of Alan Kay's doodle pre-integrated circuit. The difference is huge... we don't even have the physics knowledge to describe how a tactile hologram could exist at safe handling energies, let alone in open air, as described by those fictions. Comparatively, Alan Kay combined things that already existed... battery, circuits, screen and keys, and imagined what could happen if they were smaller and portable. We had things much more similar to what he drew decades before we had the ipad; the ipad is merely an incremental improvement over some of my own childhood toys, not to mention what was already available a few years after his own drawing, such as the Speak and Spell ( https://en.wikipedia.org/wiki/Speak_%26_Spell_(toy)#/media/File:Speak_&_Spell_(original_style).jpg https://en.wikipedia.org/wiki/Speak_%26_Spell_(toy)#/media/F... )
  - archgoon 7y ago
    Bret Victor pointed out Alan Kay had to wait 50 years before something like the Dynabook was possible. And with a bit of creativity, multi-use tactile interfaces can be accomplished today. https://en.wikipedia.org/wiki/Nintendo_Labo https://en.wikipedia.org/wiki/Nintendo_Labo
    - Palomides 7y ago
      here's a plausible fantasy, some software will include 3D-printable files describing a physical interface device, which can have some standard electronics plugged in as long as it doesn't need to be hard realtime and infinitely reconfigurable, it's very feasible like, imagine a game in which you hammer nails, you would 3D print a hammer, stick in an accelerometer module and mcu/battery/bluetooth module point being, there are definitely ways to make interfaces that are now, or will become, really usable
  - jacobolus 7y ago
    > We've been using our hands for years-keyboards, trackballs, mice, joysticks, custom wheels for driving games, the nintendo light gun for duck hunt, and so forth. These are generally quite poor interfaces compared to what is conceivably possible. A precise pressure-sensitive stylus and/or multiple fingers on a responsive high-resolution display are much richer than these: higher bandwidth of a more directly relevant input. But the finger/stylus on screen is strictly 2-dimensional without any haptic feedback. Even such tasks as rotating and moving a computer model of a single 3-dimensional object are extremely cumbersome and unintuitive using mouse, keyboard, joystick, steering wheel, etc. Now imagine using those tools for e.g. controlling a simulated puppet in real time. Pretty much impossible. And yet people control physical puppets with amazing precision and flexibility with some strings and wooden sticks. And that’s just moving an already existing collection of shapes. What if I want to design a new shape, say a newly imagined knot, or a new playground climbing structure, or a chair, or the face of an imagined deep-sea creature. Tools like pipe cleaners, modeling clay, paper & scissors, wooden sticks and a hot glue gun, ... are dramatically richer and more efficient than any of these electronic tools.
  - p_l 7y ago
galonk 7y ago
Seems pretty easy to write an essay saying "screens are old news, let's use our hands!" containing not a single word about how that would work, technically or design-wise. I'm going to write an essay saying "screens are old news, let's use free-floating holograms like in Iron Man!" with no indication how that would even be possible, and see if I can get on HN.
- Pulcinella 7y ago
  You didn't offer a solution. Yes, that's why I called it a rant, not an essay — it describes a problem, not an idea. (And FWIW, that's not the sort of thing I typically publish, or want to.) The solution isn't known, and I didn't think making stuff up would help. The point of the rant was that the solution will be discovered through long research, and that research won't happen without an awareness of the problem. My intent was to hopefully coax a few potential researchers and funders into tangible interfaces, dynamic materials, haptics, and related fields that haven't been named yet. If that happens, then mission accomplished. http://worrydream.com/ABriefRantOnTheFutureOfInteractionDesign/responses.html http://worrydream.com/ABriefRantOnTheFutureOfInteractionDesi...
- maxerickson 7y ago
  He's been working on https://dynamicland.org/ https://dynamicland.org/ since that time. So there you go.
- grumpy8 7y ago
  The author of the essay is the founder of https://dynamicland.org https://dynamicland.org which implements what he's talking about in this article.
  - galonk 7y ago
    None of the things BV has worked on -- smart tiles or putting pieces of paper under a camera/projector -- do ANYTHING AT ALL like what he's talking about here. They don't change their size, shape, weight, feel etc. in response to data. Because that would be incredibly hard. I don't think you should get points for saying "we should do this impossible thing, that would be better." I must admit I find BV's writing to be really facile... like he'll compare a piano app on the iPad with a real piano and shit on the entire idea of the iPad... without addressing the obvious point that an iPad can be a million things other than a piano.
pgodzin 7y ago
The benefit is of "Pictures Under Glass" is the flexibility - it is extremely easy to change the picture to align to almost any use case. It is extremely difficult to have the same flexible interface with something that is tactile and can manipulate its own shape to be appropriate for the task.
- icebraining 7y ago
  In 1968 the iPad also seemed extremely difficult, though.
dmix 7y ago
Aw, I was hoping it was a new essay by Bret. Needs (2011) in the title.
- Already__Taken 7y ago
  Brett victors talk innovating on principal(https://vimeo.com/36579366 https://vimeo.com/36579366) and more so actually Greg Wilsons talk from the same conf (https://vimeo.com/9270320 https://vimeo.com/9270320) marked a pivotal moment for me in my work ethic. It really motivated me to expect better from the tools i use around every day.
  - chrisweekly 7y ago
    Similar rxn to those talks. Also, if ever someone knew how to write an "evergreen" post (ie, one that holds up over time), it's Bret Victor. His "Ladder of Abstractions" is a great example: http://worrydream.com/#!2/LadderOfAbstraction http://worrydream.com/#!2/LadderOfAbstraction
    - mcphage 7y ago
      Also his essay Magic Ink: http://worrydream.com/MagicInk/ http://worrydream.com/MagicInk/
    - kragen 7y ago
      I really enjoy his writing, but I think he has a way to go before he's in the same league as Homer, Plato, Lao Tse, and Shakespeare!
      - chrisweekly 7y ago
        haha, those guys sure knew how to write a good blog post!
        kragen 7y ago
        Yup! They got people still clicking on their posts hundreds or even thousands of years later
simulate 7y ago
I remember this Bret Victor piece from 2011 and I was thinking about it recently when Google announced Soli for the Pixel 4. https://www.cnet.com/news/project-soli-is-the-secret-star-of-googles-pixel-4-self-leak/ https://www.cnet.com/news/project-soli-is-the-secret-star-of... Soli and other gesture-based technologies seem like at least an incremental move to using motion that Bret Victor was advocating for in 2011.
- Palomides 7y ago
  to me that seems like the same problem, since you still don't touch anything, the only feedback is the computer doing something (or not) that said, I'm definitely curious to see how it performs in the wild
bsanr2 7y ago
I noticed this while experimenting with VR and hand-tracking. Without tactile or precise auditory feedback, the experience was hollow, even with relatively high-fidelity visuals and tracking. You can never feel like you are somewhere else - or something immaterial is with you - until a full sensory experience coalesces, even if it's "low resolution." Until then, the result is landing in an experiential uncanny valley. On another note, we should also be looking at accessibility when approaching this research. Contrary to that last little bit at the end, not everyone has full use of their bodies, or even of their hands. I just finished listening to an NPR segment about inner-city youth dealing with permanent disability as a result of gun violence, and the unique challenges presented to them. Perhaps with people like them in mind, the mass solutions we eventually end up with will be better for focusing farther upstream than nerve endings.
- gfodor 7y ago
  Hands-in-air interactions in VR I agree lack the tactile feedback necessary to avoid it feeling hollow. However, I've often advocated that the "ideal" VR input system in 2019 is: - Freely tracked dominant hand, used for fine-grained control/pointing, expressing body language, and grab/release-like mechanics (which work decently for certain kinds of virtual objects/interactions, like throwing stuff) - Tracked 6dof controller in non-dominant hand, similar to the existing controllers except with a fully capable twiddler-like button grid for doing fast symbolic entry. For anything where tactile feedback is core, move it onto the controller hand, where the hardware buttons can provide a more reasonable proxy for things like press-and-release or sustained-holding in terms of motor interactions. (Haptics are still pretty bad on all other accounts, but controllers at least give you pressure feedback under your finger tips, instantaneous response, etc.) With the above setup, I suspect you could perform most knowledge worker oriented tasks with similar efficiency to a mouse and keyboard, and also perform existing VR-centric use cases like 3d modelling/manipulation, avatar locomotion, full body social interaction, etc. The current two-controller or dual hand-tracked paradigms fall short on the former. With the introduction of hand tracking capabilities on Quest coming, it seems that we're inching up to mainstream VR devices having the capability to deliver this, except the controllers still lack anything close to keyboard-level discrete inputs. However, once Quest hand tracking launches, it may be possible to use an off-the-shelf twiddler for this if the SDK exposes the ability to track in-hand tools (similar to the Leap Motion SDK.)
melling 7y ago
I almost made a comment in this article about using Deep Learning and Motion Tracking to have computers better understand our gestures so we can interact visually. https://news.ycombinator.com/item?id=21115863 https://news.ycombinator.com/item?id=21115863 At some point AR and VR will finally provide us the overwhelming need. Then we’ll wonder why we didn’t think of it sooner. At the moment, we’re stuck in the “a keyboard and mouse are better” stage.
- weego 7y ago
  You can't just drop the idea of Deep Learning into a problem as a magical solution. What does visual interaction entail and what is stopping that from having already been trialled during one of the previous (and current) attempts to get vr to work for people?
  - melling 7y ago
    If you can use Deep Learning and CNN’s to drive a car, you can probably use it to recognize different hand motions. [Update] I found a link where someone tried to do a simplified version of this: https://towardsdatascience.com/tutorial-using-deep-learning-and-cnns-to-make-a-hand-gesture-recognition-model-371770b63a51 https://towardsdatascience.com/tutorial-using-deep-learning-...
    - mkl 7y ago
      It's certainly possible to recognise hand motions (see e.g. Leap Motion), but I think hand motion recognition is pretty hard to use effectively without haptic feedback telling you what you're touching and where. Ideally actuated like [1] but for your whole hand, not just a stylus. I don't think technology is anywhere near that. The device in [1] pushes back against you, physically stopping you from moving through virtual objects. [1] https://www.3dsystems.com/haptics-devices/touch https://www.3dsystems.com/haptics-devices/touch
      - erikpukinskis 7y ago
        You don’t need haptic feedback, just good feedback. There’s plenty of bits in an audio stream alone to give the brain the control feedback it needs. You’d be surprised how few degrees of freedom we use for grasping.
jct3u 7y ago
The video anchoring the article shows as unavailable for me. Is there an alternate place to view it?
- lancebeet 7y ago
  Same here, I googled the url of the embedded video and it seems to be a concept video by Corning: https://www.youtube.com/watch?v=6Cf7IL_eZ38 https://www.youtube.com/watch?v=6Cf7IL_eZ38
vortico 7y ago
This is probably my all-time favorite article on the internet. I've brought it up in conversation dozens of times when referring to ways to control computers. It's why I don't really use phones for anything beyond answering phone calls and handling urgent tasks when I'm not at home/office. I just feel like I have very little control over these devices, unlike the massive control with a keyboard or mouse, or even better, a fork and spoon when eating dinner (although despite its high degree of control, you can't control a computer with them). Tech innovators and its users/consumers need to think outside of the box more often for ways to control computers. Given a physical UI device, you can place it on a spectrum of "computer-friendly to human-friendly". Touch screens are computer-friendly because it's trivial for computers to understand and developers to interact with. The downside is that users feel distanced from their control, so it is difficult to use efficiently. But developers are smarter than users at UI---the difficulty should be put on them instead, to control their software/OS. I've imagined things like stress balls with "electric muscles" for feedback to control various applications, a cube with the ability to change its texture with suction/servos, and things like https://roli.com/products/seaboard https://roli.com/products/seaboard or https://www.expressivee.com/buy-touche https://www.expressivee.com/buy-touche except for general computing applications rather than music performance. Who knows what could be invented if customers could be convinced their computing experience can be improved with better physical interfaces.
- phaedrus 7y ago
  As an engineer I finally got to take the training course our techs go through and experience how a tech uses the software which we (engineers) developed and maintain. It was eye-opening. One particular example relevant to this discussion is the old version of the equipment had physical potentiometers which the tech would adjust to get a desired meter reading. The new equipment has you just type a number into a textbox in a Windows program. BUT - it doesn't even provide a slider or up/down arrows, you have to play The Price Is Right inputting guessed over/under numbers until a measured output in a second text box matches a desired value. It's clear that when the transition was made from physical controls to software based controls, not enough thought was given to ergonomics and how it would be used by an actual human.
carapace 7y ago
Tiny grey sans-serif font means you hate my eyes. Extra-ironic on a rant about interaction design.
tboyd47 7y ago
Instead of a single spectrum from computer- to human-friendliness, picture a triangle with the third vertex being "task-friendliness" or task specificity. The author is advocating a return to human-friendliness, but what he describes is more like task-friendliness. I've become somewhat obsessed with ergonomics after a long battle with chair pain. In my mind, true human-friendliness does not require new interfaces for every task (although it may require a slightly different interface for each human). Every computing task may very well be handled best by a person sitting in a chair at a workstation. That wouldn't be bad design IMO; computers are, in fact, general purpose. If we can get to a point where each computer user is able to use their computer comfortably for 8 hours straight, without pinching a nerve or a blood vessel (yes! it's possible!), then I would consider that a human-friendly design. I'm not talking about iPads here - the author is totally right that these devices don't seem to be made for human beings.
- jacobolus 7y ago
  As far as I understand, remaining in a static position for 8 hours straight during the day and another 8 hours straight while asleep is pretty bad for many parts of the body irrespective of what that position happens to be. People need to be occasionally changing position, moving about, getting their circulatory system working, getting their muscles working, focusing their eyes at a longer distance, etc. We can improve people’s experience with sit–stand desks, frequent short breaks, periodic longer breaks, a daily commute requiring walking or other exercise, etc., but reducing the total number of hours people are staring at screens every day would also be generally helpful. It isn’t all that great to exclusively work at even the most ergonomic possible computer workstation.
  - tboyd47 7y ago
    There's truth to what you're saying, but it implies you have to accept a certain level of pain or physical discomfort if you work in a seated position, and the only way to manage the pain is to get out of your seat. But there are types of pain that can't be managed that way. Those injuries have to do more with sustained muscle tension caused by bad equipment than just sitting, and will continue to get worse until you change your equipment to eliminate the tension.
guitarnick 7y ago
Though this has been discussed many times, I must say the trend of using touchscreens to replace dials and buttons on cars is extremely worrying. It’s entirely an aesthetic and cost saving decision at the expense of safety and UX.
- arkh 7y ago
  > I must say the trend of using touchscreens to replace dials and buttons on cars is extremely worrying Most are going back on that. It is just that what is currently being sold was developped years ago.
  - reaperducer 7y ago
    I'm lucky that my car was made in the in between period, so all touch screen controls have physical controls as well. I use the physical controls, especially when driving. But my navigatrix prefers the screen, primarily because she likes the visual feedback. (Not enough of the physical controls in my car have things that light up to let you know something's happened.)
- flowerlad 7y ago
  Ford brought back physical buttons. See: https://www.theverge.com/2013/6/17/4437310/ford-dashboard-buttons-sync-myford-touch https://www.theverge.com/2013/6/17/4437310/ford-dashboard-bu...
- hunterloftis 7y ago
  My wife and I were recently searching for a new car. One of the model's reviewers dinged it for using "dated" physical dash controls instead of the present fad of a single large touchscreen. We entered that into the "positives" section of our spreadsheet.
- anonytrary 7y ago
  My friend has a BMW and the touchscreen is awful. I can't imagine using that thing to do anything while I'm driving. It looks like it was designed for someone to use on the comfort of their bed, where they can afford to mis-click, not know where the hell to find a page, etc. The only reason I absolutely need a screen in a car is for visibility. Pretty much all the risk in parking/reversing is removed when your car has external cameras. The other functions don't make much sense to me. If you wanna play video games and watch movies while you're driving, it better be a smart car.
arkh 7y ago
I'd like to extend this to fucking VR. All VR experiences lack one thing: full body haptics. Take something inspired from the "Pacific Rim" cockpits ( http://i.imgur.com/DLze6j7.jpg http://i.imgur.com/DLze6j7.jpg ) so you can have some contraption giving you a huge freedom of movement and haptic feedback everywhere. That'd make some games a lot more immersive and physical.
- erikpukinskis 7y ago
  The reason games don’t feel physical is not because of which senses are being stimulated, it’s how they are stimulated. If the game is just waiting for your hand to enter a bounding box, that’s a 0-dimensional interaction, which feels like nothing. If you could hear your fingers getting close to things, and see the angles between your body and other surfaces, sense through visual texture the relationship your body has with different fields, you would register those things a “physical” experiences. Touch doesn’t feel like touch because it’s your fingers, it feels like touch because it’s rich and continuous and textured and perfectly grounded in body-space. You can make proprioaudiovisual experiences like that too, it’s just hard and there are easier ways to make video games that make money. The science is clear that our brains don’t care which sense data comes from, they just care about the structure of the data. Although there are anatomical structures that make some senses better at processing one structure than another, they don’t prevent the brain from using those senses for other schemas, if the environment demands it (injury, prosthetics, etc)
slotkin 7y ago
ahh was really hoping this was something new from Bret
Chris_Newton 7y ago
It’s sad that since this article was first written, the trend for dumbing down both the hardware and software we use every day has only continued. When I was a kid, I used to love experimenting with programming. I first learned on a ZX81, with a magnificent 1KB of RAM, typing in simple games programs from books every session because there was no storage device to save them. That experience, that joy of being able to create something fun, sparked an interest in what these amazing technologies we have invented can do. Some of my slightly younger friends were lucky enough to have more powerful systems like the BBC Micro available by the time they reached that stage. Those were brilliant because you could connect anything to them. When they were writing simple LOGO programs at school, they didn’t just draw a circle on the screen, an actual mechanical turtle with an actual pen would draw a circle on a real piece of paper, right before their eyes. Now I want to offer that same joy and intrigue to the next generation of my family. Today’s ubiquitous computing devices are phones and tablets, each with numerical specs many orders of magnitude bigger than that ZX81. That little boy typing in listings from a book now has multiple decades of professional programming experience to share. And yet, I can’t sit down with my own child and write even the most simple game on those devices, because for all their theoretical power, they lack even a rudimentary programming interface. In some cases, I can’t even write a game myself on another system and port it, because the whole ecosystem is closed off. How is it that in a time when children seem, often all too soon, to be carrying around more processing power in their pockets than a supercomputer had when I grew up, they still can’t enjoy the sense of freedom and discovery that I experienced with my little ZX81 and its 1KB of RAM and no storage device in the 1980s?
- dr_dshiv 7y ago
  Yes! Have you considered buying a zx81 for your kid? Or better, a logo turtle?
- fragmede 7y ago
  They can! If you're on iOS, have you tried Pythonista? It's $8 on the app store, uses Python as the programming language, features an IDE, and a UI library so budding programmers can write video games. For those on Android and the technically minded, a rooted Android tablet offers far more options for on-device programming. There's likely an equivalent to Pythonista in the Google app store as well. (No affiliation; I'm sure there are other apps that are similar, that's just the one I know of.)
badsectoracula 7y ago
Back when touchscreens started to become popular i remember some talks about a 'tactile overlay' to touchscreens, essentially a grid of tiny bumps (pixels, though at a much lower resolution) that would 'pump up' and 'flatten' just enough to provide a tactile feel to the otherwise glassy screens. What happened to that?
_bxg1 7y ago
There are some gaps in reasoning, but overall I think the point stands. The biggest omission is the key advantage of "Pictures Under Glass": they can be defined purely in software. Because of that, I don't think they'll ever stop serving a role in our device interactions. No amount of space-age physical device design could've supported the smartphone's explosion of apps without pictures under glass or some equivalent. With that said, we could definitely benefit from moving some of the more constant/widespread interactions back to physical controls. Buttons and switches on smartphones are always appreciated. And don't even get me started on cars, where there's little need for a vibrant developer ecosystem and a ton of need for non-visual interface comprehension. I also think the Apple Watch's "crown" is one of the more interesting recent examples of a tactile interface that doesn't sacrifice open-ended software development.
- tabtab 7y ago
  Re: Buttons and switches on smartphones are always appreciated. I disagree, I'm always accidentally bumping or pushing them. Plus, they usually do tasks that are not my top tasks. I'd like to reassign some of them to my favorite actions.
- buckminster 7y ago
  Surely the dream is to have a programmable tactile interface. Perhaps you could program the height or texture of parts of the screen. But I have no idea how this would work!
neilobremski 7y ago
This brings to mind the phone ringing in the middle of the night. You pick it up and add mumble into the receiver and the conversation begins. But no, now the phone rings in the middle of the night (assuming it isn't text / Slack / email / etc.) and you pick it up and then try to focus your eyes on the screen to figure out WHAT is happening and then HOW to answer. It seems the massive leap forward in the CAPABILITY of our phones has actually caused them to stumble backwards in PRACTICAL usability.
- rubidium 7y ago
  So, has anyone made a “old school smartphone dock”? What I want is a dock I can put my smartphone into and then the old phone interface takes over. Preferably one with links to 1-2 other docks in the house so all three ring if I get a call. Pick up the physical receiver, say hello. Talk. Say goodbye, hangup. No screen interaction at all.
  - TeMPOraL 7y ago
    That's... genius! It's so obvious now that you wrote it. I'd love to have one at my desk, and another at my nightstand.
  - bblough 7y ago
    The cordless phone system I have lets you make/receive calls via your cell phone from any handset in the house. It connects via Bluetooth, so isn't a dock, but the functionality is similar to what you're suggesting. Mine is an older Panasonic system, but I'd be surprised if other manufacturers didn't offer something similar.
    - rubidium 7y ago
      Like this? https://shop.panasonic.com/cordless-corded-telephones/cordless-telephones/KX-TG7875S.html https://shop.panasonic.com/cordless-corded-telephones/cordle...
      - 7y ago
tabtab 7y ago
I predict the UI of the future will be via brain implants that allow one to control an app with just thoughts. It may become a necessity in order to keep up with other countries that may have less qualms about the ethical side of surgery and intrusion.
justinmchase 7y ago
> Are we really going to accept an Interface Of The Future that is less expressive than a sandwich? Great line.
- tabtab 7y ago
  Probably written by a hungry author.
modernerd 7y ago
> With an entire body at your command, do you seriously think the Future Of Interaction should be a single finger? What if you don't have an entire body at your command? The plus side of “pictures behind glass” is that it's a fairly well standardized interface — the software has to adapt to that homogenized hardware, instead of you fighting to adapt to the hardware. (Or, more accurately, you adapt to it once, instead of once for every type of object you want to interact with.) If interactive experiences all start to require a good range of motion, bodily sensitivity, and ability to instinctively interpret the interface, there's a risk it could be incredibly alienating for many. Unless we design with that consideration from day one, it could make adaptivity harder than it already is. I went looking for Bret Victor's take on this question because I was certain he would have thought of this already. So far I only found this: “Channeling all interaction through a single finger is like restricting all literature to Dr Seuss's vocabulary. Yes, it's much more accessible, both to children and to a small set of disabled adults. But a fully-functioning adult human being deserves so much more.” http://worrydream.com/ABriefRantOnTheFutureOfInteractionDesign/responses.html http://worrydream.com/ABriefRantOnTheFutureOfInteractionDesi... I find it sad to read that. Those with access issues deserve so much more too. There's already a huge access gap. If we're going to champion new modes of interaction, we should fight hard not to make the gap bigger still.
- tabtab 7y ago
  Re: Maybe, if you don't have an entire body at your command. "Boss, I'm not disco dancing, I'm preparing our Powerpoint presentation for the big meeting." Boss: "Please, do the Reboot Shuffle."
- hinkley 7y ago
  I want someone with the skills of Brett Victor to solve the problem of visualization and maintenance tools for change over time. You solve that problem and there's a world open to you for using things other than flat text files as the system of record. Without those tools, it's all talk. If we're lucky. If we're not lucky it's a death march toward a horrible, horrible boondoggles. We are already trying to visualize four (or more) dimensional code with 2.5D projections (length, width, and a few colors). You give me a tool that can use 3 or 3.5 dimensions and the world opens up to us. Don't forget to write a merge tool.
gdubs 7y ago
A few things come to mind. First, I've been reading a lot on the science behind Montessori education, and there's a lot to support the idea that our brains are much more active when we're physically interacting with something. Would be logical to assume that the richer the physical interaction, the more meaningful the connection. Second, reading Jaron Lanier's somewhat recent book on VR I was struck by how far ahead of its time the 'data glove' was. Early VR may not have had the dazzling eye-candy of today's graphics, but on interactivity we just seem to be catching up. (As an aside, I never realized the Nintendo PowerGlove was basically Data Glove 'lite'; developed by VPL.) Edit: Third, I recently read Carl Sagan's "The Dragons of Eden" which he wrote in the 1970s. It's about the evolution of the human mind. It's SO worth reading today, even if its outdated and proven wrong at times by contemporary discoveries. One thing that really stood out was this idea that it wasn't so much that humans evolved and then invented tools – but that perhaps tools shaped our evolution as much as we shaped tools. Invent a simple tool -> use the tool -> brain develops as a result -> invent even more complex tools -> etc.
fragmede 7y ago
There's a (2011) missing from the title here, at least according to the title graphic.
- dang 7y ago
  Added now!
gitgud 7y ago
I feel my Nokia 3210 in highschool had a much more intuitive UI than the latest iPhone... and it was trivial to use without looking at it, since your hands could navigate the keys... This isn't possible anymore with sliding pictures under glass...
JesseAldridge 7y ago
Embodied cognition is definitely a thing: https://en.wikipedia.org/wiki/Embodied_cognition https://en.wikipedia.org/wiki/Embodied_cognition And in software development, the primary bottleneck is thinking. Anything you can do to enable greater clarity of thought is going to have huge leverage. So I think this is a super promising angle. Bret's actively developing his ideas at DynamicLand: http://dynamicland.org http://dynamicland.org But clearly it will be quite some time before that tech becomes viable in the real world. Now I'm wondering if there are simpler ways to use the rest of my body to facilitate thinking. Like, I can't mount lasers and network connected cameras on the ceiling... but maybe there's some software that can use a webcam to do simple gesture recognition mapped to bash scripts or something like that.
dang 7y ago
A thread from 2013: https://news.ycombinator.com/item?id=6325996 https://news.ycombinator.com/item?id=6325996 Discussed at the time: https://news.ycombinator.com/item?id=3212949 https://news.ycombinator.com/item?id=3212949
Kenji 7y ago
Touch screens are made for consumption, rich interfaces (with tactile feedback, like keyboards, mice, etc.) are made for creation. Most people just consume on their digital devices. Therefore, touch screens are enough. It's as simple as that. People who want rich interfaces are in a minority but I don't think rich interfaces ever go away because someone has to create all the stuff for digital devices.
cryptozeus 7y ago
I think we are in transition faze. Current future ideas assumes that we will have same interface and hand driven intersections which may not be true. Increase of IOT devices and voice controlled interfaces have different story to tell.
syrrim 7y ago
I think maybe the principle is that we want to be able to transfer as many bits/second of data as possible. We have the computer->human interface down, because screens can already emit more information than humans can process. But human->computer we can only go so fast. This is partly limited by how many fingers we use to do the input. I can type faster on a keyboard than a screen. But I think the main issue people are worried about is that humans can only think so fast. So if the interface requires asking humans for input too much, it will take forever. So instead you try and guess what the humans want. And if you do this well enough, then the input method doesn't matter. Some people say they can type better on their phones because they can use swiftkey. So I think most visions of the future are thinking about AI, and intelligence, and asking "what will we be able to guess without asking the human first?" And this is where they think interaction design is going. Another point of fact. Such a machine requires less bps from the human. That is, it requires them to do less thinking, which most people will prefer. And if you're trying to predict the future, "machines will require less thinking (but be more mass market)" is a pretty safe bet. Sandwiches are very fancy things, as everyone knows. But I expend a lot of mental effort getting the cheese slices in the right thickness. I'm sitting there focusing on making sure I don't deflect the knife too deep or too shallow. This is a very fancy task. But many people prefer specialized cheese slicing devices, because they makes more precise slices, and because they require less mental effort. So expect UI to follow the same trend, from general tools requiring complex input to particular tools requiring simple input.
lkrubner 7y ago
The article focuses on the use of hands, but a visionary future technology might focus more on voice and NLP. This is something I have worked for some years: http://www.smashcompany.com/technology/the-advantages-of-a-natural-language-processing-interface http://www.smashcompany.com/technology/the-advantages-of-a-n...
Animats 7y ago
Victor is one of the people behind Dynamicland, the "live paper"/"room is the computer" startup that was shut down recently. That was on HN last week or so. Fancier input devices seem to have come and gone. They peaked in the 1990s, when you could see a lot of them at SIGGRAPH. My favorite was the magnetically levitated sphere in a bowl. It was like a 3D joystick/trackball with force feedback. It was really cool. It never sold. There were lots of gadgets like that. An animator friend had a workstation with a keyboard, a knob box, a button box, a joystick, a trackball, a tablet, and two screens. Some of the people doing Jurassic Park had a model dinosaur where you could move all the joints, the computer could read the angles, and the on-screen image moved to match. None of this ever caught on. Even 3D joysticks are rare. Game controllers with two joysticks caught on, but those joysticks are abstractions of what's on screen, as, for example, steering, not direct interaction. I tried Jaron Lanier's original gloves-and-goggles VR system. You couldn't do much with the gloves. That was pretty much true in later glove systems. Autodesk fooled around with VR in the 1990s, but determined that gloves and goggles were not going to make CAD easier. Lack of force feedback is a huge problem with gloves. Without force feedback, it's like trying to work in oven mittens. Much of human precision is tactile feedback. Without that, precision work is slow and tiring. As everyone who's soldered surface mount parts under a microscope knows. Back in the 1990s, when I was working on collision detection and physically based animation, I considered building an input device I called "The Handle". The Handle was to be a jointed arm, like a robot arm, with a grip handle on the end as an input device. A servomotor system (or, for cost reasons, I was thinking brakes only back then) would provide tactile feedback. The handle itself would have the ability to split, like pliers, so you'd have something to squeeze. The Hammer could potentially simulate pliers, tongs, wrenches, hammers, etc. Do simulated auto repair. Assemble Meccano. This would have been what Victor is calling for. Could it be built? Yes. Would it sell in volume? No. That's the problem.
- Nition 7y ago
  I tried out the 3D Systems Touch Haptic Device a few years ago, which sounds a bit like a simpler variant of your "handle".[1] It was quite good. Feels really strange, but good, to be interacting with things "inside the screen". [1] https://www.matterhackers.com/store/l/3d-systems-touch-haptic-device/sk/M12PCJ5U https://www.matterhackers.com/store/l/3d-systems-touch-hapti...
keenmaster 7y ago
Haptic, resistive gloves will merge the tactile with the digital in VR applications. Just like touchscreens, they will serve as a single device that can simulate an infinite number of different controls. Give 5-10 years for the tech to miniaturize and mature. https://haptx.com/ https://haptx.com/
dreamcompiler 7y ago
This is why I dislike touch screens in cars so much: You have to look at them. We have 100 years of experience with car user interfaces that don't require you to take your eyes off the road, but we threw all that away in favor of these shitty pictures under glass.
naiveai 7y ago
The condescending tone of this article does not help it whatsoever. Not to mention the huge gaps in logic that cause it to jump from merely a misguided attempt to rail against well established conventions for interfaces to an active argument for those interfaces still being good.

335 points by li4ick 7y ago

2011

cvaidya1986 7y ago

This is great! Really shows what we have today is a reductionist approach to only using our finger on a 2D surface as opposed to utilizing all our motions, muscles and senses to interact with technology in 3D space.

zdragnar 7y ago
Personally, I think it's a bit asinine. Rather than decry anything that resembles what we have today as old-fashioned and not futuristic, it should first be asked "why are screens so common today?" We've been using our hands for years- keyboards, trackballs, mice, joysticks, custom wheels for driving games, the nintendo light gun for duck hunt, and so forth. Take that- and combine it with something else that should be jumping out at you from the article: every picture with a hand shows it holding a different object. Screens are multi-purpose. They are perfect for scenarios where one control must meet many needs (compared to the pictured hammer, which at best meets two). Let's say that we could imagine some multi-purpose, tactile, 3 dimensional interface... say, like a hologram from Iron Man or Star Trek. Now compare that to the article's mention of Alan Kay's doodle pre-integrated circuit. The difference is huge... we don't even have the physics knowledge to describe how a tactile hologram could exist at safe handling energies, let alone in open air, as described by those fictions. Comparatively, Alan Kay combined things that already existed... battery, circuits, screen and keys, and imagined what could happen if they were smaller and portable. We had things much more similar to what he drew decades before we had the ipad; the ipad is merely an incremental improvement over some of my own childhood toys, not to mention what was already available a few years after his own drawing, such as the Speak and Spell ( https://en.wikipedia.org/wiki/Speak_%26_Spell_(toy)#/media/File:Speak_&_Spell_(original_style).jpg https://en.wikipedia.org/wiki/Speak_%26_Spell_(toy)#/media/F... )
- archgoon 7y ago
  Bret Victor pointed out Alan Kay had to wait 50 years before something like the Dynabook was possible. And with a bit of creativity, multi-use tactile interfaces can be accomplished today. https://en.wikipedia.org/wiki/Nintendo_Labo https://en.wikipedia.org/wiki/Nintendo_Labo
  - Palomides 7y ago
    here's a plausible fantasy, some software will include 3D-printable files describing a physical interface device, which can have some standard electronics plugged in as long as it doesn't need to be hard realtime and infinitely reconfigurable, it's very feasible like, imagine a game in which you hammer nails, you would 3D print a hammer, stick in an accelerometer module and mcu/battery/bluetooth module point being, there are definitely ways to make interfaces that are now, or will become, really usable
- jacobolus 7y ago
  > We've been using our hands for years-keyboards, trackballs, mice, joysticks, custom wheels for driving games, the nintendo light gun for duck hunt, and so forth. These are generally quite poor interfaces compared to what is conceivably possible. A precise pressure-sensitive stylus and/or multiple fingers on a responsive high-resolution display are much richer than these: higher bandwidth of a more directly relevant input. But the finger/stylus on screen is strictly 2-dimensional without any haptic feedback. Even such tasks as rotating and moving a computer model of a single 3-dimensional object are extremely cumbersome and unintuitive using mouse, keyboard, joystick, steering wheel, etc. Now imagine using those tools for e.g. controlling a simulated puppet in real time. Pretty much impossible. And yet people control physical puppets with amazing precision and flexibility with some strings and wooden sticks. And that’s just moving an already existing collection of shapes. What if I want to design a new shape, say a newly imagined knot, or a new playground climbing structure, or a chair, or the face of an imagined deep-sea creature. Tools like pipe cleaners, modeling clay, paper & scissors, wooden sticks and a hot glue gun, ... are dramatically richer and more efficient than any of these electronic tools.
- p_l 7y ago

galonk 7y ago

Seems pretty easy to write an essay saying "screens are old news, let's use our hands!" containing not a single word about how that would work, technically or design-wise. I'm going to write an essay saying "screens are old news, let's use free-floating holograms like in Iron Man!" with no indication how that would even be possible, and see if I can get on HN.

Pulcinella 7y ago
You didn't offer a solution. Yes, that's why I called it a rant, not an essay — it describes a problem, not an idea. (And FWIW, that's not the sort of thing I typically publish, or want to.) The solution isn't known, and I didn't think making stuff up would help. The point of the rant was that the solution will be discovered through long research, and that research won't happen without an awareness of the problem. My intent was to hopefully coax a few potential researchers and funders into tangible interfaces, dynamic materials, haptics, and related fields that haven't been named yet. If that happens, then mission accomplished. http://worrydream.com/ABriefRantOnTheFutureOfInteractionDesign/responses.html http://worrydream.com/ABriefRantOnTheFutureOfInteractionDesi...
maxerickson 7y ago
He's been working on https://dynamicland.org/ https://dynamicland.org/ since that time. So there you go.
grumpy8 7y ago
The author of the essay is the founder of https://dynamicland.org https://dynamicland.org which implements what he's talking about in this article.
- galonk 7y ago
  None of the things BV has worked on -- smart tiles or putting pieces of paper under a camera/projector -- do ANYTHING AT ALL like what he's talking about here. They don't change their size, shape, weight, feel etc. in response to data. Because that would be incredibly hard. I don't think you should get points for saying "we should do this impossible thing, that would be better." I must admit I find BV's writing to be really facile... like he'll compare a piano app on the iPad with a real piano and shit on the entire idea of the iPad... without addressing the obvious point that an iPad can be a million things other than a piano.

pgodzin 7y ago

The benefit is of "Pictures Under Glass" is the flexibility - it is extremely easy to change the picture to align to almost any use case. It is extremely difficult to have the same flexible interface with something that is tactile and can manipulate its own shape to be appropriate for the task.

icebraining 7y ago
In 1968 the iPad also seemed extremely difficult, though.

dmix 7y ago

Aw, I was hoping it was a new essay by Bret. Needs (2011) in the title.

Already__Taken 7y ago
Brett victors talk innovating on principal(https://vimeo.com/36579366 https://vimeo.com/36579366) and more so actually Greg Wilsons talk from the same conf (https://vimeo.com/9270320 https://vimeo.com/9270320) marked a pivotal moment for me in my work ethic. It really motivated me to expect better from the tools i use around every day.
- chrisweekly 7y ago
  Similar rxn to those talks. Also, if ever someone knew how to write an "evergreen" post (ie, one that holds up over time), it's Bret Victor. His "Ladder of Abstractions" is a great example: http://worrydream.com/#!2/LadderOfAbstraction http://worrydream.com/#!2/LadderOfAbstraction
  - mcphage 7y ago
    Also his essay Magic Ink: http://worrydream.com/MagicInk/ http://worrydream.com/MagicInk/
  - kragen 7y ago
    I really enjoy his writing, but I think he has a way to go before he's in the same league as Homer, Plato, Lao Tse, and Shakespeare!
    - chrisweekly 7y ago
      haha, those guys sure knew how to write a good blog post!
      - kragen 7y ago
        Yup! They got people still clicking on their posts hundreds or even thousands of years later

simulate 7y ago

I remember this Bret Victor piece from 2011 and I was thinking about it recently when Google announced Soli for the Pixel 4. https://www.cnet.com/news/project-soli-is-the-secret-star-of-googles-pixel-4-self-leak/ https://www.cnet.com/news/project-soli-is-the-secret-star-of... Soli and other gesture-based technologies seem like at least an incremental move to using motion that Bret Victor was advocating for in 2011.

Palomides 7y ago
to me that seems like the same problem, since you still don't touch anything, the only feedback is the computer doing something (or not) that said, I'm definitely curious to see how it performs in the wild

bsanr2 7y ago

I noticed this while experimenting with VR and hand-tracking. Without tactile or precise auditory feedback, the experience was hollow, even with relatively high-fidelity visuals and tracking. You can never feel like you are somewhere else - or something immaterial is with you - until a full sensory experience coalesces, even if it's "low resolution." Until then, the result is landing in an experiential uncanny valley. On another note, we should also be looking at accessibility when approaching this research. Contrary to that last little bit at the end, not everyone has full use of their bodies, or even of their hands. I just finished listening to an NPR segment about inner-city youth dealing with permanent disability as a result of gun violence, and the unique challenges presented to them. Perhaps with people like them in mind, the mass solutions we eventually end up with will be better for focusing farther upstream than nerve endings.

gfodor 7y ago
Hands-in-air interactions in VR I agree lack the tactile feedback necessary to avoid it feeling hollow. However, I've often advocated that the "ideal" VR input system in 2019 is: - Freely tracked dominant hand, used for fine-grained control/pointing, expressing body language, and grab/release-like mechanics (which work decently for certain kinds of virtual objects/interactions, like throwing stuff) - Tracked 6dof controller in non-dominant hand, similar to the existing controllers except with a fully capable twiddler-like button grid for doing fast symbolic entry. For anything where tactile feedback is core, move it onto the controller hand, where the hardware buttons can provide a more reasonable proxy for things like press-and-release or sustained-holding in terms of motor interactions. (Haptics are still pretty bad on all other accounts, but controllers at least give you pressure feedback under your finger tips, instantaneous response, etc.) With the above setup, I suspect you could perform most knowledge worker oriented tasks with similar efficiency to a mouse and keyboard, and also perform existing VR-centric use cases like 3d modelling/manipulation, avatar locomotion, full body social interaction, etc. The current two-controller or dual hand-tracked paradigms fall short on the former. With the introduction of hand tracking capabilities on Quest coming, it seems that we're inching up to mainstream VR devices having the capability to deliver this, except the controllers still lack anything close to keyboard-level discrete inputs. However, once Quest hand tracking launches, it may be possible to use an off-the-shelf twiddler for this if the SDK exposes the ability to track in-hand tools (similar to the Leap Motion SDK.)

melling 7y ago

I almost made a comment in this article about using Deep Learning and Motion Tracking to have computers better understand our gestures so we can interact visually. https://news.ycombinator.com/item?id=21115863 https://news.ycombinator.com/item?id=21115863 At some point AR and VR will finally provide us the overwhelming need. Then we’ll wonder why we didn’t think of it sooner. At the moment, we’re stuck in the “a keyboard and mouse are better” stage.

weego 7y ago
You can't just drop the idea of Deep Learning into a problem as a magical solution. What does visual interaction entail and what is stopping that from having already been trialled during one of the previous (and current) attempts to get vr to work for people?
- melling 7y ago
  If you can use Deep Learning and CNN’s to drive a car, you can probably use it to recognize different hand motions. [Update] I found a link where someone tried to do a simplified version of this: https://towardsdatascience.com/tutorial-using-deep-learning-and-cnns-to-make-a-hand-gesture-recognition-model-371770b63a51 https://towardsdatascience.com/tutorial-using-deep-learning-...
  - mkl 7y ago
    It's certainly possible to recognise hand motions (see e.g. Leap Motion), but I think hand motion recognition is pretty hard to use effectively without haptic feedback telling you what you're touching and where. Ideally actuated like [1] but for your whole hand, not just a stylus. I don't think technology is anywhere near that. The device in [1] pushes back against you, physically stopping you from moving through virtual objects. [1] https://www.3dsystems.com/haptics-devices/touch https://www.3dsystems.com/haptics-devices/touch
    - erikpukinskis 7y ago
      You don’t need haptic feedback, just good feedback. There’s plenty of bits in an audio stream alone to give the brain the control feedback it needs. You’d be surprised how few degrees of freedom we use for grasping.

jct3u 7y ago

The video anchoring the article shows as unavailable for me. Is there an alternate place to view it?

lancebeet 7y ago
Same here, I googled the url of the embedded video and it seems to be a concept video by Corning: https://www.youtube.com/watch?v=6Cf7IL_eZ38 https://www.youtube.com/watch?v=6Cf7IL_eZ38

vortico 7y ago

This is probably my all-time favorite article on the internet. I've brought it up in conversation dozens of times when referring to ways to control computers. It's why I don't really use phones for anything beyond answering phone calls and handling urgent tasks when I'm not at home/office. I just feel like I have very little control over these devices, unlike the massive control with a keyboard or mouse, or even better, a fork and spoon when eating dinner (although despite its high degree of control, you can't control a computer with them). Tech innovators and its users/consumers need to think outside of the box more often for ways to control computers. Given a physical UI device, you can place it on a spectrum of "computer-friendly to human-friendly". Touch screens are computer-friendly because it's trivial for computers to understand and developers to interact with. The downside is that users feel distanced from their control, so it is difficult to use efficiently. But developers are smarter than users at UI---the difficulty should be put on them instead, to control their software/OS. I've imagined things like stress balls with "electric muscles" for feedback to control various applications, a cube with the ability to change its texture with suction/servos, and things like https://roli.com/products/seaboard https://roli.com/products/seaboard or https://www.expressivee.com/buy-touche https://www.expressivee.com/buy-touche except for general computing applications rather than music performance. Who knows what could be invented if customers could be convinced their computing experience can be improved with better physical interfaces.

phaedrus 7y ago
As an engineer I finally got to take the training course our techs go through and experience how a tech uses the software which we (engineers) developed and maintain. It was eye-opening. One particular example relevant to this discussion is the old version of the equipment had physical potentiometers which the tech would adjust to get a desired meter reading. The new equipment has you just type a number into a textbox in a Windows program. BUT - it doesn't even provide a slider or up/down arrows, you have to play The Price Is Right inputting guessed over/under numbers until a measured output in a second text box matches a desired value. It's clear that when the transition was made from physical controls to software based controls, not enough thought was given to ergonomics and how it would be used by an actual human.

carapace 7y ago

Tiny grey sans-serif font means you hate my eyes. Extra-ironic on a rant about interaction design.

tboyd47 7y ago

Instead of a single spectrum from computer- to human-friendliness, picture a triangle with the third vertex being "task-friendliness" or task specificity. The author is advocating a return to human-friendliness, but what he describes is more like task-friendliness. I've become somewhat obsessed with ergonomics after a long battle with chair pain. In my mind, true human-friendliness does not require new interfaces for every task (although it may require a slightly different interface for each human). Every computing task may very well be handled best by a person sitting in a chair at a workstation. That wouldn't be bad design IMO; computers are, in fact, general purpose. If we can get to a point where each computer user is able to use their computer comfortably for 8 hours straight, without pinching a nerve or a blood vessel (yes! it's possible!), then I would consider that a human-friendly design. I'm not talking about iPads here - the author is totally right that these devices don't seem to be made for human beings.

jacobolus 7y ago
As far as I understand, remaining in a static position for 8 hours straight during the day and another 8 hours straight while asleep is pretty bad for many parts of the body irrespective of what that position happens to be. People need to be occasionally changing position, moving about, getting their circulatory system working, getting their muscles working, focusing their eyes at a longer distance, etc. We can improve people’s experience with sit–stand desks, frequent short breaks, periodic longer breaks, a daily commute requiring walking or other exercise, etc., but reducing the total number of hours people are staring at screens every day would also be generally helpful. It isn’t all that great to exclusively work at even the most ergonomic possible computer workstation.
- tboyd47 7y ago
  There's truth to what you're saying, but it implies you have to accept a certain level of pain or physical discomfort if you work in a seated position, and the only way to manage the pain is to get out of your seat. But there are types of pain that can't be managed that way. Those injuries have to do more with sustained muscle tension caused by bad equipment than just sitting, and will continue to get worse until you change your equipment to eliminate the tension.

guitarnick 7y ago

Though this has been discussed many times, I must say the trend of using touchscreens to replace dials and buttons on cars is extremely worrying. It’s entirely an aesthetic and cost saving decision at the expense of safety and UX.

arkh 7y ago
> I must say the trend of using touchscreens to replace dials and buttons on cars is extremely worrying Most are going back on that. It is just that what is currently being sold was developped years ago.
- reaperducer 7y ago
  I'm lucky that my car was made in the in between period, so all touch screen controls have physical controls as well. I use the physical controls, especially when driving. But my navigatrix prefers the screen, primarily because she likes the visual feedback. (Not enough of the physical controls in my car have things that light up to let you know something's happened.)
flowerlad 7y ago
Ford brought back physical buttons. See: https://www.theverge.com/2013/6/17/4437310/ford-dashboard-buttons-sync-myford-touch https://www.theverge.com/2013/6/17/4437310/ford-dashboard-bu...
hunterloftis 7y ago
My wife and I were recently searching for a new car. One of the model's reviewers dinged it for using "dated" physical dash controls instead of the present fad of a single large touchscreen. We entered that into the "positives" section of our spreadsheet.
anonytrary 7y ago
My friend has a BMW and the touchscreen is awful. I can't imagine using that thing to do anything while I'm driving. It looks like it was designed for someone to use on the comfort of their bed, where they can afford to mis-click, not know where the hell to find a page, etc. The only reason I absolutely need a screen in a car is for visibility. Pretty much all the risk in parking/reversing is removed when your car has external cameras. The other functions don't make much sense to me. If you wanna play video games and watch movies while you're driving, it better be a smart car.

arkh 7y ago

I'd like to extend this to fucking VR. All VR experiences lack one thing: full body haptics. Take something inspired from the "Pacific Rim" cockpits ( http://i.imgur.com/DLze6j7.jpg http://i.imgur.com/DLze6j7.jpg ) so you can have some contraption giving you a huge freedom of movement and haptic feedback everywhere. That'd make some games a lot more immersive and physical.

erikpukinskis 7y ago
The reason games don’t feel physical is not because of which senses are being stimulated, it’s how they are stimulated. If the game is just waiting for your hand to enter a bounding box, that’s a 0-dimensional interaction, which feels like nothing. If you could hear your fingers getting close to things, and see the angles between your body and other surfaces, sense through visual texture the relationship your body has with different fields, you would register those things a “physical” experiences. Touch doesn’t feel like touch because it’s your fingers, it feels like touch because it’s rich and continuous and textured and perfectly grounded in body-space. You can make proprioaudiovisual experiences like that too, it’s just hard and there are easier ways to make video games that make money. The science is clear that our brains don’t care which sense data comes from, they just care about the structure of the data. Although there are anatomical structures that make some senses better at processing one structure than another, they don’t prevent the brain from using those senses for other schemas, if the environment demands it (injury, prosthetics, etc)

slotkin 7y ago

ahh was really hoping this was something new from Bret

Chris_Newton 7y ago

It’s sad that since this article was first written, the trend for dumbing down both the hardware and software we use every day has only continued. When I was a kid, I used to love experimenting with programming. I first learned on a ZX81, with a magnificent 1KB of RAM, typing in simple games programs from books every session because there was no storage device to save them. That experience, that joy of being able to create something fun, sparked an interest in what these amazing technologies we have invented can do. Some of my slightly younger friends were lucky enough to have more powerful systems like the BBC Micro available by the time they reached that stage. Those were brilliant because you could connect anything to them. When they were writing simple LOGO programs at school, they didn’t just draw a circle on the screen, an actual mechanical turtle with an actual pen would draw a circle on a real piece of paper, right before their eyes. Now I want to offer that same joy and intrigue to the next generation of my family. Today’s ubiquitous computing devices are phones and tablets, each with numerical specs many orders of magnitude bigger than that ZX81. That little boy typing in listings from a book now has multiple decades of professional programming experience to share. And yet, I can’t sit down with my own child and write even the most simple game on those devices, because for all their theoretical power, they lack even a rudimentary programming interface. In some cases, I can’t even write a game myself on another system and port it, because the whole ecosystem is closed off. How is it that in a time when children seem, often all too soon, to be carrying around more processing power in their pockets than a supercomputer had when I grew up, they still can’t enjoy the sense of freedom and discovery that I experienced with my little ZX81 and its 1KB of RAM and no storage device in the 1980s?

dr_dshiv 7y ago
Yes! Have you considered buying a zx81 for your kid? Or better, a logo turtle?
fragmede 7y ago
They can! If you're on iOS, have you tried Pythonista? It's $8 on the app store, uses Python as the programming language, features an IDE, and a UI library so budding programmers can write video games. For those on Android and the technically minded, a rooted Android tablet offers far more options for on-device programming. There's likely an equivalent to Pythonista in the Google app store as well. (No affiliation; I'm sure there are other apps that are similar, that's just the one I know of.)

badsectoracula 7y ago

Back when touchscreens started to become popular i remember some talks about a 'tactile overlay' to touchscreens, essentially a grid of tiny bumps (pixels, though at a much lower resolution) that would 'pump up' and 'flatten' just enough to provide a tactile feel to the otherwise glassy screens. What happened to that?

_bxg1 7y ago

There are some gaps in reasoning, but overall I think the point stands. The biggest omission is the key advantage of "Pictures Under Glass": they can be defined purely in software. Because of that, I don't think they'll ever stop serving a role in our device interactions. No amount of space-age physical device design could've supported the smartphone's explosion of apps without pictures under glass or some equivalent. With that said, we could definitely benefit from moving some of the more constant/widespread interactions back to physical controls. Buttons and switches on smartphones are always appreciated. And don't even get me started on cars, where there's little need for a vibrant developer ecosystem and a ton of need for non-visual interface comprehension. I also think the Apple Watch's "crown" is one of the more interesting recent examples of a tactile interface that doesn't sacrifice open-ended software development.

tabtab 7y ago
Re: Buttons and switches on smartphones are always appreciated. I disagree, I'm always accidentally bumping or pushing them. Plus, they usually do tasks that are not my top tasks. I'd like to reassign some of them to my favorite actions.
buckminster 7y ago
Surely the dream is to have a programmable tactile interface. Perhaps you could program the height or texture of parts of the screen. But I have no idea how this would work!

neilobremski 7y ago

This brings to mind the phone ringing in the middle of the night. You pick it up and add mumble into the receiver and the conversation begins. But no, now the phone rings in the middle of the night (assuming it isn't text / Slack / email / etc.) and you pick it up and then try to focus your eyes on the screen to figure out WHAT is happening and then HOW to answer. It seems the massive leap forward in the CAPABILITY of our phones has actually caused them to stumble backwards in PRACTICAL usability.

rubidium 7y ago
So, has anyone made a “old school smartphone dock”? What I want is a dock I can put my smartphone into and then the old phone interface takes over. Preferably one with links to 1-2 other docks in the house so all three ring if I get a call. Pick up the physical receiver, say hello. Talk. Say goodbye, hangup. No screen interaction at all.
- TeMPOraL 7y ago
  That's... genius! It's so obvious now that you wrote it. I'd love to have one at my desk, and another at my nightstand.
- bblough 7y ago
  The cordless phone system I have lets you make/receive calls via your cell phone from any handset in the house. It connects via Bluetooth, so isn't a dock, but the functionality is similar to what you're suggesting. Mine is an older Panasonic system, but I'd be surprised if other manufacturers didn't offer something similar.
  - rubidium 7y ago
    Like this? https://shop.panasonic.com/cordless-corded-telephones/cordless-telephones/KX-TG7875S.html https://shop.panasonic.com/cordless-corded-telephones/cordle...
    - 7y ago

tabtab 7y ago

I predict the UI of the future will be via brain implants that allow one to control an app with just thoughts. It may become a necessity in order to keep up with other countries that may have less qualms about the ethical side of surgery and intrusion.

justinmchase 7y ago

> Are we really going to accept an Interface Of The Future that is less expressive than a sandwich? Great line.

tabtab 7y ago
Probably written by a hungry author.

modernerd 7y ago

> With an entire body at your command, do you seriously think the Future Of Interaction should be a single finger? What if you don't have an entire body at your command? The plus side of “pictures behind glass” is that it's a fairly well standardized interface — the software has to adapt to that homogenized hardware, instead of you fighting to adapt to the hardware. (Or, more accurately, you adapt to it once, instead of once for every type of object you want to interact with.) If interactive experiences all start to require a good range of motion, bodily sensitivity, and ability to instinctively interpret the interface, there's a risk it could be incredibly alienating for many. Unless we design with that consideration from day one, it could make adaptivity harder than it already is. I went looking for Bret Victor's take on this question because I was certain he would have thought of this already. So far I only found this: “Channeling all interaction through a single finger is like restricting all literature to Dr Seuss's vocabulary. Yes, it's much more accessible, both to children and to a small set of disabled adults. But a fully-functioning adult human being deserves so much more.” http://worrydream.com/ABriefRantOnTheFutureOfInteractionDesign/responses.html http://worrydream.com/ABriefRantOnTheFutureOfInteractionDesi... I find it sad to read that. Those with access issues deserve so much more too. There's already a huge access gap. If we're going to champion new modes of interaction, we should fight hard not to make the gap bigger still.

tabtab 7y ago
Re: Maybe, if you don't have an entire body at your command. "Boss, I'm not disco dancing, I'm preparing our Powerpoint presentation for the big meeting." Boss: "Please, do the Reboot Shuffle."
hinkley 7y ago
I want someone with the skills of Brett Victor to solve the problem of visualization and maintenance tools for change over time. You solve that problem and there's a world open to you for using things other than flat text files as the system of record. Without those tools, it's all talk. If we're lucky. If we're not lucky it's a death march toward a horrible, horrible boondoggles. We are already trying to visualize four (or more) dimensional code with 2.5D projections (length, width, and a few colors). You give me a tool that can use 3 or 3.5 dimensions and the world opens up to us. Don't forget to write a merge tool.

gdubs 7y ago

A few things come to mind. First, I've been reading a lot on the science behind Montessori education, and there's a lot to support the idea that our brains are much more active when we're physically interacting with something. Would be logical to assume that the richer the physical interaction, the more meaningful the connection. Second, reading Jaron Lanier's somewhat recent book on VR I was struck by how far ahead of its time the 'data glove' was. Early VR may not have had the dazzling eye-candy of today's graphics, but on interactivity we just seem to be catching up. (As an aside, I never realized the Nintendo PowerGlove was basically Data Glove 'lite'; developed by VPL.) Edit: Third, I recently read Carl Sagan's "The Dragons of Eden" which he wrote in the 1970s. It's about the evolution of the human mind. It's SO worth reading today, even if its outdated and proven wrong at times by contemporary discoveries. One thing that really stood out was this idea that it wasn't so much that humans evolved and then invented tools – but that perhaps tools shaped our evolution as much as we shaped tools. Invent a simple tool -> use the tool -> brain develops as a result -> invent even more complex tools -> etc.

fragmede 7y ago

There's a (2011) missing from the title here, at least according to the title graphic.

dang 7y ago
Added now!

gitgud 7y ago

I feel my Nokia 3210 in highschool had a much more intuitive UI than the latest iPhone... and it was trivial to use without looking at it, since your hands could navigate the keys... This isn't possible anymore with sliding pictures under glass...

JesseAldridge 7y ago

Embodied cognition is definitely a thing: https://en.wikipedia.org/wiki/Embodied_cognition https://en.wikipedia.org/wiki/Embodied_cognition And in software development, the primary bottleneck is thinking. Anything you can do to enable greater clarity of thought is going to have huge leverage. So I think this is a super promising angle. Bret's actively developing his ideas at DynamicLand: http://dynamicland.org http://dynamicland.org But clearly it will be quite some time before that tech becomes viable in the real world. Now I'm wondering if there are simpler ways to use the rest of my body to facilitate thinking. Like, I can't mount lasers and network connected cameras on the ceiling... but maybe there's some software that can use a webcam to do simple gesture recognition mapped to bash scripts or something like that.

dang 7y ago

A thread from 2013: https://news.ycombinator.com/item?id=6325996 https://news.ycombinator.com/item?id=6325996 Discussed at the time: https://news.ycombinator.com/item?id=3212949 https://news.ycombinator.com/item?id=3212949

Kenji 7y ago

Touch screens are made for consumption, rich interfaces (with tactile feedback, like keyboards, mice, etc.) are made for creation. Most people just consume on their digital devices. Therefore, touch screens are enough. It's as simple as that. People who want rich interfaces are in a minority but I don't think rich interfaces ever go away because someone has to create all the stuff for digital devices.

cryptozeus 7y ago

I think we are in transition faze. Current future ideas assumes that we will have same interface and hand driven intersections which may not be true. Increase of IOT devices and voice controlled interfaces have different story to tell.

syrrim 7y ago

I think maybe the principle is that we want to be able to transfer as many bits/second of data as possible. We have the computer->human interface down, because screens can already emit more information than humans can process. But human->computer we can only go so fast. This is partly limited by how many fingers we use to do the input. I can type faster on a keyboard than a screen. But I think the main issue people are worried about is that humans can only think so fast. So if the interface requires asking humans for input too much, it will take forever. So instead you try and guess what the humans want. And if you do this well enough, then the input method doesn't matter. Some people say they can type better on their phones because they can use swiftkey. So I think most visions of the future are thinking about AI, and intelligence, and asking "what will we be able to guess without asking the human first?" And this is where they think interaction design is going. Another point of fact. Such a machine requires less bps from the human. That is, it requires them to do less thinking, which most people will prefer. And if you're trying to predict the future, "machines will require less thinking (but be more mass market)" is a pretty safe bet. Sandwiches are very fancy things, as everyone knows. But I expend a lot of mental effort getting the cheese slices in the right thickness. I'm sitting there focusing on making sure I don't deflect the knife too deep or too shallow. This is a very fancy task. But many people prefer specialized cheese slicing devices, because they makes more precise slices, and because they require less mental effort. So expect UI to follow the same trend, from general tools requiring complex input to particular tools requiring simple input.

lkrubner 7y ago

The article focuses on the use of hands, but a visionary future technology might focus more on voice and NLP. This is something I have worked for some years: http://www.smashcompany.com/technology/the-advantages-of-a-natural-language-processing-interface http://www.smashcompany.com/technology/the-advantages-of-a-n...

Animats 7y ago

Victor is one of the people behind Dynamicland, the "live paper"/"room is the computer" startup that was shut down recently. That was on HN last week or so. Fancier input devices seem to have come and gone. They peaked in the 1990s, when you could see a lot of them at SIGGRAPH. My favorite was the magnetically levitated sphere in a bowl. It was like a 3D joystick/trackball with force feedback. It was really cool. It never sold. There were lots of gadgets like that. An animator friend had a workstation with a keyboard, a knob box, a button box, a joystick, a trackball, a tablet, and two screens. Some of the people doing Jurassic Park had a model dinosaur where you could move all the joints, the computer could read the angles, and the on-screen image moved to match. None of this ever caught on. Even 3D joysticks are rare. Game controllers with two joysticks caught on, but those joysticks are abstractions of what's on screen, as, for example, steering, not direct interaction. I tried Jaron Lanier's original gloves-and-goggles VR system. You couldn't do much with the gloves. That was pretty much true in later glove systems. Autodesk fooled around with VR in the 1990s, but determined that gloves and goggles were not going to make CAD easier. Lack of force feedback is a huge problem with gloves. Without force feedback, it's like trying to work in oven mittens. Much of human precision is tactile feedback. Without that, precision work is slow and tiring. As everyone who's soldered surface mount parts under a microscope knows. Back in the 1990s, when I was working on collision detection and physically based animation, I considered building an input device I called "The Handle". The Handle was to be a jointed arm, like a robot arm, with a grip handle on the end as an input device. A servomotor system (or, for cost reasons, I was thinking brakes only back then) would provide tactile feedback. The handle itself would have the ability to split, like pliers, so you'd have something to squeeze. The Hammer could potentially simulate pliers, tongs, wrenches, hammers, etc. Do simulated auto repair. Assemble Meccano. This would have been what Victor is calling for. Could it be built? Yes. Would it sell in volume? No. That's the problem.

Nition 7y ago
I tried out the 3D Systems Touch Haptic Device a few years ago, which sounds a bit like a simpler variant of your "handle".[1] It was quite good. Feels really strange, but good, to be interacting with things "inside the screen". [1] https://www.matterhackers.com/store/l/3d-systems-touch-haptic-device/sk/M12PCJ5U https://www.matterhackers.com/store/l/3d-systems-touch-hapti...

keenmaster 7y ago

Haptic, resistive gloves will merge the tactile with the digital in VR applications. Just like touchscreens, they will serve as a single device that can simulate an infinite number of different controls. Give 5-10 years for the tech to miniaturize and mature. https://haptx.com/ https://haptx.com/

dreamcompiler 7y ago

This is why I dislike touch screens in cars so much: You have to look at them. We have 100 years of experience with car user interfaces that don't require you to take your eyes off the road, but we threw all that away in favor of these shitty pictures under glass.

naiveai 7y ago

The condescending tone of this article does not help it whatsoever. Not to mention the huge gaps in logic that cause it to jump from merely a misguided attempt to rail against well established conventions for interfaces to an active argument for those interfaces still being good.