11 ms·

Past and Present Futures of User Interface Design

212 points by sedboyz 2y ago

tony-allan 2y ago
I know that most developers prefer keyboard shortcuts when developing software but I prefer using the mouse mostly because I cannot remember all of the shortcuts in a range of different environments. Given my preference it would be interesting to explore a more tactile interface. - a series of physical knobs to skip back and forward by function, variable reference, etc - a separate touch screen with haptic feedback for common functions and jump to predefined functions in my code - a macro-pad with real buttons to do the above Other thoughts When watching videos physical buttons and knobs would be good. I know professional video and audio engineers already use these technologies but i've never tried them myself.
- lolinder 2y ago
  > I prefer using the mouse mostly because I cannot remember all of the shortcuts in a range of different environments. This is why one of the greatest changes in power user tooling in recent years is the "find anywhere" hotkey, which is now available almost everywhere. Mouse interaction is slow and hardly a panacea for finding features buried in menus. "Find anywhere" type interactions with fuzzy search allow you to use they keyboard and highly mnemonic abbreviations to turn up what you're looking for. With a few exceptions, I tend to lean on them even for things that I use regularly, because it's easier to learn which few keystrokes will turn up the option I'm looking for than it is to rebind a fresh hotkey in each environment or, as you say, memorize the built-in one.
  - skydhash 2y ago
    > one of the greatest changes in power user tooling in recent years Alt+x in Emacs was here for ages. Even the command prompt in vim follows the same pattern. And while it's useful, I still prefers to bind commonly use commands to keybinding, especially if there can be prefix.
    - lolinder 2y ago
      There's a reason I didn't say "innovation"—I knew that people would immediately point out it's been around forever. What's new is that it's in mainstream tooling.
  - RossBencina 2y ago
    > "find anywhere" hotkey, which is now available almost everywhere Almost everywhere? I'd love to see a list of other examples. The only place I can name is the vscode Ctrl-Shift-P thing, and in that case it's a wholesale replacement for an explorable/discoverable UI (i.e. traditional menu bar). Sure there are search boxes in other places, but usually that's literally for finding things, not for performing application domain commands/actions/manipulations, which is what I understood the parent to be describing.
  - fifticon 2y ago
    I wish the 'find anywhere' instead expanded and highlighted the correct place of that command in the menu. In that way, it would teach me gradually a mental map of the shape and set of commands I have available, ie I imagine it would let me notice 'neighbour commands'. I realise this is not the most efficient way to do this, but it would be 'incremental on the job learning'.
    - csb6 2y ago
      macOS does this when searching for a command in the Help menu - it will open up the context menu and highlight the place where the command is so you know where it is next time.
  - vincnetas 2y ago
    Fun anectada. Was launching Remote desktop app which was unsurprisingly called: "Microsoft Remote Desktop" by just typing "Remo" in to a spotlight. Then one day it stoped working. I thought i was going crazy, because i dont remember that i uninstalled it. Spotlight was returning no results. Then i found out that someone at microsoft thought that it was a great idea to rename the app to ... "Windows App". What the hell were they smoking?
  - zombot 2y ago
    And what is the "find anywhere" hotkey that works everywhere?
    - swah 2y ago
      OP is talking about the "Cmd + P" aka "Run command" command. Ideally it should list the shortcuts of the commands you run so next time you can do that directly.
      - lolinder 2y ago
        The MacOS-wide equivalent is Cmd+Shift+/, which searches all menu items. KDE has a similar idea built into the global menu.
  - swah 2y ago
    If OSes were optimized like RTS games, maybe the mouse could be plenty fast. Something like https://charmstone.app/ https://charmstone.app/ but for many actions.
- thrdbndndn 2y ago
  I use the mouse a lot, even for (typing) coding. I'm pretty "fluent" with navigation shortcuts, things like Ctrl/Alt/Shift combined with arrow keys, PgUp/PgDn/Home/End etc. and I do use them extensively. And yeah, constantly switching between the keyboard and mouse with my right hand is a bit annoying. But still, in many cases, using the mouse is just faster. For example, jumping to a specific position in a source code file, scrolling and clicking gets me there much quicker than navigating with the keyboard alone. (This is also one thing I really hate about using terminals: you can't just click to move the cursor quickly! Editing part of a long string without spaces is a pain in the ass, and it's something I have to do surprisingly often.) When it comes to shortcuts, I prefer one- or two-key combinations whenever possible. Three-key shortcuts, however, depend on their layout: many just aren’t that convenient. Sometimes I’ll just click through the menu manually, even if I know the shortcut.
  - snide 2y ago
    > For example, jumping to a specific position in a source code file, scrolling and clicking gets me there much quicker than navigating with the keyboard alone I say this with the intention of providing context, not to say the way you do things is bad. It's all user preference in the end and there is no wrong way. Lots of folks consider your "fast" example with a mouse as their "slow" example that forced them into learning more advanced features of their editor. For example. Most Vim users can get to any character or partial string, or parameter, or line, or paragraph, or function start or what you have within three quick keys on their home row. They do this quickly, and can immediately start doing other things right after because their hands never moved. The mouse is fast because people don't need to memorize things. The keyboard is fast because the keyboard is fast. It's like the old joke from the movie Heist. "What do you mean you don't like money? That's why they call it money".
    - thrdbndndn 2y ago
      No offence taken; there's always room to learn. Just curious, since I never use Vim, how exactly one navigates to (and/or selects) "aaa" part in the following string with just three keys in Vim? url = "https://example.com/keyword=aaa&name=john"
DidYaWipe 2y ago
Pretty superficial. And no mention of the much-hyped "Minority Report" UI that failed spectacularly for obvious reasons.
furyofantares 2y ago
This is a lot of fairly interesting build up and background leading to a very short and shallow takedown of voice control that's all about audio as IO for a couple practical reasons. No discussion of whether natural language will be a powerful addition to or replacement of current UI, which of course can just be text you type and read.
gyomu 2y ago
I once worked in a design research lab for a famous company. There was a fairly senior, respected guy there who was determined to kill the keyboard as an input mechanism. I was there for about a decade and every year he'd have some new take on how he'd take down the keyboard. I eventually heard every argument and strategy against the keyboard you can come up with - the QWERTY layout is over a century old, surely we can do better now. We have touchscreens/voice input/etc., surely we can do better now. Keyboards lead to RSI, surely we can come up with input mechanisms that don't cause RSI. If we design an input mechanism that works really well for children, then they'll grow up not wanting to use keyboards, and that's how we kill the keyboard. Etc etc. Every time his team would come up with some wacky input demos that were certainly interesting from an academic HCI point of view, and were theoretically so much better than a keyboard on a key dimension or two... but when you actually used them, they sucked way more than a keyboard. My takeaway from that as an interface designer is that you have to be descriptivist, not prescriptivist, when it comes to interfaces. If people are using something, it's usually not because they're idiots who don't know any better or who haven't seen the Truth, it's because it works for them. I think the keyboard is here to stay, just as touchscreens are here to stay and yes, even voice input is here to stay. People do lots of different things with computers, it makes sense that we'd have all these different modalities to do these things. Pro video editors want keyboard shortcuts, not voice commands. Illustrators want to draw on touch screens with styluses, not a mouse. People rushing on their way to work with a kid in tow want to quickly dictate a voice message, not type. The last thing I'll add is that it's also super important, when you're designing interfaces, to actually design prototypes people can try and use to do things. I've encountered way too many "interface designers" in my career who are actually video editors (whether they realize it or not). They'll come up with really slick demo videos that look super cool, but make no sense as an interface because "looking cool in video form" and "being a good interface to use" are just 2 completely different things. This is why all those scifi movies and video commercials should not be used as starting points for interface design.
- kurthr 2y ago
  I may have worked for that company, but I came away with a different take. People are User Interface bigots! People get used to something and that's all they want. The amazing thing Apple was able to do was get people to use the mouse, then the scrollwheel, and then the touchscreen. Usually, that doesn't mean that you get rid of an interface that already exists, but when you create a new device you can rethink the interface. I used the scroll wheel for the iPod before it came out and it was not intuitive, but the ads showed how it worked, and once you used it 20-50x it just seemed right... and everything else was wrong! People would tell me how intuitive it was, and I would laugh, because without the ads and other people using it, it was not at all. Now we're in a weird space, because an entire generation is growing up with swipe interfaces (and a bit of game controller), and that's going to shape their UI attitudes for another generation. I think the keyboard will have a large space, but with LLM prediction, maybe not as much as we've come to expect. I could go on about Fitts testing and cognitive load and the performance of various interfaces, but frankly people ignore it.
deleted 2y ago
[deleted]
alt219 2y ago
> Imagine having to raise your arm to swipe, pinch and tap across an ultra-wide screen several times per minute. Touch works best on small surfaces, even if it looks impressive on a bigger screen. I regularly find myself wishing pinch zoom were available on my large multi-monitor setup, even if i only used it occasionally, i.e. to augment interactions, not as a replacement for other input methods. As a (poor) substitute, I keep an Apple trackpad handy and switch from a mouse to trackpad to do zooming. Sadly I’ve found not all macOS apps respond to Magic Mouse zooming maneuvers.
awesome_dude 2y ago
I, for one, am glad we didn't end up with monkey hands/minority report screens that we interact with by flailing our hands and arms in front of.
ojschwa 2y ago
I'm actually working on a voice controlled, tldraw canvas based UI – and I'm a designer. So I feel quite seen by this article. For my app, I'm trying to visualise and express the 'context' between the user and the AI assistant. The context can be quite complex! We've got quite a challenge to help humans keep up with reasoning and realtime models speed/accuracy. Having a voice input and output (in the form of an optional text to speech) ups the throughput on understanding and updating the context. The canvas is useful for the user to apply spatial understanding, given that users can screen share with the assistant, you can even transfer understanding that way too. I'm not reaching for the future, I'm solving a real pain point of a user now. You can see a demo of it in action here -> https://x.com/ojschwa/status/1901581761827713134 https://x.com/ojschwa/status/1901581761827713134
dataviz1000 2y ago
The next biggest shift in interface is moving from a tactile input -- keyboard, mouse, touch screen, ect. -- and visual screen output to none tactile input -- voice, brain implants, ect. -- and none visual output mostly automation or multistep tasks. Some attempts so far haven't been successful, i.e. Alexa and Siri, others look promising like OpenAI Operator, and it exists in sci-fi like Iron Man's JARVIS, nonetheless, it is definitely the future. I worked on a browser automation virtual assistant for close to a year -- injecting JavaScript into third party webpages like a zombie-ant fungus to control the pages. The idea of tactile input and visual output is so hard coded into the concept of an internet browser, to rethink the input and output of the interface between the human and the machine, everything becomes hack. After a decade working in UI, it was strange to be on a project where the output wasn't something visual or occasionally audio, but rather the output of the UI was automation.
- ch4s3 2y ago
  I’m highly skeptical about voice for most interactions. It’s inherently inappropriate in most public settings.
  - rochak 2y ago
    +1. I see it being helpful for the differently abled but for everyone to just speak out loud their every action would drive everyone nuts in a public setting. Not to mention I can type “code” much faster than if I were to speak it.
  - regularfry 2y ago
    There are a few ideas about subvocal recognition kicking about which might change that. If your voice assistant is in an earpiece that can (somehow) read what your vocal muscles are doing without you actually needing to make a sound, it makes it practical to the extent that it could become the default. There's a lot of ocean between here and there, though. Particularly in the actual sensor tech. That's got to get to the point where you can wear it in public on a highly visible part of the body without feeling like a loon, and that's not trivial.
danjl 2y ago
AI chat will replace the huge number of menus and complex interactions in many content development applications. In VSCode, for example I rarely touch 98% of the UI anymore. I just use Copilot. I no longer need large, complex chunks of the IDE, like the old refactoring code, or the formatting tools. Similarly, in 2D and 3D content creation, isn't it clear that we'll move to some iterative process using AI chat instead of millions of mouse clicks? There are clearly still issues with this, but there were issues with the early GUI tools too. I think huge swaths of complex GUI interfaces will disappear over the next few years. I know many folks who have spent decades in the VFX industry learning how to use Maya, video editors, paint tools, and proprietary tools, and lots of developers who built custom pipelines who might think it will take a bit longer. But it is coming for sure. Maybe you can optionally use voice, but that's not the important bit. This feels like it is going to be a significant shift in how we use computers, and it is already happening.
- jayd16 2y ago
  Why use a 3 key refactor shortcut when you can describe what you want in only a few back and forth conversational chats?
  - danjl 2y ago
    Because what you describe in the chat has a much more significant impact. Sure, you could mimic each little mouse click and short cut, or you could say "make the arms longer", or "write a test for the new endpoint".
    - spartanatreyu 2y ago
      And why do you think delegating the responsibility of tasks to the AI is going to end well? "Make the arms longer" could lead to way too many polygons, or messed up UVs, or will create arms that don't have edge loops, or have them but along the wrong axis, or etc... "Write a test for the new endpoint" is going to lead to "ok, now make the test pass" which is just going to make the test return true, or return a false positive by checking that a function exists instead of calling the function, or calling the wrong value, or changing the code to fit a test that doesn't do the thing it's supposed to but at least the test passes now, or etc... The only thing "AI" is doing is either spitting out something that already exists from google, or making a mess that will need to be cleaned up later.
submeta 2y ago
There are compromises that should have never been made. Just to please the eye. The Blackberry keybord worked fantastically well. It was killed because Steve beleieved that touch screen is always better. Then there are the knobs in cars replaced by touchscreens. I hope there are no attempts at replacing physical keyboards by touchscreens in laptops. I cannot type on touchscreens. I love keyboards. Especially my mechanical keyboard. Love it.
- Cthulhu_ 2y ago
  I think Steve (et al, I will not attribute everything to one person) did have a point; it's 2025 and an on-screen keyboard makes different inputs easier, for a lot more languages than physical keyboards can, without needing separate production lines. There's multiple input methods; typing, swiping/path writing, braille, dictation. It's not for everyone, sure, but for others it's ideal and preferable over a physical keyboard.
  - inetknght 2y ago
    > an on-screen keyboard makes different inputs easier Sure, it can certainly make different inputs easier. But you'll be hard-pressed to convince me that those different inputs are better specifically because a touchscreen doesn't have the kind of physical feedback that comes from feeling the boundaries of different selections (eg, letters/characters) for that input. With a keyboard, I can center my fingers without looking at them. I know exactly what input selections are nearby and in what directions; and I know exactly when my finger has crossed the boundary of one and is in a null-zone or is on another input. For many keyboards, I can even feel the embossing of the individual input selection (character).
- crubier 2y ago
  > It was killed because Steve beleieved that touch screen is always better. No, it was killed because people much preferred Steve's product than Blackberry's. In large part thanks to trading off a 5% of keyboard effectiveness in favor of 50% additional screen space.
- mystified5016 2y ago
mansoorsheriff 2y ago
Mechanical/Analog will always win in the end
scyzoryk_xyz 2y ago
“In a commercial for the Alto, we meet a man - some kind of upper middle management, presumably - going about his daily business. He works in a spacious private office and, using the Alto, he can read and send email and produce laser printouts. Eventually, the Alto conjures up a high resolution image of flowers. The man wonders why, and the computer replies - with text on screen - that it's the man's wedding anniversary. "I forgot," says the man, to which the Alto replies, "It's okay, we're only human."” Sounds like fruit intelligence features
zabzonk 2y ago
At least for games, design everything to be useable for mouse input only, if you want my money. Mind you, do not play FP shooters and the like.
rifty 2y ago
Surely i'm not the only one but I've always felt when it comes to the discussions about replacing the computer desktop experience there's some serious historical blinders on. Work surfaces like the desk as an idea has existed and had been refined for millennias as human interactive spaces before being borrowed and codified to work on computers. it's not bad to look into but i don't think its surprising to find that human physiology hasn't changed, and so the interface environment ideas are still useful for similar situations
zombot 2y ago
I know one thing for certain: The future ain't what it used to be.
fedeb95 2y ago
I don't agree with the fact that touchscreens are necessarily cheaper everywhere: take cars as an example. If a touchscreen makes me more prone to car crashes, because I have to look at it more often than with knobs, in the end it costs more. May take a while to fully reach better interfaces. Sitting at my desk, otherwise, means I can look at a touchscreen.
- TeMPOraL 2y ago
  It's cheaper for the manufacturer. It shifts the UI work from expensive hardware design and manufacturing, that needs to be in-sync with the rest of design and manufacturing of the car, into purely software realm that can be outsourced to some cheap noname company. All the hardware complexity goes away, you only have a glass slab to fit somewhere. Though arguably, the starker difference is with home appliances, where touch interfaces replaced buttons. There, there's no dynamic display (at best, only a light behind a button turning on or off) - so you don't have touch screens, you have a much simpler touch recognition technology, that has zero moving parts and can be pretty much etched onto the board, which is a significant saving over mechanical buttons. And it looks cool, which helped with marketing initially.
  - fedeb95 2y ago
    that's correct, however, as consumers, my opinion is that we (that is, we that share my point of view, obviously!) should try to buy what we prefer regardless of what's cheaper for the manufacturer. If possible of course, we should spend more for something that in the end we think it's better and not fall for flashy designs that makes it look like you're driving some kind of futuristic car. About home appliances, specifically kitchens, in my country (Italy) there has been a widespread wave of electric kitchens with touch interface, which are awful if you do intense cooking: it all becomes covered in water and other substances making it harder to cook. Maybe cooking quality and ease is not that relevant, considered the other benefits of having a touch screen, i don't know: but the design itself, in my opinion, expressed as beauty that better vehicles functionality, is "broken", that is, breaks this definition. Of course this is all my biased opinion.
DonHopkins 2y ago
On Star Trek TNG they have what appears to be a bank of Fujitsu Eagle disk drives on the bridge to the right of the elevator. Looks like 10 for a total of 4.7 GB worth about $100,000! Those are so noisy, you'd think the'd put them in another room so they didn't have to work next to them. https://en.wikipedia.org/wiki/Fujitsu_Eagle https://en.wikipedia.org/wiki/Fujitsu_Eagle https://www.syfy.com/syfy-wire/chosen-one-of-the-day-those-cushy-chairs-on-the-enterprise-bridge https://www.syfy.com/syfy-wire/chosen-one-of-the-day-those-c...
- TeMPOraL 2y ago
  Hah, I never made the connection :). Though on the show, the computer core is far from the bridge, and it only makes cute sci-fi noises. As for the thing you refer to, I don't recall it being ever revealed what those are. They kind of look more like equipment lockers, but if they were, the placement doesn't make sense.
rapnie 2y ago
Different UI paradigms? Then Dynamicland [0] communal computing by Bret Victor [1] comes to mind. Anyone knows other examples where truly different UI paradigms are involved? [0] https://dynamicland.org/ https://dynamicland.org/ [1] https://worrydream.com/ https://worrydream.com/
codelikeawolf 2y ago
> I instead urge UI designers to consider the powerful concepts of consistency and familiarity just a bit more often than they presently seem to do. I've been developing UIs in some form or another for almost 20 years and this really hit home for me. I often use the analogy that UI/UX designers are like interior designers and architects rolled into one while developers are the general contractors and tradespeople (you need to pretend "software architecture" isn't a thing for the analogy to work). In my experience, I could ask a general contractor to come up with a design/layout for a kitchen and I would probably be satisfied with the result. That contractor has built hundreds or thousands of kitchens. A good GC will know how to lay out the space efficiently given the dimensions of the room. You won't end up with weird issues like needing to open the oven door in order to access a cabinet drawer (something I'm dealing with in my current apartment). It probably won't be flashy or unique or have a "wow" factor. In fact, it would probably be so unremarkable as to be forgettable, but it would feel immediately familiar to anyone who comes to visit and function perfectly as a kitchen. I try to push for this type of approach in my current job. I don't want to try implementing novel design ideas. Novel design ideas don't have reference implementations or accessibility guidelines. I think familiarity is the best UX. Are the familiar approaches always the best ones? Maybe not, but I don't want to be the trailblazer. Whenever I encounter a novel design, I ask the person who came up with it to find me 3 other apps that implemented something similar. If they can't, I suggest a more idiomatic solution. Sometimes people push back, but once I start asking very specific questions about how this thing is supposed to behave, they usually end up deciding to drop it and go the idiomatic route. I can't say for sure, but I'm guessing a lot of the articles and blog posts out there complaining about the state of user interfaces aren't written by the folks that have to implement them. There are so many variables involved in accommodating for user interactions with basic controls, let alone complex ones (if you've ever built a custom <select> component that meets all a11y requirements, you know what I mean). So to end my rant: just use the dang controls we have available, don't try to reinvent the wheel, and focus on making your product really good at what people are paying for it to do. I wouldn't spend extra money on an application just because they came up with a novel alternative to a slider control, and I'm guessing most other people wouldn't either.
WillAdams 2y ago
More than anything, the interface which I want to normalize is holding a laptop as a book and using the touchscreen (and optional stylus) on one side, and the keyboard (say for drawing shortcuts/modifiers) on the other side. Still surprised that this wasn't a standard for say the Voyager e-book reader. Hopeful that the Lenovo Yogabook 9i will help to popularize this (and if it had Wacom EMR, I'd have one and be working on such concepts) --- annoyingly, my Samsung Galaxy Book 3 Pro 360 is just a bit too large for this, and the screen is so impossibly thin, trying it makes me worry about breaking it.
curtisszmania 2y ago
[dead]
mncharity 2y ago
> For almost half a century now, we haven't really managed to come up with something better, and that's not for lack of trying. In contrast, my impression is that deprioritization of trying is a defining characteristic. Patents - "yeah it's neat... you can't sell/buy/have/buildon anything like it". Narrowing optimization - "thinkpad is 2-key rollover, because business apps don't use 3, and it saves some cents". Software design-space badlands - "to tweak that, reimplement the full stack and apps". Unicorn dreams - "mass-market or nothing... most often nothing". So instead of a rich ferment of DIY and multi-scale commercial exploring for diverse viable niches, we have an innovation monoculture desert of "wait years for bigco to maybe find incredibly-challenging mass-market fit and afterward backfill niches". So a "we can't sell it because patents so here's a kit"-crippled but still creative AR/VR DIY community hits "Facebook bought Oculus for how much!?!" unicorn dreams and dies. So... etc.
stevetron 2y ago
I just wish I had a hotkey that would turn everything on my screen to Liberation Sans Font, 14-pixel, Boldface and keep it there until I possible hit another hotkey to reverse it.
stevetron 2y ago
I just wish I had a hotkey that would turn everything on my screen to Liberation Sans Font, 14-pixel, Boldface and keep it there until I possibly hit another hotkey to reverse it. If the occasional item needs to be emphasized, then it can be underlined.

212 points by sedboyz 2y ago

I know that most developers prefer keyboard shortcuts when developing software but I prefer using the mouse mostly because I cannot remember all of the shortcuts in a range of different environments. Given my preference it would be interesting to explore a more tactile interface. - a series of physical knobs to skip back and forward by function, variable reference, etc - a separate touch screen with haptic feedback for common functions and jump to predefined functions in my code - a macro-pad with real buttons to do the above Other thoughts When watching videos physical buttons and knobs would be good. I know professional video and audio engineers already use these technologies but i've never tried them myself.

lolinder 2y ago
> I prefer using the mouse mostly because I cannot remember all of the shortcuts in a range of different environments. This is why one of the greatest changes in power user tooling in recent years is the "find anywhere" hotkey, which is now available almost everywhere. Mouse interaction is slow and hardly a panacea for finding features buried in menus. "Find anywhere" type interactions with fuzzy search allow you to use they keyboard and highly mnemonic abbreviations to turn up what you're looking for. With a few exceptions, I tend to lean on them even for things that I use regularly, because it's easier to learn which few keystrokes will turn up the option I'm looking for than it is to rebind a fresh hotkey in each environment or, as you say, memorize the built-in one.
- skydhash 2y ago
  > one of the greatest changes in power user tooling in recent years Alt+x in Emacs was here for ages. Even the command prompt in vim follows the same pattern. And while it's useful, I still prefers to bind commonly use commands to keybinding, especially if there can be prefix.
  - lolinder 2y ago
    There's a reason I didn't say "innovation"—I knew that people would immediately point out it's been around forever. What's new is that it's in mainstream tooling.
- RossBencina 2y ago
  > "find anywhere" hotkey, which is now available almost everywhere Almost everywhere? I'd love to see a list of other examples. The only place I can name is the vscode Ctrl-Shift-P thing, and in that case it's a wholesale replacement for an explorable/discoverable UI (i.e. traditional menu bar). Sure there are search boxes in other places, but usually that's literally for finding things, not for performing application domain commands/actions/manipulations, which is what I understood the parent to be describing.
- fifticon 2y ago
  I wish the 'find anywhere' instead expanded and highlighted the correct place of that command in the menu. In that way, it would teach me gradually a mental map of the shape and set of commands I have available, ie I imagine it would let me notice 'neighbour commands'. I realise this is not the most efficient way to do this, but it would be 'incremental on the job learning'.
  - csb6 2y ago
    macOS does this when searching for a command in the Help menu - it will open up the context menu and highlight the place where the command is so you know where it is next time.
- vincnetas 2y ago
  Fun anectada. Was launching Remote desktop app which was unsurprisingly called: "Microsoft Remote Desktop" by just typing "Remo" in to a spotlight. Then one day it stoped working. I thought i was going crazy, because i dont remember that i uninstalled it. Spotlight was returning no results. Then i found out that someone at microsoft thought that it was a great idea to rename the app to ... "Windows App". What the hell were they smoking?
- zombot 2y ago
  And what is the "find anywhere" hotkey that works everywhere?
  - swah 2y ago
    OP is talking about the "Cmd + P" aka "Run command" command. Ideally it should list the shortcuts of the commands you run so next time you can do that directly.
    - lolinder 2y ago
      The MacOS-wide equivalent is Cmd+Shift+/, which searches all menu items. KDE has a similar idea built into the global menu.
- swah 2y ago
  If OSes were optimized like RTS games, maybe the mouse could be plenty fast. Something like https://charmstone.app/ https://charmstone.app/ but for many actions.
thrdbndndn 2y ago
I use the mouse a lot, even for (typing) coding. I'm pretty "fluent" with navigation shortcuts, things like Ctrl/Alt/Shift combined with arrow keys, PgUp/PgDn/Home/End etc. and I do use them extensively. And yeah, constantly switching between the keyboard and mouse with my right hand is a bit annoying. But still, in many cases, using the mouse is just faster. For example, jumping to a specific position in a source code file, scrolling and clicking gets me there much quicker than navigating with the keyboard alone. (This is also one thing I really hate about using terminals: you can't just click to move the cursor quickly! Editing part of a long string without spaces is a pain in the ass, and it's something I have to do surprisingly often.) When it comes to shortcuts, I prefer one- or two-key combinations whenever possible. Three-key shortcuts, however, depend on their layout: many just aren’t that convenient. Sometimes I’ll just click through the menu manually, even if I know the shortcut.
- snide 2y ago
  > For example, jumping to a specific position in a source code file, scrolling and clicking gets me there much quicker than navigating with the keyboard alone I say this with the intention of providing context, not to say the way you do things is bad. It's all user preference in the end and there is no wrong way. Lots of folks consider your "fast" example with a mouse as their "slow" example that forced them into learning more advanced features of their editor. For example. Most Vim users can get to any character or partial string, or parameter, or line, or paragraph, or function start or what you have within three quick keys on their home row. They do this quickly, and can immediately start doing other things right after because their hands never moved. The mouse is fast because people don't need to memorize things. The keyboard is fast because the keyboard is fast. It's like the old joke from the movie Heist. "What do you mean you don't like money? That's why they call it money".
  - thrdbndndn 2y ago
    No offence taken; there's always room to learn. Just curious, since I never use Vim, how exactly one navigates to (and/or selects) "aaa" part in the following string with just three keys in Vim? url = "https://example.com/keyword=aaa&name=john"

DidYaWipe 2y ago

Pretty superficial. And no mention of the much-hyped "Minority Report" UI that failed spectacularly for obvious reasons.

furyofantares 2y ago

This is a lot of fairly interesting build up and background leading to a very short and shallow takedown of voice control that's all about audio as IO for a couple practical reasons. No discussion of whether natural language will be a powerful addition to or replacement of current UI, which of course can just be text you type and read.

gyomu 2y ago

I once worked in a design research lab for a famous company. There was a fairly senior, respected guy there who was determined to kill the keyboard as an input mechanism. I was there for about a decade and every year he'd have some new take on how he'd take down the keyboard. I eventually heard every argument and strategy against the keyboard you can come up with - the QWERTY layout is over a century old, surely we can do better now. We have touchscreens/voice input/etc., surely we can do better now. Keyboards lead to RSI, surely we can come up with input mechanisms that don't cause RSI. If we design an input mechanism that works really well for children, then they'll grow up not wanting to use keyboards, and that's how we kill the keyboard. Etc etc. Every time his team would come up with some wacky input demos that were certainly interesting from an academic HCI point of view, and were theoretically so much better than a keyboard on a key dimension or two... but when you actually used them, they sucked way more than a keyboard. My takeaway from that as an interface designer is that you have to be descriptivist, not prescriptivist, when it comes to interfaces. If people are using something, it's usually not because they're idiots who don't know any better or who haven't seen the Truth, it's because it works for them. I think the keyboard is here to stay, just as touchscreens are here to stay and yes, even voice input is here to stay. People do lots of different things with computers, it makes sense that we'd have all these different modalities to do these things. Pro video editors want keyboard shortcuts, not voice commands. Illustrators want to draw on touch screens with styluses, not a mouse. People rushing on their way to work with a kid in tow want to quickly dictate a voice message, not type. The last thing I'll add is that it's also super important, when you're designing interfaces, to actually design prototypes people can try and use to do things. I've encountered way too many "interface designers" in my career who are actually video editors (whether they realize it or not). They'll come up with really slick demo videos that look super cool, but make no sense as an interface because "looking cool in video form" and "being a good interface to use" are just 2 completely different things. This is why all those scifi movies and video commercials should not be used as starting points for interface design.

kurthr 2y ago
I may have worked for that company, but I came away with a different take. People are User Interface bigots! People get used to something and that's all they want. The amazing thing Apple was able to do was get people to use the mouse, then the scrollwheel, and then the touchscreen. Usually, that doesn't mean that you get rid of an interface that already exists, but when you create a new device you can rethink the interface. I used the scroll wheel for the iPod before it came out and it was not intuitive, but the ads showed how it worked, and once you used it 20-50x it just seemed right... and everything else was wrong! People would tell me how intuitive it was, and I would laugh, because without the ads and other people using it, it was not at all. Now we're in a weird space, because an entire generation is growing up with swipe interfaces (and a bit of game controller), and that's going to shape their UI attitudes for another generation. I think the keyboard will have a large space, but with LLM prediction, maybe not as much as we've come to expect. I could go on about Fitts testing and cognitive load and the performance of various interfaces, but frankly people ignore it.

deleted 2y ago

[deleted]

alt219 2y ago

> Imagine having to raise your arm to swipe, pinch and tap across an ultra-wide screen several times per minute. Touch works best on small surfaces, even if it looks impressive on a bigger screen. I regularly find myself wishing pinch zoom were available on my large multi-monitor setup, even if i only used it occasionally, i.e. to augment interactions, not as a replacement for other input methods. As a (poor) substitute, I keep an Apple trackpad handy and switch from a mouse to trackpad to do zooming. Sadly I’ve found not all macOS apps respond to Magic Mouse zooming maneuvers.

awesome_dude 2y ago

I, for one, am glad we didn't end up with monkey hands/minority report screens that we interact with by flailing our hands and arms in front of.

ojschwa 2y ago

I'm actually working on a voice controlled, tldraw canvas based UI – and I'm a designer. So I feel quite seen by this article. For my app, I'm trying to visualise and express the 'context' between the user and the AI assistant. The context can be quite complex! We've got quite a challenge to help humans keep up with reasoning and realtime models speed/accuracy. Having a voice input and output (in the form of an optional text to speech) ups the throughput on understanding and updating the context. The canvas is useful for the user to apply spatial understanding, given that users can screen share with the assistant, you can even transfer understanding that way too. I'm not reaching for the future, I'm solving a real pain point of a user now. You can see a demo of it in action here -> https://x.com/ojschwa/status/1901581761827713134 https://x.com/ojschwa/status/1901581761827713134

dataviz1000 2y ago

The next biggest shift in interface is moving from a tactile input -- keyboard, mouse, touch screen, ect. -- and visual screen output to none tactile input -- voice, brain implants, ect. -- and none visual output mostly automation or multistep tasks. Some attempts so far haven't been successful, i.e. Alexa and Siri, others look promising like OpenAI Operator, and it exists in sci-fi like Iron Man's JARVIS, nonetheless, it is definitely the future. I worked on a browser automation virtual assistant for close to a year -- injecting JavaScript into third party webpages like a zombie-ant fungus to control the pages. The idea of tactile input and visual output is so hard coded into the concept of an internet browser, to rethink the input and output of the interface between the human and the machine, everything becomes hack. After a decade working in UI, it was strange to be on a project where the output wasn't something visual or occasionally audio, but rather the output of the UI was automation.

ch4s3 2y ago
I’m highly skeptical about voice for most interactions. It’s inherently inappropriate in most public settings.
- rochak 2y ago
  +1. I see it being helpful for the differently abled but for everyone to just speak out loud their every action would drive everyone nuts in a public setting. Not to mention I can type “code” much faster than if I were to speak it.
- regularfry 2y ago
  There are a few ideas about subvocal recognition kicking about which might change that. If your voice assistant is in an earpiece that can (somehow) read what your vocal muscles are doing without you actually needing to make a sound, it makes it practical to the extent that it could become the default. There's a lot of ocean between here and there, though. Particularly in the actual sensor tech. That's got to get to the point where you can wear it in public on a highly visible part of the body without feeling like a loon, and that's not trivial.

danjl 2y ago

AI chat will replace the huge number of menus and complex interactions in many content development applications. In VSCode, for example I rarely touch 98% of the UI anymore. I just use Copilot. I no longer need large, complex chunks of the IDE, like the old refactoring code, or the formatting tools. Similarly, in 2D and 3D content creation, isn't it clear that we'll move to some iterative process using AI chat instead of millions of mouse clicks? There are clearly still issues with this, but there were issues with the early GUI tools too. I think huge swaths of complex GUI interfaces will disappear over the next few years. I know many folks who have spent decades in the VFX industry learning how to use Maya, video editors, paint tools, and proprietary tools, and lots of developers who built custom pipelines who might think it will take a bit longer. But it is coming for sure. Maybe you can optionally use voice, but that's not the important bit. This feels like it is going to be a significant shift in how we use computers, and it is already happening.

jayd16 2y ago
Why use a 3 key refactor shortcut when you can describe what you want in only a few back and forth conversational chats?
- danjl 2y ago
  Because what you describe in the chat has a much more significant impact. Sure, you could mimic each little mouse click and short cut, or you could say "make the arms longer", or "write a test for the new endpoint".
  - spartanatreyu 2y ago
    And why do you think delegating the responsibility of tasks to the AI is going to end well? "Make the arms longer" could lead to way too many polygons, or messed up UVs, or will create arms that don't have edge loops, or have them but along the wrong axis, or etc... "Write a test for the new endpoint" is going to lead to "ok, now make the test pass" which is just going to make the test return true, or return a false positive by checking that a function exists instead of calling the function, or calling the wrong value, or changing the code to fit a test that doesn't do the thing it's supposed to but at least the test passes now, or etc... The only thing "AI" is doing is either spitting out something that already exists from google, or making a mess that will need to be cleaned up later.

submeta 2y ago

There are compromises that should have never been made. Just to please the eye. The Blackberry keybord worked fantastically well. It was killed because Steve beleieved that touch screen is always better. Then there are the knobs in cars replaced by touchscreens. I hope there are no attempts at replacing physical keyboards by touchscreens in laptops. I cannot type on touchscreens. I love keyboards. Especially my mechanical keyboard. Love it.

Cthulhu_ 2y ago
I think Steve (et al, I will not attribute everything to one person) did have a point; it's 2025 and an on-screen keyboard makes different inputs easier, for a lot more languages than physical keyboards can, without needing separate production lines. There's multiple input methods; typing, swiping/path writing, braille, dictation. It's not for everyone, sure, but for others it's ideal and preferable over a physical keyboard.
- inetknght 2y ago
  > an on-screen keyboard makes different inputs easier Sure, it can certainly make different inputs easier. But you'll be hard-pressed to convince me that those different inputs are better specifically because a touchscreen doesn't have the kind of physical feedback that comes from feeling the boundaries of different selections (eg, letters/characters) for that input. With a keyboard, I can center my fingers without looking at them. I know exactly what input selections are nearby and in what directions; and I know exactly when my finger has crossed the boundary of one and is in a null-zone or is on another input. For many keyboards, I can even feel the embossing of the individual input selection (character).
crubier 2y ago
> It was killed because Steve beleieved that touch screen is always better. No, it was killed because people much preferred Steve's product than Blackberry's. In large part thanks to trading off a 5% of keyboard effectiveness in favor of 50% additional screen space.
mystified5016 2y ago

mansoorsheriff 2y ago

Mechanical/Analog will always win in the end

scyzoryk_xyz 2y ago

“In a commercial for the Alto, we meet a man - some kind of upper middle management, presumably - going about his daily business. He works in a spacious private office and, using the Alto, he can read and send email and produce laser printouts. Eventually, the Alto conjures up a high resolution image of flowers. The man wonders why, and the computer replies - with text on screen - that it's the man's wedding anniversary. "I forgot," says the man, to which the Alto replies, "It's okay, we're only human."” Sounds like fruit intelligence features

zabzonk 2y ago

At least for games, design everything to be useable for mouse input only, if you want my money. Mind you, do not play FP shooters and the like.

rifty 2y ago

Surely i'm not the only one but I've always felt when it comes to the discussions about replacing the computer desktop experience there's some serious historical blinders on. Work surfaces like the desk as an idea has existed and had been refined for millennias as human interactive spaces before being borrowed and codified to work on computers. it's not bad to look into but i don't think its surprising to find that human physiology hasn't changed, and so the interface environment ideas are still useful for similar situations

zombot 2y ago

I know one thing for certain: The future ain't what it used to be.

fedeb95 2y ago

I don't agree with the fact that touchscreens are necessarily cheaper everywhere: take cars as an example. If a touchscreen makes me more prone to car crashes, because I have to look at it more often than with knobs, in the end it costs more. May take a while to fully reach better interfaces. Sitting at my desk, otherwise, means I can look at a touchscreen.

TeMPOraL 2y ago
It's cheaper for the manufacturer. It shifts the UI work from expensive hardware design and manufacturing, that needs to be in-sync with the rest of design and manufacturing of the car, into purely software realm that can be outsourced to some cheap noname company. All the hardware complexity goes away, you only have a glass slab to fit somewhere. Though arguably, the starker difference is with home appliances, where touch interfaces replaced buttons. There, there's no dynamic display (at best, only a light behind a button turning on or off) - so you don't have touch screens, you have a much simpler touch recognition technology, that has zero moving parts and can be pretty much etched onto the board, which is a significant saving over mechanical buttons. And it looks cool, which helped with marketing initially.
- fedeb95 2y ago
  that's correct, however, as consumers, my opinion is that we (that is, we that share my point of view, obviously!) should try to buy what we prefer regardless of what's cheaper for the manufacturer. If possible of course, we should spend more for something that in the end we think it's better and not fall for flashy designs that makes it look like you're driving some kind of futuristic car. About home appliances, specifically kitchens, in my country (Italy) there has been a widespread wave of electric kitchens with touch interface, which are awful if you do intense cooking: it all becomes covered in water and other substances making it harder to cook. Maybe cooking quality and ease is not that relevant, considered the other benefits of having a touch screen, i don't know: but the design itself, in my opinion, expressed as beauty that better vehicles functionality, is "broken", that is, breaks this definition. Of course this is all my biased opinion.

DonHopkins 2y ago

On Star Trek TNG they have what appears to be a bank of Fujitsu Eagle disk drives on the bridge to the right of the elevator. Looks like 10 for a total of 4.7 GB worth about $100,000! Those are so noisy, you'd think the'd put them in another room so they didn't have to work next to them. https://en.wikipedia.org/wiki/Fujitsu_Eagle https://en.wikipedia.org/wiki/Fujitsu_Eagle https://www.syfy.com/syfy-wire/chosen-one-of-the-day-those-cushy-chairs-on-the-enterprise-bridge https://www.syfy.com/syfy-wire/chosen-one-of-the-day-those-c...

TeMPOraL 2y ago
Hah, I never made the connection :). Though on the show, the computer core is far from the bridge, and it only makes cute sci-fi noises. As for the thing you refer to, I don't recall it being ever revealed what those are. They kind of look more like equipment lockers, but if they were, the placement doesn't make sense.

rapnie 2y ago

Different UI paradigms? Then Dynamicland [0] communal computing by Bret Victor [1] comes to mind. Anyone knows other examples where truly different UI paradigms are involved? [0] https://dynamicland.org/ https://dynamicland.org/ [1] https://worrydream.com/ https://worrydream.com/

codelikeawolf 2y ago

> I instead urge UI designers to consider the powerful concepts of consistency and familiarity just a bit more often than they presently seem to do. I've been developing UIs in some form or another for almost 20 years and this really hit home for me. I often use the analogy that UI/UX designers are like interior designers and architects rolled into one while developers are the general contractors and tradespeople (you need to pretend "software architecture" isn't a thing for the analogy to work). In my experience, I could ask a general contractor to come up with a design/layout for a kitchen and I would probably be satisfied with the result. That contractor has built hundreds or thousands of kitchens. A good GC will know how to lay out the space efficiently given the dimensions of the room. You won't end up with weird issues like needing to open the oven door in order to access a cabinet drawer (something I'm dealing with in my current apartment). It probably won't be flashy or unique or have a "wow" factor. In fact, it would probably be so unremarkable as to be forgettable, but it would feel immediately familiar to anyone who comes to visit and function perfectly as a kitchen. I try to push for this type of approach in my current job. I don't want to try implementing novel design ideas. Novel design ideas don't have reference implementations or accessibility guidelines. I think familiarity is the best UX. Are the familiar approaches always the best ones? Maybe not, but I don't want to be the trailblazer. Whenever I encounter a novel design, I ask the person who came up with it to find me 3 other apps that implemented something similar. If they can't, I suggest a more idiomatic solution. Sometimes people push back, but once I start asking very specific questions about how this thing is supposed to behave, they usually end up deciding to drop it and go the idiomatic route. I can't say for sure, but I'm guessing a lot of the articles and blog posts out there complaining about the state of user interfaces aren't written by the folks that have to implement them. There are so many variables involved in accommodating for user interactions with basic controls, let alone complex ones (if you've ever built a custom <select> component that meets all a11y requirements, you know what I mean). So to end my rant: just use the dang controls we have available, don't try to reinvent the wheel, and focus on making your product really good at what people are paying for it to do. I wouldn't spend extra money on an application just because they came up with a novel alternative to a slider control, and I'm guessing most other people wouldn't either.

WillAdams 2y ago

More than anything, the interface which I want to normalize is holding a laptop as a book and using the touchscreen (and optional stylus) on one side, and the keyboard (say for drawing shortcuts/modifiers) on the other side. Still surprised that this wasn't a standard for say the Voyager e-book reader. Hopeful that the Lenovo Yogabook 9i will help to popularize this (and if it had Wacom EMR, I'd have one and be working on such concepts) --- annoyingly, my Samsung Galaxy Book 3 Pro 360 is just a bit too large for this, and the screen is so impossibly thin, trying it makes me worry about breaking it.

curtisszmania 2y ago

[dead]

mncharity 2y ago

> For almost half a century now, we haven't really managed to come up with something better, and that's not for lack of trying. In contrast, my impression is that deprioritization of trying is a defining characteristic. Patents - "yeah it's neat... you can't sell/buy/have/buildon anything like it". Narrowing optimization - "thinkpad is 2-key rollover, because business apps don't use 3, and it saves some cents". Software design-space badlands - "to tweak that, reimplement the full stack and apps". Unicorn dreams - "mass-market or nothing... most often nothing". So instead of a rich ferment of DIY and multi-scale commercial exploring for diverse viable niches, we have an innovation monoculture desert of "wait years for bigco to maybe find incredibly-challenging mass-market fit and afterward backfill niches". So a "we can't sell it because patents so here's a kit"-crippled but still creative AR/VR DIY community hits "Facebook bought Oculus for how much!?!" unicorn dreams and dies. So... etc.

stevetron 2y ago

I just wish I had a hotkey that would turn everything on my screen to Liberation Sans Font, 14-pixel, Boldface and keep it there until I possible hit another hotkey to reverse it.

stevetron 2y ago

I just wish I had a hotkey that would turn everything on my screen to Liberation Sans Font, 14-pixel, Boldface and keep it there until I possibly hit another hotkey to reverse it. If the occasional item needs to be emphasized, then it can be underlined.