35 ms·

Enough AI copilots, we need AI HUDs

979 points by walterbell 1y ago

keyle 1y ago
> anyone serious about designing for AI should consider non-copilot form factors that more directly extend the human mind. Aren't auto-completes doing exactly this? It's not a co-pilot in the sense of a virtual human, but already more in the direction of a HUD. Sure you can converse with LLMs but you can also clearly just send orders and they eagerly follow and auto-complete. I think what the author might be trying to express in a quirky fashion, is that AI should work alongside us, looking in the same direction as we are, and not being opposite to us at the table, staring at each other's and arguing. We'll have true AI when they'll be doing our bidding, without any interaction from us.
- gklitt 1y ago
  Author here. Yes, I think the original GitHub Copilot autocomplete UI is (ironically) a good example of a HUD! Tab autocomplete just becomes part of your mental flow. Recent coding interfaces are all trending towards chat agents though. It’s interesting to consider what a “tab autocomplete” UI for coding might look like at a higher level of abstraction, letting you mold code in a direct-feeling way without being bogged down in details.
  - juped 1y ago
    If that's what you think a HUD is, then a HUD is definitely way, way worse. Rather than a copilot sitting next to you, that's someone grabbing your hands and doing things with them while you're at the controls. But if I invoke the death of the author and pretend HUD meant HUD, then it's a good point: tools are things you can form a cybernetic system with, classic examples being things like hand tools or cars, and you can't form a cybernetic system with something trying to be an "agent". To be in a cybernetic system with something you need predictable control and fast feedback, roughly.
    - CharlieDigital 1y ago
      I take "HUD" here to just mean "in your line of vision" or "in the context of your actual task" or minimizing any context switch to another interaction (chat window). Rather I think most implementations of HUD AI interactions so far have been quite poor because the interaction model itself is perhaps immature and no one has quite hit the sweet spot yet (that I know of). Tab autocompletion is a simple gesture, but trades off too much control for more complex scenarios and is too easy to accidentally activate. Inline chat is still a context switch and also not quite right.
samfriedman 1y ago
On this topic, can anyone find a document I saw on HN but can no longer locate? A historical computing essay, it was presented in a plaintext (monospaced) text page. It outlined a computer assistant and how it should feel to use. The author believed it should be unobtrusive, something that pops into awareness when needed and then gets back out of the way. I don't believe any of the references in TFA are what it was.
- thehappypm 1y ago
  Designing Calm Technology?
- gsmt 1y ago
  beyond intelligent machines?
cadamsdotcom 1y ago
Love the idea & spitballing ways to generalize to coding.. Thought experiment: as you write code, an LLM generates tests for it & the IDE runs those tests as you type, showing which ones are passing & failing, updating in real time. Imagine 10-100 tests that take <1ms to run, being rerun with every keystroke, and the result being shown in a non-intrusive way. The tests could appear in a separated panel next to your code, and pass/fail status in the gutter of that panel. As simple as red and green dots for tests that passed or failed in the last run. The presence or absence and content of certain tests, plus their pass/fail state, tells you what the code you’re writing does from an outside perspective. Not seeing the LLM write a test you think you’ll need? Either your test generator prompt is wrong, or the code you’re writing doesn’t do the things you think they do! Making it realtime helps you shape the code. Or if you want to do traditional TDD, the tooling could be reversed so you write the tests and the LLM makes them pass as soon as you stop typing by writing the code.
- hnthrowaway121 1y ago
  Yes the reverse makes much more sense to me. AI help to spec out the software & then the code has an accepted definition of correctness. People focus on this way less than they should I think
- callc 1y ago
  Humans writing the test first and LLM writing the code is much better than the reverse. And that is because tests are simply the “truth” and “intention” of the code as a contract. When you give up the work of deciding what the expected inputs and outputs of the code/program is you are no longer in the drivers seat.
  - JimDabell 1y ago
    > When you give up the work of deciding what the expected inputs and outputs of the code/program is you are no longer in the drivers seat. You don’t need to write tests for that, you need to write acceptance criteria.
    - 1y ago
jawns 1y ago
The author gives an example of a HUD: an AI that writes a debugger program to enable the human developer to more easily debug code. But why should it be only the human developer who benefits? What if that debugger program becomes a tool that AI agents can use to more accurately resolve bugs? Indeed, why can't any programming HUD be used by AI tools? If they benefit humans, wouldn't they benefit AI as well? I think we'll be pretty quickly at the point where AI agents are more often than not autonomously taking care of business, and humans only need to know about that work at critical points (like when approvals are needed). Once we're there, the idea that this HUD concept should be only human-oriented breaks down.
deleted 1y ago
[deleted]
hi_hi 1y ago
Doesn't it all come down to "what is the ideal interface for humans to deal with digital information"? We're getting more and more information thrown at us each day, and the AIs are adding to that, not reducing it. The ability to summarise dense and specialist information (I'm thinking error logs, but could be anything really) just means more ways for people to access and view that information who previously wouldn't. How do we, as individuals, best deal with all this information efficiently? Currently we have a variety of interfaces, websites, dashboards, emails, chat. Are all these necessary anymore? They might be now, but what about the next 10 years. Do I even need to visit a companies website if can get the same information from some single chat interface? The fact we have AIs building us websites, apps, web UI's just seems so...redundant.
- moomoo11 1y ago
  I like the smartphone. It’s honestly perfect and underutilized.
- sipjca 1y ago
  yep I think this is the fundamental question as well, everything else is intermediate
- guardiang 1y ago
  Every human is different, don't generalize the interface. Dynamically customize it on the fly.
- AlotOfReading 1y ago
  Websites were a way to get authoritative information about a company, from that company (or another trusted source like Wikipedia). That trust is powerful, which is why we collectively spent so much time trying to educate users about the "line of death" in browsers, drawing padlock icons, chasing down impersonator sites, mitigating homoglyph attacks, etc. This all rested on the assumption that certain sites were authoritative sources of information worth seeking out. I'm not really sure what trust means in a world where everyone relies uncritically on LLM output. Even if the information from the LLM is usually accurate, can I rely on that in some particularly important instance?
roywiggins 1y ago
> The agentic option is a “copilot” — a virtual human who you talk with to get help flying the plane. If you’re about to run into another plane it might yell at you “collision, go right and down!” Planes do actually have this now. It seems to work okay: https://en.m.wikipedia.org/wiki/Traffic_collision_avoidance_system https://en.m.wikipedia.org/wiki/Traffic_collision_avoidance_...
- gklitt 1y ago
  Author here. I thought about including TCAS in the article but it felt too tangential… You’re right that there’s a voice alert. But TCAS also has a map of nearby planes which is much more “HUD”! So it’s a combo of both approaches. (Interestingly it seems that TCAS may predate Weiser’s 1992 talk)
droideqa 1y ago
I want a link to the GitHub for this[0] which he linked to. Makes Prolog quite interesting. [0]: https://www.geoffreylitt.com/2024/12/22/making-programming-more-fun-with-an-ai-generated-debugger https://www.geoffreylitt.com/2024/12/22/making-programming-m...
- johnisgood 1y ago
  Yeah, it looks interesting. I would give it a spin. Maybe then I could finally make a MUD in Prolog. :D
thepuglor 1y ago
I'd rather flip this around and be in a fully immersive environment watching agents do things in such a way that I am able to interject and guide in realtime. How do we build that, and build it in such a way that the content and delivery of my guidance becomes critical to what they learn? The best teacher gets the best AI students.
- sitkack 1y ago
  > fully immersive environment Start with a snapshot of what you are envisioning using Blender.
aaron695 1y ago
This is a mess. The 1992 talk wasn't at all about AI and since then our phones have given us "ubiquitous computing" en masse. The original talk required no 'artificial intelligence' for relevance which makes it strange to apply to todays artificial intelligence. The original talk made good points for instance "voice recognition" has been solved forever at a reasonable level, yet people kept claiming if it was 'better' a 'magic experience' would pop out as if voice was different to typing. Idiots have been around for a long time. Don't get what OP is trying to say. 'AI HUD metaphors' are very hard, that's why they are not ubiquitous, they require constant input. Spellcheck runs every character typed. Agents are because of less $. 'Hallucinations' also make 'AI HUD metaphors' problematic, for spellcheck squiggly red lines would be blinking on off all over the page as a LLM keeps coming back with different results.
- precompute 1y ago
  FYI, you're shadowbanned.
jpm_sd 1y ago
This is how ship's AI is depicted in The Expanse (TV series) and I think it's really compelling. Quiet and unobtrusive, but Alex can ask the Rocinante to plot a new course or display the tactical situation and it's fast, effective and effortlessly superhuman with no back-talk or unnecessary personality. Compare another sci-fi depiction taken to the opposite extreme: Sirius Cybernetics products in the Hitchhikers Guide books. "Thank you for making a simple door very happy!"
- skydhash 1y ago
  I may remember wrongly, but I don't believe the expanse was depicting AI. It was more powerful computation and unobtrusive interfaces. There was nothing like Jarvis. Rocinante was a war vessel and all its feature was tailored to that. I believe even the mechanical suits of the Martians were very manual (no cortana a la Master Chief).
  - jpm_sd 1y ago
    That's exactly what this article is about, though. Assistive systems that are incredibly capable without needing to have a fake personality.
kova12 1y ago
aren't we missing the point that co-pilot is there in case pilot gets incapacitated?
- melagonster 1y ago
  The pilot is the copilot today. These computer can handle most of the tasks automatically.
benjaminwootton 1y ago
I think there is a third and distinct model which is AI that runs in the background autonomously amd over a long period and pushes things to you. It can detect situations intelligently, do the filtering, summarisation of what’s happening and possibly a recommendation. This feels a lot more natural to me, especially in a business context when you want to monitor for 100 situations about thousands of customers.
- skydhash 1y ago
  Actually defining those situations and collecting the data (which should help identify those situations) are the hard parts. Having an autonomous system that do it has been solved for ages.
ares623 1y ago
It should have a paperclip mascot
stan_kirdey 1y ago
needed to find a kids’ orthodontist. made a tiny voice agent: feed it numbers, it calls, asks about price/availability/insurance, logs the gist. it kind of worked. the magic was the smallest UI around it: - timeline of dials + retries - "call me back" flags - when it tried, who picked up - short summaries with links to the raw transcript once i could see the behavior, it stopped feeling spooky and started feeling useful. so yeah, copilots are cool, but i want HUDs: quiet most of the time, glanceable, easy to interrupt, receipts for every action.
perching_aix 1y ago
Been thinking about something similar, from fairly grounded ideas like letting a model autogenerate new features with their own name, keybind and icon, all the way to silly ideas, like letting a model synthesize arbitrary shader code and just letting it do whatever within the viewport. Think the entire UI being created on the fly specifically for the task you're working on, constantly evolving in mesh with your workflow habits. Now if only I went beyond being an idea man...
- caleblloyd 1y ago
  The reason we are not seeing this in mainstream software may also be due to cost. Paying for tokens on every interaction means paying to use the app. Upfront development may actually be cheaper, but the incremental cost per interaction could cost much more in the long term, especially if the software is used frequently and has a long lifetime. As the cost of tokens goes down, or commodity hardware can handle running models capable of driving these interactions, we may start to see these UIs emerge.
  - perching_aix 1y ago
    Oh yeah, I was 100% thinking in terms of local models.
ag2s 1y ago
A very relevant article from lesswrong, titled Cyborgism https://www.lesswrong.com/s/f2YA4eGskeztcJsqT/p/bxt7uCiHam4QXrQAA https://www.lesswrong.com/s/f2YA4eGskeztcJsqT/p/bxt7uCiHam4Q...
sothatsit 1y ago
AI building complex visualisations for you on-the-fly seems like a great use-case. For example, if you are debugging memory leaks in a specific code path, you could get AI to write a visualisation of all the memory allocations and frees under that code path to help you identify the problem. This opens up an interesting new direction where building visualisations to debug specific problems is probably becoming viable. This idea reminds me of Jonathan Blow's recent talk at LambdaConf. In it, he shows a tool he made to visualise his programs in different ways to help with identifying potential problems. I could imagine AI being good at building these. The talk: https://youtu.be/IdpD5QIVOKQ?si=roTcCcHHMqCPzqSh&t=1108 https://youtu.be/IdpD5QIVOKQ?si=roTcCcHHMqCPzqSh&t=1108
- federiconafria 1y ago
  Independently of visualizations, I think LLMs allow in general the creation of ad-hoc tools. I've experienced the case where asking for a quick python script was faster and more powerful than learning how to use a cli to interact with an API.
  - twbarber 1y ago
    Anecdotally, the low time to prototype has had me throwing usable tool projects up on my local network. I wrote a meal planner and review tool for our weekly vegetable share, and local board game / book library that guests can connect to and reference. It scratches the itch to build and ship with the benefit of a growing library of low scope, high performance, highly customized web tools I can write over a few hours in an evening instead of devoting weekends to it. It feels like switching from hand to power tools
  - borgel 1y ago
    Same. It's very handy to be able to ask for one-off tools for things that I _could_ try to figure out, but probably wouldn't be worth the time. My favorite example so far was a Python script to help debug communication between a microcontroller and a motor driver. I was able to dump the entire datasheet PDF for the driver into Gemini and ask it for a Python CLI to decode communication traffic, and 30 seconds later it was done. Fantastic bang/buck ratio!
wewewedxfgdf 1y ago
Kind of a weird article because the computer systems that is "invisible" i.e. an integrated part of the flight control systems - is exactly what we have now. He's sort of arguing for .... computer software. Like, we have HUDs - that's what a HUD is - it's a computer program.
- CGamesPlay 1y ago
  A HUD is typically non-interactive, which is the core distinction he’s advocating for. The “copilot” responds to your requests, the “HUD” surfaces relevant information passively.
  - skydhash 1y ago
    Isn't this one of the core selling of an IDE compared to a simple editor? You have the analyzer running in the background and telling you things. And a host of buttons around the editor to act on stuff, all within the context of a project.
nioj 1y ago
Concurrent posting from 5 hours ago (currently no comments): https://news.ycombinator.com/item?id=44705018 https://news.ycombinator.com/item?id=44705018
satisfice 1y ago
Yes, this is a non-creepy way of applying AI.
SilverElfin 1y ago
Isn’t that what all the AI browsers like Comet, or the things like Cluey are trying to do
eboynyc32 1y ago
Excited for the next wave of ai innovation.
clbrmbr 1y ago
A thought-provoking analogy! What comes immediately to mind for me is using embeddings to show closest matches to current cursor position on the right tab for fast jumping to related files.
henriquegodoy 1y ago
Great post! i've been thinking along similar lines about human-AI interfaces beyond the copilot paradigm. I see two major patterns emerging: Orchestration platforms - Evolution of tools like n8n/Make into cybernetic process design systems where each node is an intelligent agent with its own optimization criteria. The key insight: treat processes as processes, not anthropomorphize LLMs as humans. Build walls around probabilistic systems to ensure deterministic outcomes where needed. This solves massive "communication problems" Oracle systems - AI that holds entire organizations in working memory, understanding temporal context and extracting implicit knowledge from all communications. Not just storage but active synthesis. Imagine AI digesting every email/doc/meeting to build a living organizational consciousness that identifies patterns humans miss and generates strategic insights. just explored more about it on my personal blog https://henriquegodoy.com/blog/stream-of-consciousness https://henriquegodoy.com/blog/stream-of-consciousness
ankit219 1y ago
The current paradigm is driven by two factors: one is the reliability of the models and that constraints how much autonomy you can give to an agent. Second is about chat as a medium which everyone went to because ChatGPT became a thing. I see the value in HUDs, but only when you can be sure output is correct. If that number is only 80% or so, copilots work better so that humans in the loop can review and course correct - the pair programmer/worker. This is not to say we need ai to get to higher levels of correctness inherently, just that systems deployed need to do so before they display some information on HUD.
- psychoslave 1y ago
  This is missing the addictive/engaging part of a conversational interface for most people out there. Which is in line with the critics highlighted in the fine article. Just because most people are fond of it doesn't actually mean it improves their life, goals and productivity.
makaking 1y ago
> "non-copilot form factors that more directly extend the human mind." I think Cursor's tab completion and next edit prediction roughly fits the pattern, you don't chat, you don't ask or explain, you just do... And the more in coherent your actions are the more useful the HUB becomes.
ravila4 1y ago
I think one key reason HUDs haven’t taken off more broadly is the fundamental limitation of our current display medium - computer screens and mobile devices are terrible at providing ambient, peripheral information without being intrusive. When I launch an AI agent to fix a bug or handle a complex task, there’s this awkward wait time where it takes too long for me to sit there staring at the screen waiting for output, but it’s too short for me to disengage and do something else meaningful. A HUD approach would give me a much shorter feedback loop. I could see what the AI is doing in my peripheral vision and decide moment-to-moment whether to jump in and take over the coding myself, or let the agent continue while I work on something else. Instead of being locked into either “full attention on the agent” or “completely disengaged,” I’d have that ambient awareness that lets me dynamically choose my level of involvement. This makes me think VR/AR could be the killer application for AI HUDs. Spatial computing gives us the display paradigm where AI assistance can be truly ambient rather than demanding your full visual attention on a 2D screen. I picture that this would be especially helpful for help on more physical tasks, such as cooking, or fixing a bike.
- elliotec 1y ago
  You just described what I do with my ultrawide monitor and laptop screen. I can be fully immersed in a game or anything and keep Claude in a corner of a tmux window next to a browser on the other monitor and jump in whenever I see it get to the next step or whatever.
  - ravila4 1y ago
    It’s a similar idea, but imagine you could fire off a task, and go for a run, or do the dishes. Then be notified when it completes, and have the option to review the changes, or see a summary of tests that are failing, without having to be at your workstation.
    - bigyabai 1y ago
      I kinda do this today, with Alpaca[0]'s sandboxed terminal runner and GSConnect[1] syncing the response notifications to my phone over LAN. [0] https://jeffser.com/alpaca/ https://jeffser.com/alpaca/ [1] https://github.com/GSConnect/gnome-shell-extension-gsconnect https://github.com/GSConnect/gnome-shell-extension-gsconnect
Der_Einzige 1y ago
Actual HUDs, like the ones for cars - along with vector displays, and laser display tech in general, is criminally undervalued and underutilized.
radres 1y ago
One of the simplest and best working applications of AI is the gpts living in your clipboard. Any prompt/workflow you have is assignable to a shortcut on demand and it pops up a chat window with response on demand. It's been game changing honestly.
- furyofantares 1y ago
  Can you elaborate on this?
furyofantares 1y ago
I'm very curious if a toggle would be useful that would display a heatmap of a source file showing how surprising each token is to the model. Red tokens are more likely to be errors, bad names, or wrong comments.
- WithinReason 1y ago
  previously undefined variable and function names would be red as well
  - Too 1y ago
    All editors do this already.
    - WithinReason 1y ago
      No, I mean when an LLM encounters a previously unseen name it doesn't expect it so it would be red, even though it's perfectly valid.
      - jakelazaroff 1y ago
        I don't think that's necessarily true. I've definitely seen LLMs hallucinating variables they never defined.
      - furyofantares 1y ago
        I'm imagining everything the LLM could produce (with a given top_k setting) would be shades of green to yellow. Just outside of that orange, and far outside red. LLMs generate new functions all the time, I'd guess these would be light green, maybe the first token in the name would be yellow and it would get brighter green as the name unfolds. The logits are probably all small when in the global scope where it's not clear what will be defined next. I'm not imagining mapping logits directly to heat, the ordering of tokens seems much more appropriate.
  - fwip 1y ago
yurimo 1y ago
I might be wrong but isn't the HUD the author suggesting for coding is basically AREPL? For debugging I can see it work, but chatbox and inline q&a I feel has awider application. On a wider note, I buy the argument for alternative interfaces other than chat, but chat permeates our lives every day, smartphone is full of chat interfaces. HUD might be good for AR glasses though, literal HUD.
aantix 1y ago
I would love to see a HUD that allows me to see the change that corresponds to Claude Code's TODO item. I don't want inline comments as those accumulate, don't get cleaned up appropriately by the LLM.
kn81198 1y ago
About a decade back Bret Victor [1] talked about how his principle in life is to reduce the delay in feedback, and having faster iteration cycles not just helps in doing things (coding) better but also contributes to new creative insights. He had a bunch of examples built to showcase alternative ways of coding, which is very close to being HUDs - one example shown in the OP is very similar to the one he presents to "step through time to figure out the working of the code". [1]: https://www.youtube.com/watch?v=PUv66718DII https://www.youtube.com/watch?v=PUv66718DII
- latorf 1y ago
  This!!! Thank you, couldn't have said it better. One of the best talks I've ever seen, and a huge inspiration for a lot of work I've done.
piker 1y ago
Absolutely agree, and spellchecker is a great analogy. I've recently been snoozing co-pilot for hours at a time in VS Code because it’s adding a ton of latency to my keystrokes. Instead, it turns out that `rust_analyzer` is actually all that I need. Go-to definition and hover-over give me exactly what the article describes: extra senses. Rust is straightforward, but the tricky part may be figuring out what additional “senses” are helpful in each domain. In that way, it seems like adding value with AI comes full circle to being a software design problem. ChatGPT and Claude are great as assistants for strategizing problems, but even the typeahead value seems to me negligible in a large enough project. My experience with them as "coding agents" is generally that they fail miserably or are regurgitating some existing code base on a well known problem. But they are great at helping config things and as teachers in (the Socratic sense) to help you get up-to-speed with some technical issue. The heads-up display is the thesis for Tritium[1], going back to its founding. Lawyers' time and attention (like fighter pilots') is critical but they're still required in the cockpit. And there's some argument they always will be. [1] https://news.ycombinator.com/item?id=44256765 https://news.ycombinator.com/item?id=44256765 ("an all-in-one drafting cockpit")
- kibwen 1y ago
  On the topic of Rust IDE plugins that give you more senses, take a look at Flowistry: https://github.com/willcrichton/flowistry https://github.com/willcrichton/flowistry . It's not AI, it's using information flow analysis.
Animats 1y ago
Nobody mentioned Manna [1] yet? That suggests a mostly audio headset giving orders. There is a real-world version using AR glasses.[2] [1] https://marshallbrain.com/manna1 https://marshallbrain.com/manna1 [2] https://www.six-15.com/vision-picking https://www.six-15.com/vision-picking
utf_8x 1y ago
You know that feature in JetBrains (and possibly other) IDEs that highlights non-errors, like code that could be optimized for speed or readability (inverting ifs, using LINQ instead of a foreach, and so on)? As far as I can tell, these are just heuristics, and it feels like the perfect place for an “AI HUD.” I don’t use Copilot or other coding AIs directly in the IDE because, most of the time, they just get in the way. I mainly use ChatGPT as a more powerful search engine, and this feels like exactly the kind of IDE integration that would fit well with my workflow.
obscure-enigma 1y ago
> design the cockpit so that the human pilot is naturally aware of their surroundings. This is a design interface problem. Self-driving cars can easily ingest this HUD. This is the reason what makes Apple's AI different from other microservice-like AI. The spell checker, rewrite, proofread are naturally integrated into the UI to the extent it doesn't feel like AI powered operations.
iddan 1y ago
This is what we are going for in Closer - an AI native experience that extends the mind of salespeople. https://closer.so https://closer.so
evrennetwork 1y ago
[dead]
Oras 1y ago
Great post, informative and precise. I think the challenge is primarily the context and intent. The spellchecker knows my context easily, and there is a setting to choose from (American English, British English, etc.), as well as the paragraphs I'm writing. The intent is easy to recognise. While in a codebase, the context is longer and vaguer, the assistant would hardly know why I'm changing a function and how that impacts the rest of the codebase. However, as the article mentions, it may not be a universal solution, but it's a perspective to consider when designing AI systems.
thinkingemote 1y ago
We also need what goes along with HUDs : switches, nubs, switches, dials. Actual controls. Although we are talking HUDs, I'm not really talking about UI widgets having the good old skew-morphism or better buttons. In the cockpit the pilot doesn't have his controls on a touch screen, he has an array of buttons and dials and switches all around him. It's these controls that are used in response to what the pilot sees on the HUD and it's these controls that change the aircraft according to the pilots will, which in turn change what the HUD shows.
TimCTRL 1y ago
makes sense, i guess tools like cluely are HUDs
- barrenko 1y ago
  Yup, not affiliated, but this is what cluely kinda does - https://nitter.poast.org/im_roy_lee/status/1938719006002921751 https://nitter.poast.org/im_roy_lee/status/19387190060029217...
_samjarman 1y ago
I'd love to mount a top-down webcam above me when I work on electronics and have a real time feed into ChatGPT or similar for Q&A, for a real time coach that can see. One day... :)
precompute 1y ago
HUDs are primarily of use to people that are used to parsing dense visual information. Wanting HUDs on a platform that promises to do your work for you is quite pointless.
afro88 1y ago
A HUD is an even more "confident" display of data than text though. What do you do with a HUD that hallucinates? Is there a button on each element that shows you sources?
- Dilettante_ 1y ago
  I could imagine a system in which the model only chooses which data points to show at what time, but the actual passing is still handled by good old deterministic programming.
- alwa 1y ago
  Night-vision optics come to mind: prone to noise and visual artifacts, and especially strange under certain edge conditions. Some of their specs tend to be strictly inferior to a Mark I Eyeball—narrow FOV, limited focusing power, whatever else. But an operator learns to intuit which aspects to trust and which to double-check. The fact that it’s an “extra sense” can outweigh the fact that it’s not a perfect source of truth, no? Trust the tech where it proves useful to you, and find ways to compensate (or outright don’t use it) where it’s not.
  - bpfrh 1y ago
    I think the major different between that and KI is that the the Night-Vision is mostly static and knowable through experience/teaching, while KI is an ever moving target where you experience is zero on every new display/question. I think the idea of an hud is better than the current paradigm, but it doesn't solve the fundamental problem.
simianwords 1y ago
I believe this is the framework vs library debate but in a different context. HUD is like me using a library of AI enabled tools. The human agency is given more importance here - they decide how to use the tools to suit their purpose. Copilot is more like a framework where an AI system exists which tells me what to do (a bit like the inverse of a library).
amelius 1y ago
I want an AI HUD that blocks ads.
- Cthulhu_ 1y ago
  What if you could get AR glasses that translates ads into their core message like "CONSUME", "MARRY AND REPRODUCE" and "OBEY"? That would be cool. You could make a movie out of that!
est 1y ago
It's like the magic moment when GUIs replace CLIs. I remember the first time I started up Win95 from DOS days. Stunning.
- Cthulhu_ 1y ago
  Or terminal GUIs in DOS. Before (and during) Windows, our go-to was 3DMENU [0] for starting up games and the like. [0] https://archive.org/details/3DMENUPLUSDOS https://archive.org/details/3DMENUPLUSDOS
smolder 1y ago
[flagged]
smolder 1y ago
Let's stop allowing this clearly incentivized hype for investable shit and have honest conversation, please.
anaisbetts 1y ago
I mean, fundamentally the reason that we don't build most AI HUD ideas right now is that they are either stupidly insanely expensive at scale, or you have to run the AI on-device on an NPU so underpowered that it's going to give you really really bad results. Lots of great ideas in this space but it's tough to make something that delivers value and also is economically viable
smolder 1y ago
[flagged]
- tomhow 1y ago
  It's explicitly against the guidelines to comment like this: Please don't post insinuations about astroturfing, shilling, brigading, foreign agents, and the like. It degrades discussion and is usually mistaken. If you're worried about abuse, email hn@ycombinator.com and we'll look at the data. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
vidarh 1y ago
> “You’ll no more run into another airplane than you would try to walk through a wall.” People do walk into walls, though. I see the appeal of the idea, but it's not a replacement for something that actually agressively demands your attention when there are high risk events.
kevinphy 1y ago
This whole Copilot vs HUD debate instantly brought to mind a classic Japanese anime from 1991 called Future GPX Cyber Formula (https://en.wikipedia.org/wiki/Future_GPX_Cyber_Formula https://en.wikipedia.org/wiki/Future_GPX_Cyber_Formula). Yeah, it’s a racing anime set in the then-distant future of 2015, where cars come with full-on intelligent AIs. The main character’s car, Asurada, is basically a "Copilot" in every sense. It was designed by his dad to be more than just a tool, more like a partner that learns, adapts, and grows with the driver. Think emotional support plus tactical analysis with a synthetic voice. Later in the series, his rival shows up driving a car that feels very much like a HUD concept. It's all about cold data, raw feedback, and zero bonding. Total opposite philosophy. What’s wild is how accurately it captures the trade-offs we’re still talking about in 2025. If you’re into human-AI interaction or just want to see some shockingly ahead-of-its-time design thinking wrapped in early '90s cyber aesthetics, it’s absolutely worth a watch.
- yahoozoo 1y ago
  Anywhere to stream this?
  - deleted 1y ago
    [deleted]
  - petercooper 1y ago
    Quality is so-so but it appears to be on YouTube: https://www.youtube.com/watch?v=cXtvPUtSTRY https://www.youtube.com/watch?v=cXtvPUtSTRY
- 1317 1y ago
  any release you'd recommend?
  - layer8 1y ago
    669198800283 if you need English subtitles. VPXY-71923 if you’re fine with Japanese.
- 1y ago
torginus 1y ago
I read you, but unfortunately despite the limitations of current AI tech, all companies covertly/overtly market themselves as human replacement not human augmentation solutions with the aim of cutting out expensive experts out of the loop, not making them more productive, even though the messaging might say otherwise from time to time.
endymion-light 1y ago
I think there's room for a mixture of both use-cases. I think personally programming is far more like being a mixture of a pilot and an engineer. There are times, like in this article where it's far more useful to have a HUD, while i'm focused, i'd much rather have a better eye on things that are interesting to me. I often find things like cursor do the opposite - they slow down your processing time, you shift into an environment that often feels like i'm correcting a junior engineers work rather than expanding my knowledge of the codebase. For example, if i'm working on authentication logic, i'd much rather see a smart heads up display, you look at a function and get advice on where this might mess other things up in the codebase, edge cases, etc. A smarter form of current IDES that doesn't mean you click through 50 different files to work out that this third party package doesn't work on specific code. A HUD in this case is ideal. But I find there's a more detailed, slower development, I still really often use the chat function on claude mixed with Obsidian to hold bits of information i've found useful, this is more related to getting a deeper understanding of certain concepts. As a stupid developer, I often find I might need something explained 20 times in 20 different ways, and actually a predictive text model is perfect in so many circumstances to explain a massive algorithm step by step. It's ideal for things like shader code, where I might come back to it 6 months later and want to work out what was in my head at the time, a historic chat is perfect for those things. There's definitely a balance to be struck, I think now that the hype cycle is peaking we can hopefully seperate the profit seeking AI tools from the useful day to day knowledge expansion. Currently it feels like the discovery of perspective in the reneissance - instead of using it to further advancements, we're attempting to sell perspective courses to people.
camdroidw 1y ago
AI powered this, AI powered that, what most people don't realise is that the next game changer would be Al powered batteries
- nunodonato 1y ago
  what?
newsclues 1y ago
More sensors and more data, humans become a limiting factor until a better way to communicate the information is used.
your_challenger 1y ago
The Ink & Switch guys always have the best-est of ideas!
roca 1y ago
We have explored that sort of debugging/visualization tool in https://pernos.co https://pernos.co. We built it before the age of genAI, but I think for coming up with powerful visualizations AI is neither necessary nor (yet) sufficient.
OJFord 1y ago
I do think typing text prompts at LLMs will go down as a barely remembered brief period in history, that seems quaint and dumb to anyone who remembers or is told about it. (I don't know what we'll be doing instead, I just think text prompts feel dumb.)
kotaKat 1y ago
Enough AI copilots and AI HUDs, we need “none of this bullshit forced upon us”.
julik 1y ago
So... Google Glass?
cmrdporcupine 1y ago
100% agree with this. I want augmentation not automation. That said, CoPilot in the form of "autocomplete" is kind of that. I have been enjoying the hell out of Claude Code, but I'd feel much better about it if it wasn't a case of "here take this pile of diffs" and had a mode Socratic modality. A tool that works with me instead of for me, because I'm going to have to review everything it does anyways.
aleksituk 1y ago
I like HUDs as an analogy for other forms of content/experience enrichment too e.g. what's a HUD for other forms of interaction than just looking at screens (like audio, or multimodal). Can we embed useful data behind the content we produce/consume or derive personalised versions for the user? -> Text/audio/video tailored for me and my interests? (ie. not just content recommended for me, but the content itself is tailored for me in terms of the insights and practical applications) -> Podcasts I can interact with? -> Audiobooks I can ask questions about? Richer interaction that allows for more back&forth and key insights for the user and use case.
FrustratedMonky 1y ago
I think the OpenAI left panel/ right panel code, seems to fit this.
Havoc 1y ago
Some sort of neural interface feels inevitable at this point
betterhealth12 1y ago
How about contextual AI insights as you're interacting with the world, wherever you are? Like a quality signal. Oh wait, this is something FB/Goog could never do because what would it say about the quality of information hosted...
- soco 1y ago
  AR for the regular guy is something which I remember installing on my mobile over ten years ago - with low expectations, and it never got any better than that, quite the contrary. Powered by AI or not, seems nobody has found (or looked for) a market for this.
  - betterhealth12 1y ago
    Another decade, another attempt? We are aiming for this with our work at Credible (https://mycredible.ai https://mycredible.ai and our chrome extension) - product is still early, I'd be curious to get your first thoughts or reaction.
cat-whisperer 1y ago
I'm thinking AR glasses with AI HUDs and we're basically there - my to-do list just got a whole lot more interesting.
monkpit 1y ago
I was recently thinking, if services offered sufficient api docs/specs you could really just provide an llm to the client and have any client build their own UI to fit their needs. Maybe not that you’d want to. But you could!
bitwize 1y ago
I want AI (machine learning statistical techniques) to convert my brainwaves into text so that I can think my programs into the computer. That would be a useful breakthrough in AI-assisted programming,
_jab 1y ago
There’s a lot of ideation for coding HUDs in the comments, but ironically I think the core feature of most coding copilots is already best described as a HUD: tab completion. And interestingly, that is indeed the feature I find most compelling from Cursor. I particularly love when I’m doing a small refactor, like changing a naming convention for a few variables, and after I make the first edit manually Cursor will jump in with tab suggestions for the rest. To me, that fully encapsulates the definition of a HUD. It’s a delightful experience, and it’s also why I think anyone who pushes the exclusively-copilot oriented Claude Code as a superior replacement is just wrong.
- cleverwebble 1y ago
  Agreed! I've spent the last few months using Claude Code and Cursor - experimenting with both. For simple tasks, both are pretty good (like identifying a bug given console output) - but when it comes to making a big change, like adding a brand new feature to existing code that requires changes to lots of files, writing tests, etc - it often will make at least a few mistakes I catch on review, and then prompting the model to fix those mistakes often causes it to fix things in strange ways. A few days ago, I had a bug I just couldn't figure out. I prompted Claude to diagnose and fix the issue - but after 5 minutes or so of it trying out different ideas, rerunning the test, and getting stuck just like I did - it just turned off the test and called it complete. If I wasn't watching what it was doing, I could have missed that it did that and deployed bad code. The last week or so, I've totally switched from relying on prompting to just writing the code myself and using tab complete to autocomplete like 80% of it. It is slower, but I have more control and honestly, it's much more enjoyable of an experience.
  - jonchurch_ 1y ago
    Drop in a lint rule to fail on skipped tests. Ive added these at a previous job after finding that tests skipped during dev sometimes slipped through review and got merged.
- Garlef 1y ago
  I think there's a lot of room for even better UI. I'd love to have something that operates more at the codebase level. Autocomplete is very local. (Maybe "tab completion" when setting up a new package in a monorepo? Or make architectural patterns consistent across a whole project? Highlight areas in the codebase where the tests are weak? Or collect on the fly a full view of a path from FE to BE to DB?)
timeinput 1y ago
There is a giant confusing tech stack here, but opening up this simple plain text page resulted in like 100% CPU usage from from gnome-shell when visiting from firefox on a pretty non-customized Ubuntu 24 install. In the firefox task manager nothing really looked odd, but opening that tab and displaying it is insanely CPU intensive. Pausing the autoplaying video makes it seem like a sane web page in terms of CPU usage. I'm surprised how much CPU playing that video consumed.
- itslennysfault 1y ago
  Very odd. This must be some kind of Ubuntu issue. Out of curiosity I tried it in Chrome, Firefox and Safari on MacOS 15.5 and saw very little CPU usage and no difference with/without the video playing. I don't have a Ubuntu Desktop handy to confirm. Looking at the video it's nothing special. Just a fairly small mp4 using a native html5 video element to play it. Really no reason this should be causing issues.
  - timeinput 1y ago
    Yeah, totally agreed. It was quite surprising to me. Who knows it may even be related to the random GPU I happen to have or ....
dreamcompiler 1y ago
The main reasons I don't have "smart" devices in my house: 1. They introduce new and fascinating failure modes that never happened with the old, dumb devices (e.g. if your router fails you lose the ability to control your lights). 2. They demand human attention at the slightest provocation (e.g. the microwave beeps loudly forever when your food is done, every app on your phone insists on interacting with you whenever the company would like to upsell you something, etc.) Item 2 above is what TFA is about. Yes you can often turn this shit off, but that's not the point. The point is you shouldn't have to. Useful technology should never call attention to itself in the manner of someone with narcissistic personality disorder. But what about emergency situations? Glad you asked. Many airplane crashes in modern aircraft have happened because of "warning buzzer overload" which happens when one important system on the aircraft fails and then causes a cascade of secondary warnings, while giving the pilot no insight as to the root cause. A true AI assistant would reason about such situations and guide the pilot toward the root solution. A true coding assistant would do the same kind of reasoning about program errors and suppress multipage error dumps in favor of flagging the root issue.
jFriedensreich 1y ago
My aha moment into this direction happened when i got XR glasses and tried embracing voice agents to see where this leads. Did not expect the HUD aspect be so cool of how this setup works. At first i tried using the glasses as a replacement for my main screen and using voice agents as a replacement to the sidebars and chat windows we are used to, which worked ok but then i started using the main screen and main agent interfaces again but kept the glasses with a second screen on a different visual focal plane connected to the voice agents. Dark mode gets a totally new importance there because black is transparent in XR which is important to work well without obscuring the main screen. I can switch between screens by shifting eye focus and have a sense what the other screen is doing a far better than just how a physical second screen is in the visual periphery. When i need to combine images more i move the focal planes closer together so i can see both screens better at the same time. The voice agents can answer side questions, start research that is needed for the next step of the main workflow or fix minor issues that are not important enough to interrupt the main workflow. Its so obvious how HUD, voice ai, XR and agents can grow together into this new computing environment but i am afraid what happens once android and ios shape that reality. I want this to be part of the web.
- kroaton 1y ago
  What glasses are you using?
  - jFriedensreich 1y ago
    viture pro xr but connected just as external display to my macbook. the experimental hud software is build from scratch for my usecase using web presentation api and a custom svelte framework that does stereoscopic rendering with css matrix transformations.
Einenlum 1y ago
Makes me think that I would love to have an AI tool that not only allows autocomplete but also provides suggestions while I write code. "Have you thought about using useCallback here?" "This feature seems already provided by library X and Y" "It seems like this code already exists in function Foo. Maybe a small refactoring would be helpful"
throwaway12345t 1y ago
This is now (more) dated. Copilots as an interface are dated. The current initiative is full agents with human in the loop at the very start, occasionally the middle and the end. Huds are just good UI, something copilots can natively exist as part of in the form of contextual insights and alerts. We’re moving on to agency where it’s everything else vs an entirely different entity taking the action of flying the plane from take off to landing.
- ColonelPhantom 1y ago
  I don't think you understood what a "copilot" and a "HUD" is. Confusingly, GitHub Copilot (the original one that suggests completions) is basically a HUD. On the other hand, agents that you give a task are clearly copilots that work via a natural language interface. The article also mentions that agents are copilots: > Here’s another personal example from AI coding. Let’s say you want to fix a bug. The obvious “copilot” way is to open an agent chat and ask it to do the fix.
LeoPanthera 1y ago
All I have ever wanted in my life is a heads-up display for my eyes that puts floating icons over other people telling me their name, birthday, the names of their kids, and what we talked about the last time we met.
- maksimur 1y ago
  You basically want a cognitive aid device?
  - LeoPanthera 1y ago
    Yes please.
bhaktatejas922 1y ago
doing AI code edits on a JS web app is probably the way to do it. probably using something like groq kimi k2 + morph for fast edits now we need a web framework with faster reload times
galaxyLogic 1y ago
Heads-Up-Display is of course useful if it gives you useful information. But you can't tell that display "Ok fly me to Rome". That's what "co-pilot" is about. It can do things, rather than just display information it decides you might find useful.
amir-h 1y ago
I love this analogy. I've been particularly feeling like this regarding AI code reviewers recently - I don't want a copilot that will do their own review, I want a hud that will make it easier for me to understand the change. I've been toying with crafting such a code review tool as a side project recently: https://useglide.ai https://useglide.ai
revskill 1y ago
Agree. Current copilot bullshits lost their context, ahhh no, they know nothing bout context, such a joke.
djtriptych 1y ago
I'm pretty sure this is want I want to work on for the next decade if anyone's looking for a world class UI engineer.
WASDAai 1y ago
You are right and i am working on it for months. It is just not easy to make. Especially if your team is small
DonHopkins 1y ago
Ben Shneiderman wrote "Beyond Intelligent Machines" published in the Jan 1993 IEEE Software, and the subject of this hn discussion: Beyond Intelligent Machines: Just Do It (1993) (umd.edu) https://news.ycombinator.com/item?id=22742100 https://news.ycombinator.com/item?id=22742100 Mark Weiser and Ben Shneiderman and I worked at the University of Maryland College Park before he went to PARC, and they have a similar take on agents versus augmentation. I think Douglass Engelbart, Maurice Wilkes, and Joseph Weizenbaum would agree with many of their points. https://www.cs.umd.edu/~ben/papers/Shneiderman1993Beyond.pdf https://www.cs.umd.edu/~ben/papers/Shneiderman1993Beyond.pdf Easier to read text version: http://www.cs.umd.edu/hcil/trs/93-03/93-03.html http://www.cs.umd.edu/hcil/trs/93-03/93-03.html Beyond Intelligent Machines, IEEE Software, January 1993 https://news.ycombinator.com/item?id=22744573 https://news.ycombinator.com/item?id=22744573 DonHopkins on April 1, 2020 | parent | context | favorite | on: Beyond Intelligent Machines: Just Do It (1993) I love this article! I worked with Ben Shneiderman at HCIL, and when he sent me a copy of this article to review, I was inspired by the "Dynamic Queries" of his "Dynamic Home Finder" demo, to implement something like that for SimCity (the Frob-O-Matic window). [Video and transcript below.] Here are some of the main points of his article: Don't label machines as "intelligent". It limits the imagination. We should have greater ambitions. Enable humans to accomplish tasks that weren't before possible, instead of trying to enable machines to accomplish tasks people can already do. Predictability and control are desirable qualities. Give users the feeling of mastery, competence, and understanding, sense of accomplishment. The "intelligent machine" label limits or even eliminates human responsibility. If you treat machines like people, you're likely to end up treating people like machines. Ben Shneiderman called his lab HCIL instead of CHIL, to explicitly put Humans before Computers. Here's his description of "Dynamic Queries" and the "Dynamic Home Finder": >Dynamic Queries. These animations let you rapidly adjust query parameters and immediately display updated result sets, which makes them very effective when a visual environment like a map, calendar, or schematic diagram is available. The immediate display of results lets users more easily develop intuitions, discover patterns, spot trends, find exceptions, and see anomalies. >Figure 2 shows a screen from Dynamic HomeFinder, a prototype interface for real-estate agents that uses dynamic queries, written by Christopher Williamson of UM. Users can adjust the cost, number of bedrooms, and location of the A and B markers, among other characteristics, and points of light appear on a map to indicate a home that matches their specifications. Clicking on a point of light brings up a home description or image. >Users of Dynamic HomeFinder can execute up to 100 queries per second (rather than one query per 100 seconds as is typical in a database query language), producing a revealing animated view of where high- or low-price homes are found-and there are no syntax errors. >Our empirical study of 18 users showed Dynamic HomeFinder to be more effective than a natural-language interface using Q&A from Symantec (C. Williamson and B. Shneiderman, The Dynamic HomeFinder: Evaluating Dynamic Queries in a Real-Estate Information Exploration System," Proc. SIG Information Retrieval, ACM Press, 1992, pp. 338-346). https://www.cs.umd.edu/users/ben/papers/Williamson1992dynamic.pdf https://www.cs.umd.edu/users/ben/papers/Williamson1992dynami... That inspired me to implement a version of the "Dynamic Home Finder" in SimCity: Multi Player SimCityNet for X11 on Linux (Dynamic Home Finder / Frob-O-Matic demo at 3:35). https://www.youtube.com/watch?v=_fVl4dGwUrA&t=3m35s https://www.youtube.com/watch?v=_fVl4dGwUrA&t=3m35s Transcript: Here's an interesting thing inspired by Ben Shneiderman. We can look at the ... This is the "Dynamic Zone Filter". So we are going to set this to be dynamic. Now it's going to show all the zones, but it's not going to show the ones that don't pass this filter. So this filter is currently all the way open. Now we're going to change population density. This is a two-ended slider. This is the segment of the population density, from zero to 81. So everything else will disappear. Say I'm looking for a home. I want low population density. But I want high rate of growth. And now, these are the places that have low population density and high rate of growth. And then you can just interactively ... So each of these filters out some of the places. So you can look at ... I don't want high traffic density. I don't want any pollution. I don't want any crime. I want land value to be high. I'm getting pretty picky. So maybe I'll deal with more people. Lower rate of growth. I'm too picky about pollution. I'm too picky about land value. That's it. So basically, Ben Shneiderman demonstrated this as the "Dynamic Home Finder", and I realized that SimCity has all these layers of information that it can draw on, as fictitious as they are, to do that kind of real time, interactive, smooth database query. It's just a much higher bandwidth way to query a database than is conventionally used. Anyway, that was the dynamic zone finder. Also: Here's some email I wrote to Ben about this article, after re-reading the article again in 2009: Date: 28/02/2009 18:51 Subject: Re-reading "Beyond Intelligent Machines" A long time ago, you sent us (me, Brad Myers, Jack Callahan and Mark Weiser) a preview of your IEEE Software article "Beyond Intelligent Machines", which I ran across and have re-read. It's still delightful, inspiring and relevant today. Speech synthesis and recognition has come a long way, to the point where the "talking car" scenario is quite common (TomToms that speak and recognize street names and addresses). But I think the development of user-customizable user interfaces has been dreadfully stalled since HyperCard died. An "Interactive Learning Environment" is a great description of what I'm developing SimCity into. Remember when you visited CMU and I showed you the version of SimCity with the dynamic query feature, an "homage" to your Dynamic Home Finder? You could dial a series of filters on spatial properties like population density, pollution, traffic, land value, police coverage, etc. That code is now open source, and I'm redeveloping it! It's now very easy to write Python code to dynamically query and filter the map and data layers, and visualize with transparent colored tiles and pixel overlays, depending on arbitrary Python functions over the state of the map and its data layers. And it's also possible to script agents in Python! Here's a picture of SimCity with the traffic overlay enabled (showing yellow/orange/red haze over high traffic areas), with some PacMan agents, which are programmed to follow roads, go towards high traffic and eat the cars! Notice the clear road behind each PacMan! http://www.donhopkins.com/home/images/SimCityPacMan.png http://www.donhopkins.com/home/images/SimCityPacMan.png -Don Micropolis Online (SimCity) Web Demo: https://www.youtube.com/watch?v=8snnqQSI0GE https://www.youtube.com/watch?v=8snnqQSI0GE A demo of the open source Micropolis Online game (based on the original SimCity Classic source code from Maxis), running on a web server, written in C++ and Python, and displaying in a web browser, written in OpenLaszlo and JavaScript, running in the Flash player. Developed by Don Hopkins. Source Code: https://github.com/SimHacker/micropolis https://github.com/SimHacker/micropolis HAR 2009 talk: Constructionist Educational Open Source https://medium.com/@donhopkins/har-2009-lightning-talk-transcript-constructionist-educational-open-source-simcity-by-don-3a9e010bf305 https://medium.com/@donhopkins/har-2009-lightning-talk-trans...
Bellary 1y ago
[dead]
careful_ai 1y ago
[dead]

979 points by walterbell 1y ago

> anyone serious about designing for AI should consider non-copilot form factors that more directly extend the human mind. Aren't auto-completes doing exactly this? It's not a co-pilot in the sense of a virtual human, but already more in the direction of a HUD. Sure you can converse with LLMs but you can also clearly just send orders and they eagerly follow and auto-complete. I think what the author might be trying to express in a quirky fashion, is that AI should work alongside us, looking in the same direction as we are, and not being opposite to us at the table, staring at each other's and arguing. We'll have true AI when they'll be doing our bidding, without any interaction from us.

gklitt 1y ago
Author here. Yes, I think the original GitHub Copilot autocomplete UI is (ironically) a good example of a HUD! Tab autocomplete just becomes part of your mental flow. Recent coding interfaces are all trending towards chat agents though. It’s interesting to consider what a “tab autocomplete” UI for coding might look like at a higher level of abstraction, letting you mold code in a direct-feeling way without being bogged down in details.
- juped 1y ago
  If that's what you think a HUD is, then a HUD is definitely way, way worse. Rather than a copilot sitting next to you, that's someone grabbing your hands and doing things with them while you're at the controls. But if I invoke the death of the author and pretend HUD meant HUD, then it's a good point: tools are things you can form a cybernetic system with, classic examples being things like hand tools or cars, and you can't form a cybernetic system with something trying to be an "agent". To be in a cybernetic system with something you need predictable control and fast feedback, roughly.
  - CharlieDigital 1y ago
    I take "HUD" here to just mean "in your line of vision" or "in the context of your actual task" or minimizing any context switch to another interaction (chat window). Rather I think most implementations of HUD AI interactions so far have been quite poor because the interaction model itself is perhaps immature and no one has quite hit the sweet spot yet (that I know of). Tab autocompletion is a simple gesture, but trades off too much control for more complex scenarios and is too easy to accidentally activate. Inline chat is still a context switch and also not quite right.

samfriedman 1y ago

On this topic, can anyone find a document I saw on HN but can no longer locate? A historical computing essay, it was presented in a plaintext (monospaced) text page. It outlined a computer assistant and how it should feel to use. The author believed it should be unobtrusive, something that pops into awareness when needed and then gets back out of the way. I don't believe any of the references in TFA are what it was.

thehappypm 1y ago
Designing Calm Technology?
gsmt 1y ago
beyond intelligent machines?

cadamsdotcom 1y ago

Love the idea & spitballing ways to generalize to coding.. Thought experiment: as you write code, an LLM generates tests for it & the IDE runs those tests as you type, showing which ones are passing & failing, updating in real time. Imagine 10-100 tests that take <1ms to run, being rerun with every keystroke, and the result being shown in a non-intrusive way. The tests could appear in a separated panel next to your code, and pass/fail status in the gutter of that panel. As simple as red and green dots for tests that passed or failed in the last run. The presence or absence and content of certain tests, plus their pass/fail state, tells you what the code you’re writing does from an outside perspective. Not seeing the LLM write a test you think you’ll need? Either your test generator prompt is wrong, or the code you’re writing doesn’t do the things you think they do! Making it realtime helps you shape the code. Or if you want to do traditional TDD, the tooling could be reversed so you write the tests and the LLM makes them pass as soon as you stop typing by writing the code.

hnthrowaway121 1y ago
Yes the reverse makes much more sense to me. AI help to spec out the software & then the code has an accepted definition of correctness. People focus on this way less than they should I think
callc 1y ago
Humans writing the test first and LLM writing the code is much better than the reverse. And that is because tests are simply the “truth” and “intention” of the code as a contract. When you give up the work of deciding what the expected inputs and outputs of the code/program is you are no longer in the drivers seat.
- JimDabell 1y ago
  > When you give up the work of deciding what the expected inputs and outputs of the code/program is you are no longer in the drivers seat. You don’t need to write tests for that, you need to write acceptance criteria.
  - 1y ago

jawns 1y ago

The author gives an example of a HUD: an AI that writes a debugger program to enable the human developer to more easily debug code. But why should it be only the human developer who benefits? What if that debugger program becomes a tool that AI agents can use to more accurately resolve bugs? Indeed, why can't any programming HUD be used by AI tools? If they benefit humans, wouldn't they benefit AI as well? I think we'll be pretty quickly at the point where AI agents are more often than not autonomously taking care of business, and humans only need to know about that work at critical points (like when approvals are needed). Once we're there, the idea that this HUD concept should be only human-oriented breaks down.

deleted 1y ago

[deleted]

hi_hi 1y ago

Doesn't it all come down to "what is the ideal interface for humans to deal with digital information"? We're getting more and more information thrown at us each day, and the AIs are adding to that, not reducing it. The ability to summarise dense and specialist information (I'm thinking error logs, but could be anything really) just means more ways for people to access and view that information who previously wouldn't. How do we, as individuals, best deal with all this information efficiently? Currently we have a variety of interfaces, websites, dashboards, emails, chat. Are all these necessary anymore? They might be now, but what about the next 10 years. Do I even need to visit a companies website if can get the same information from some single chat interface? The fact we have AIs building us websites, apps, web UI's just seems so...redundant.

moomoo11 1y ago
I like the smartphone. It’s honestly perfect and underutilized.
sipjca 1y ago
yep I think this is the fundamental question as well, everything else is intermediate
guardiang 1y ago
Every human is different, don't generalize the interface. Dynamically customize it on the fly.
AlotOfReading 1y ago
Websites were a way to get authoritative information about a company, from that company (or another trusted source like Wikipedia). That trust is powerful, which is why we collectively spent so much time trying to educate users about the "line of death" in browsers, drawing padlock icons, chasing down impersonator sites, mitigating homoglyph attacks, etc. This all rested on the assumption that certain sites were authoritative sources of information worth seeking out. I'm not really sure what trust means in a world where everyone relies uncritically on LLM output. Even if the information from the LLM is usually accurate, can I rely on that in some particularly important instance?

roywiggins 1y ago

> The agentic option is a “copilot” — a virtual human who you talk with to get help flying the plane. If you’re about to run into another plane it might yell at you “collision, go right and down!” Planes do actually have this now. It seems to work okay: https://en.m.wikipedia.org/wiki/Traffic_collision_avoidance_system https://en.m.wikipedia.org/wiki/Traffic_collision_avoidance_...

gklitt 1y ago
Author here. I thought about including TCAS in the article but it felt too tangential… You’re right that there’s a voice alert. But TCAS also has a map of nearby planes which is much more “HUD”! So it’s a combo of both approaches. (Interestingly it seems that TCAS may predate Weiser’s 1992 talk)

droideqa 1y ago

I want a link to the GitHub for this[0] which he linked to. Makes Prolog quite interesting. [0]: https://www.geoffreylitt.com/2024/12/22/making-programming-more-fun-with-an-ai-generated-debugger https://www.geoffreylitt.com/2024/12/22/making-programming-m...

johnisgood 1y ago
Yeah, it looks interesting. I would give it a spin. Maybe then I could finally make a MUD in Prolog. :D

thepuglor 1y ago

I'd rather flip this around and be in a fully immersive environment watching agents do things in such a way that I am able to interject and guide in realtime. How do we build that, and build it in such a way that the content and delivery of my guidance becomes critical to what they learn? The best teacher gets the best AI students.

sitkack 1y ago
> fully immersive environment Start with a snapshot of what you are envisioning using Blender.

aaron695 1y ago

This is a mess. The 1992 talk wasn't at all about AI and since then our phones have given us "ubiquitous computing" en masse. The original talk required no 'artificial intelligence' for relevance which makes it strange to apply to todays artificial intelligence. The original talk made good points for instance "voice recognition" has been solved forever at a reasonable level, yet people kept claiming if it was 'better' a 'magic experience' would pop out as if voice was different to typing. Idiots have been around for a long time. Don't get what OP is trying to say. 'AI HUD metaphors' are very hard, that's why they are not ubiquitous, they require constant input. Spellcheck runs every character typed. Agents are because of less $. 'Hallucinations' also make 'AI HUD metaphors' problematic, for spellcheck squiggly red lines would be blinking on off all over the page as a LLM keeps coming back with different results.

precompute 1y ago
FYI, you're shadowbanned.

jpm_sd 1y ago

This is how ship's AI is depicted in The Expanse (TV series) and I think it's really compelling. Quiet and unobtrusive, but Alex can ask the Rocinante to plot a new course or display the tactical situation and it's fast, effective and effortlessly superhuman with no back-talk or unnecessary personality. Compare another sci-fi depiction taken to the opposite extreme: Sirius Cybernetics products in the Hitchhikers Guide books. "Thank you for making a simple door very happy!"

skydhash 1y ago
I may remember wrongly, but I don't believe the expanse was depicting AI. It was more powerful computation and unobtrusive interfaces. There was nothing like Jarvis. Rocinante was a war vessel and all its feature was tailored to that. I believe even the mechanical suits of the Martians were very manual (no cortana a la Master Chief).
- jpm_sd 1y ago
  That's exactly what this article is about, though. Assistive systems that are incredibly capable without needing to have a fake personality.

kova12 1y ago

aren't we missing the point that co-pilot is there in case pilot gets incapacitated?

melagonster 1y ago
The pilot is the copilot today. These computer can handle most of the tasks automatically.

benjaminwootton 1y ago

I think there is a third and distinct model which is AI that runs in the background autonomously amd over a long period and pushes things to you. It can detect situations intelligently, do the filtering, summarisation of what’s happening and possibly a recommendation. This feels a lot more natural to me, especially in a business context when you want to monitor for 100 situations about thousands of customers.

skydhash 1y ago
Actually defining those situations and collecting the data (which should help identify those situations) are the hard parts. Having an autonomous system that do it has been solved for ages.

ares623 1y ago

It should have a paperclip mascot

stan_kirdey 1y ago

needed to find a kids’ orthodontist. made a tiny voice agent: feed it numbers, it calls, asks about price/availability/insurance, logs the gist. it kind of worked. the magic was the smallest UI around it: - timeline of dials + retries - "call me back" flags - when it tried, who picked up - short summaries with links to the raw transcript once i could see the behavior, it stopped feeling spooky and started feeling useful. so yeah, copilots are cool, but i want HUDs: quiet most of the time, glanceable, easy to interrupt, receipts for every action.

perching_aix 1y ago

Been thinking about something similar, from fairly grounded ideas like letting a model autogenerate new features with their own name, keybind and icon, all the way to silly ideas, like letting a model synthesize arbitrary shader code and just letting it do whatever within the viewport. Think the entire UI being created on the fly specifically for the task you're working on, constantly evolving in mesh with your workflow habits. Now if only I went beyond being an idea man...

caleblloyd 1y ago
The reason we are not seeing this in mainstream software may also be due to cost. Paying for tokens on every interaction means paying to use the app. Upfront development may actually be cheaper, but the incremental cost per interaction could cost much more in the long term, especially if the software is used frequently and has a long lifetime. As the cost of tokens goes down, or commodity hardware can handle running models capable of driving these interactions, we may start to see these UIs emerge.
- perching_aix 1y ago
  Oh yeah, I was 100% thinking in terms of local models.

ag2s 1y ago

A very relevant article from lesswrong, titled Cyborgism https://www.lesswrong.com/s/f2YA4eGskeztcJsqT/p/bxt7uCiHam4QXrQAA https://www.lesswrong.com/s/f2YA4eGskeztcJsqT/p/bxt7uCiHam4Q...

sothatsit 1y ago

AI building complex visualisations for you on-the-fly seems like a great use-case. For example, if you are debugging memory leaks in a specific code path, you could get AI to write a visualisation of all the memory allocations and frees under that code path to help you identify the problem. This opens up an interesting new direction where building visualisations to debug specific problems is probably becoming viable. This idea reminds me of Jonathan Blow's recent talk at LambdaConf. In it, he shows a tool he made to visualise his programs in different ways to help with identifying potential problems. I could imagine AI being good at building these. The talk: https://youtu.be/IdpD5QIVOKQ?si=roTcCcHHMqCPzqSh&t=1108 https://youtu.be/IdpD5QIVOKQ?si=roTcCcHHMqCPzqSh&t=1108

federiconafria 1y ago
Independently of visualizations, I think LLMs allow in general the creation of ad-hoc tools. I've experienced the case where asking for a quick python script was faster and more powerful than learning how to use a cli to interact with an API.
- twbarber 1y ago
  Anecdotally, the low time to prototype has had me throwing usable tool projects up on my local network. I wrote a meal planner and review tool for our weekly vegetable share, and local board game / book library that guests can connect to and reference. It scratches the itch to build and ship with the benefit of a growing library of low scope, high performance, highly customized web tools I can write over a few hours in an evening instead of devoting weekends to it. It feels like switching from hand to power tools
- borgel 1y ago
  Same. It's very handy to be able to ask for one-off tools for things that I _could_ try to figure out, but probably wouldn't be worth the time. My favorite example so far was a Python script to help debug communication between a microcontroller and a motor driver. I was able to dump the entire datasheet PDF for the driver into Gemini and ask it for a Python CLI to decode communication traffic, and 30 seconds later it was done. Fantastic bang/buck ratio!

wewewedxfgdf 1y ago

Kind of a weird article because the computer systems that is "invisible" i.e. an integrated part of the flight control systems - is exactly what we have now. He's sort of arguing for .... computer software. Like, we have HUDs - that's what a HUD is - it's a computer program.

CGamesPlay 1y ago
A HUD is typically non-interactive, which is the core distinction he’s advocating for. The “copilot” responds to your requests, the “HUD” surfaces relevant information passively.
- skydhash 1y ago
  Isn't this one of the core selling of an IDE compared to a simple editor? You have the analyzer running in the background and telling you things. And a host of buttons around the editor to act on stuff, all within the context of a project.

nioj 1y ago

Concurrent posting from 5 hours ago (currently no comments): https://news.ycombinator.com/item?id=44705018 https://news.ycombinator.com/item?id=44705018

satisfice 1y ago

Yes, this is a non-creepy way of applying AI.

SilverElfin 1y ago

Isn’t that what all the AI browsers like Comet, or the things like Cluey are trying to do

eboynyc32 1y ago

Excited for the next wave of ai innovation.

clbrmbr 1y ago

A thought-provoking analogy! What comes immediately to mind for me is using embeddings to show closest matches to current cursor position on the right tab for fast jumping to related files.

henriquegodoy 1y ago

Great post! i've been thinking along similar lines about human-AI interfaces beyond the copilot paradigm. I see two major patterns emerging: Orchestration platforms - Evolution of tools like n8n/Make into cybernetic process design systems where each node is an intelligent agent with its own optimization criteria. The key insight: treat processes as processes, not anthropomorphize LLMs as humans. Build walls around probabilistic systems to ensure deterministic outcomes where needed. This solves massive "communication problems" Oracle systems - AI that holds entire organizations in working memory, understanding temporal context and extracting implicit knowledge from all communications. Not just storage but active synthesis. Imagine AI digesting every email/doc/meeting to build a living organizational consciousness that identifies patterns humans miss and generates strategic insights. just explored more about it on my personal blog https://henriquegodoy.com/blog/stream-of-consciousness https://henriquegodoy.com/blog/stream-of-consciousness

ankit219 1y ago

The current paradigm is driven by two factors: one is the reliability of the models and that constraints how much autonomy you can give to an agent. Second is about chat as a medium which everyone went to because ChatGPT became a thing. I see the value in HUDs, but only when you can be sure output is correct. If that number is only 80% or so, copilots work better so that humans in the loop can review and course correct - the pair programmer/worker. This is not to say we need ai to get to higher levels of correctness inherently, just that systems deployed need to do so before they display some information on HUD.

psychoslave 1y ago
This is missing the addictive/engaging part of a conversational interface for most people out there. Which is in line with the critics highlighted in the fine article. Just because most people are fond of it doesn't actually mean it improves their life, goals and productivity.

makaking 1y ago

> "non-copilot form factors that more directly extend the human mind." I think Cursor's tab completion and next edit prediction roughly fits the pattern, you don't chat, you don't ask or explain, you just do... And the more in coherent your actions are the more useful the HUB becomes.

ravila4 1y ago

I think one key reason HUDs haven’t taken off more broadly is the fundamental limitation of our current display medium - computer screens and mobile devices are terrible at providing ambient, peripheral information without being intrusive. When I launch an AI agent to fix a bug or handle a complex task, there’s this awkward wait time where it takes too long for me to sit there staring at the screen waiting for output, but it’s too short for me to disengage and do something else meaningful. A HUD approach would give me a much shorter feedback loop. I could see what the AI is doing in my peripheral vision and decide moment-to-moment whether to jump in and take over the coding myself, or let the agent continue while I work on something else. Instead of being locked into either “full attention on the agent” or “completely disengaged,” I’d have that ambient awareness that lets me dynamically choose my level of involvement. This makes me think VR/AR could be the killer application for AI HUDs. Spatial computing gives us the display paradigm where AI assistance can be truly ambient rather than demanding your full visual attention on a 2D screen. I picture that this would be especially helpful for help on more physical tasks, such as cooking, or fixing a bike.

elliotec 1y ago
You just described what I do with my ultrawide monitor and laptop screen. I can be fully immersed in a game or anything and keep Claude in a corner of a tmux window next to a browser on the other monitor and jump in whenever I see it get to the next step or whatever.
- ravila4 1y ago
  It’s a similar idea, but imagine you could fire off a task, and go for a run, or do the dishes. Then be notified when it completes, and have the option to review the changes, or see a summary of tests that are failing, without having to be at your workstation.
  - bigyabai 1y ago
    I kinda do this today, with Alpaca[0]'s sandboxed terminal runner and GSConnect[1] syncing the response notifications to my phone over LAN. [0] https://jeffser.com/alpaca/ https://jeffser.com/alpaca/ [1] https://github.com/GSConnect/gnome-shell-extension-gsconnect https://github.com/GSConnect/gnome-shell-extension-gsconnect

Der_Einzige 1y ago

Actual HUDs, like the ones for cars - along with vector displays, and laser display tech in general, is criminally undervalued and underutilized.

radres 1y ago

One of the simplest and best working applications of AI is the gpts living in your clipboard. Any prompt/workflow you have is assignable to a shortcut on demand and it pops up a chat window with response on demand. It's been game changing honestly.

furyofantares 1y ago
Can you elaborate on this?

furyofantares 1y ago

I'm very curious if a toggle would be useful that would display a heatmap of a source file showing how surprising each token is to the model. Red tokens are more likely to be errors, bad names, or wrong comments.

WithinReason 1y ago
previously undefined variable and function names would be red as well
- Too 1y ago
  All editors do this already.
  - WithinReason 1y ago
    No, I mean when an LLM encounters a previously unseen name it doesn't expect it so it would be red, even though it's perfectly valid.
    - jakelazaroff 1y ago
      I don't think that's necessarily true. I've definitely seen LLMs hallucinating variables they never defined.
    - furyofantares 1y ago
      I'm imagining everything the LLM could produce (with a given top_k setting) would be shades of green to yellow. Just outside of that orange, and far outside red. LLMs generate new functions all the time, I'd guess these would be light green, maybe the first token in the name would be yellow and it would get brighter green as the name unfolds. The logits are probably all small when in the global scope where it's not clear what will be defined next. I'm not imagining mapping logits directly to heat, the ordering of tokens seems much more appropriate.
- fwip 1y ago

yurimo 1y ago

I might be wrong but isn't the HUD the author suggesting for coding is basically AREPL? For debugging I can see it work, but chatbox and inline q&a I feel has awider application. On a wider note, I buy the argument for alternative interfaces other than chat, but chat permeates our lives every day, smartphone is full of chat interfaces. HUD might be good for AR glasses though, literal HUD.

aantix 1y ago

I would love to see a HUD that allows me to see the change that corresponds to Claude Code's TODO item. I don't want inline comments as those accumulate, don't get cleaned up appropriately by the LLM.

kn81198 1y ago

About a decade back Bret Victor [1] talked about how his principle in life is to reduce the delay in feedback, and having faster iteration cycles not just helps in doing things (coding) better but also contributes to new creative insights. He had a bunch of examples built to showcase alternative ways of coding, which is very close to being HUDs - one example shown in the OP is very similar to the one he presents to "step through time to figure out the working of the code". [1]: https://www.youtube.com/watch?v=PUv66718DII https://www.youtube.com/watch?v=PUv66718DII

latorf 1y ago
This!!! Thank you, couldn't have said it better. One of the best talks I've ever seen, and a huge inspiration for a lot of work I've done.

piker 1y ago

Absolutely agree, and spellchecker is a great analogy. I've recently been snoozing co-pilot for hours at a time in VS Code because it’s adding a ton of latency to my keystrokes. Instead, it turns out that `rust_analyzer` is actually all that I need. Go-to definition and hover-over give me exactly what the article describes: extra senses. Rust is straightforward, but the tricky part may be figuring out what additional “senses” are helpful in each domain. In that way, it seems like adding value with AI comes full circle to being a software design problem. ChatGPT and Claude are great as assistants for strategizing problems, but even the typeahead value seems to me negligible in a large enough project. My experience with them as "coding agents" is generally that they fail miserably or are regurgitating some existing code base on a well known problem. But they are great at helping config things and as teachers in (the Socratic sense) to help you get up-to-speed with some technical issue. The heads-up display is the thesis for Tritium[1], going back to its founding. Lawyers' time and attention (like fighter pilots') is critical but they're still required in the cockpit. And there's some argument they always will be. [1] https://news.ycombinator.com/item?id=44256765 https://news.ycombinator.com/item?id=44256765 ("an all-in-one drafting cockpit")

kibwen 1y ago
On the topic of Rust IDE plugins that give you more senses, take a look at Flowistry: https://github.com/willcrichton/flowistry https://github.com/willcrichton/flowistry . It's not AI, it's using information flow analysis.

Animats 1y ago

Nobody mentioned Manna [1] yet? That suggests a mostly audio headset giving orders. There is a real-world version using AR glasses.[2] [1] https://marshallbrain.com/manna1 https://marshallbrain.com/manna1 [2] https://www.six-15.com/vision-picking https://www.six-15.com/vision-picking

utf_8x 1y ago

You know that feature in JetBrains (and possibly other) IDEs that highlights non-errors, like code that could be optimized for speed or readability (inverting ifs, using LINQ instead of a foreach, and so on)? As far as I can tell, these are just heuristics, and it feels like the perfect place for an “AI HUD.” I don’t use Copilot or other coding AIs directly in the IDE because, most of the time, they just get in the way. I mainly use ChatGPT as a more powerful search engine, and this feels like exactly the kind of IDE integration that would fit well with my workflow.

obscure-enigma 1y ago

> design the cockpit so that the human pilot is naturally aware of their surroundings. This is a design interface problem. Self-driving cars can easily ingest this HUD. This is the reason what makes Apple's AI different from other microservice-like AI. The spell checker, rewrite, proofread are naturally integrated into the UI to the extent it doesn't feel like AI powered operations.

iddan 1y ago

This is what we are going for in Closer - an AI native experience that extends the mind of salespeople. https://closer.so https://closer.so

evrennetwork 1y ago

[dead]

Oras 1y ago

Great post, informative and precise. I think the challenge is primarily the context and intent. The spellchecker knows my context easily, and there is a setting to choose from (American English, British English, etc.), as well as the paragraphs I'm writing. The intent is easy to recognise. While in a codebase, the context is longer and vaguer, the assistant would hardly know why I'm changing a function and how that impacts the rest of the codebase. However, as the article mentions, it may not be a universal solution, but it's a perspective to consider when designing AI systems.

thinkingemote 1y ago

We also need what goes along with HUDs : switches, nubs, switches, dials. Actual controls. Although we are talking HUDs, I'm not really talking about UI widgets having the good old skew-morphism or better buttons. In the cockpit the pilot doesn't have his controls on a touch screen, he has an array of buttons and dials and switches all around him. It's these controls that are used in response to what the pilot sees on the HUD and it's these controls that change the aircraft according to the pilots will, which in turn change what the HUD shows.

TimCTRL 1y ago

makes sense, i guess tools like cluely are HUDs

barrenko 1y ago
Yup, not affiliated, but this is what cluely kinda does - https://nitter.poast.org/im_roy_lee/status/1938719006002921751 https://nitter.poast.org/im_roy_lee/status/19387190060029217...

_samjarman 1y ago

I'd love to mount a top-down webcam above me when I work on electronics and have a real time feed into ChatGPT or similar for Q&A, for a real time coach that can see. One day... :)

precompute 1y ago

HUDs are primarily of use to people that are used to parsing dense visual information. Wanting HUDs on a platform that promises to do your work for you is quite pointless.

afro88 1y ago

A HUD is an even more "confident" display of data than text though. What do you do with a HUD that hallucinates? Is there a button on each element that shows you sources?

Dilettante_ 1y ago
I could imagine a system in which the model only chooses which data points to show at what time, but the actual passing is still handled by good old deterministic programming.
alwa 1y ago
Night-vision optics come to mind: prone to noise and visual artifacts, and especially strange under certain edge conditions. Some of their specs tend to be strictly inferior to a Mark I Eyeball—narrow FOV, limited focusing power, whatever else. But an operator learns to intuit which aspects to trust and which to double-check. The fact that it’s an “extra sense” can outweigh the fact that it’s not a perfect source of truth, no? Trust the tech where it proves useful to you, and find ways to compensate (or outright don’t use it) where it’s not.
- bpfrh 1y ago
  I think the major different between that and KI is that the the Night-Vision is mostly static and knowable through experience/teaching, while KI is an ever moving target where you experience is zero on every new display/question. I think the idea of an hud is better than the current paradigm, but it doesn't solve the fundamental problem.

simianwords 1y ago

I believe this is the framework vs library debate but in a different context. HUD is like me using a library of AI enabled tools. The human agency is given more importance here - they decide how to use the tools to suit their purpose. Copilot is more like a framework where an AI system exists which tells me what to do (a bit like the inverse of a library).

amelius 1y ago

I want an AI HUD that blocks ads.

Cthulhu_ 1y ago
What if you could get AR glasses that translates ads into their core message like "CONSUME", "MARRY AND REPRODUCE" and "OBEY"? That would be cool. You could make a movie out of that!

est 1y ago

It's like the magic moment when GUIs replace CLIs. I remember the first time I started up Win95 from DOS days. Stunning.

Cthulhu_ 1y ago
Or terminal GUIs in DOS. Before (and during) Windows, our go-to was 3DMENU [0] for starting up games and the like. [0] https://archive.org/details/3DMENUPLUSDOS https://archive.org/details/3DMENUPLUSDOS

smolder 1y ago

[flagged]

smolder 1y ago

Let's stop allowing this clearly incentivized hype for investable shit and have honest conversation, please.

anaisbetts 1y ago

I mean, fundamentally the reason that we don't build most AI HUD ideas right now is that they are either stupidly insanely expensive at scale, or you have to run the AI on-device on an NPU so underpowered that it's going to give you really really bad results. Lots of great ideas in this space but it's tough to make something that delivers value and also is economically viable

smolder 1y ago

[flagged]

tomhow 1y ago
It's explicitly against the guidelines to comment like this: Please don't post insinuations about astroturfing, shilling, brigading, foreign agents, and the like. It degrades discussion and is usually mistaken. If you're worried about abuse, email hn@ycombinator.com and we'll look at the data. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html

vidarh 1y ago

> “You’ll no more run into another airplane than you would try to walk through a wall.” People do walk into walls, though. I see the appeal of the idea, but it's not a replacement for something that actually agressively demands your attention when there are high risk events.

kevinphy 1y ago

This whole Copilot vs HUD debate instantly brought to mind a classic Japanese anime from 1991 called Future GPX Cyber Formula (https://en.wikipedia.org/wiki/Future_GPX_Cyber_Formula https://en.wikipedia.org/wiki/Future_GPX_Cyber_Formula). Yeah, it’s a racing anime set in the then-distant future of 2015, where cars come with full-on intelligent AIs. The main character’s car, Asurada, is basically a "Copilot" in every sense. It was designed by his dad to be more than just a tool, more like a partner that learns, adapts, and grows with the driver. Think emotional support plus tactical analysis with a synthetic voice. Later in the series, his rival shows up driving a car that feels very much like a HUD concept. It's all about cold data, raw feedback, and zero bonding. Total opposite philosophy. What’s wild is how accurately it captures the trade-offs we’re still talking about in 2025. If you’re into human-AI interaction or just want to see some shockingly ahead-of-its-time design thinking wrapped in early '90s cyber aesthetics, it’s absolutely worth a watch.

yahoozoo 1y ago
Anywhere to stream this?
- deleted 1y ago
  [deleted]
- petercooper 1y ago
  Quality is so-so but it appears to be on YouTube: https://www.youtube.com/watch?v=cXtvPUtSTRY https://www.youtube.com/watch?v=cXtvPUtSTRY
1317 1y ago
any release you'd recommend?
- layer8 1y ago
  669198800283 if you need English subtitles. VPXY-71923 if you’re fine with Japanese.
1y ago

torginus 1y ago

I read you, but unfortunately despite the limitations of current AI tech, all companies covertly/overtly market themselves as human replacement not human augmentation solutions with the aim of cutting out expensive experts out of the loop, not making them more productive, even though the messaging might say otherwise from time to time.

endymion-light 1y ago

I think there's room for a mixture of both use-cases. I think personally programming is far more like being a mixture of a pilot and an engineer. There are times, like in this article where it's far more useful to have a HUD, while i'm focused, i'd much rather have a better eye on things that are interesting to me. I often find things like cursor do the opposite - they slow down your processing time, you shift into an environment that often feels like i'm correcting a junior engineers work rather than expanding my knowledge of the codebase. For example, if i'm working on authentication logic, i'd much rather see a smart heads up display, you look at a function and get advice on where this might mess other things up in the codebase, edge cases, etc. A smarter form of current IDES that doesn't mean you click through 50 different files to work out that this third party package doesn't work on specific code. A HUD in this case is ideal. But I find there's a more detailed, slower development, I still really often use the chat function on claude mixed with Obsidian to hold bits of information i've found useful, this is more related to getting a deeper understanding of certain concepts. As a stupid developer, I often find I might need something explained 20 times in 20 different ways, and actually a predictive text model is perfect in so many circumstances to explain a massive algorithm step by step. It's ideal for things like shader code, where I might come back to it 6 months later and want to work out what was in my head at the time, a historic chat is perfect for those things. There's definitely a balance to be struck, I think now that the hype cycle is peaking we can hopefully seperate the profit seeking AI tools from the useful day to day knowledge expansion. Currently it feels like the discovery of perspective in the reneissance - instead of using it to further advancements, we're attempting to sell perspective courses to people.

camdroidw 1y ago

AI powered this, AI powered that, what most people don't realise is that the next game changer would be Al powered batteries

nunodonato 1y ago
what?

newsclues 1y ago

More sensors and more data, humans become a limiting factor until a better way to communicate the information is used.

your_challenger 1y ago

The Ink & Switch guys always have the best-est of ideas!

roca 1y ago

We have explored that sort of debugging/visualization tool in https://pernos.co https://pernos.co. We built it before the age of genAI, but I think for coming up with powerful visualizations AI is neither necessary nor (yet) sufficient.

OJFord 1y ago

I do think typing text prompts at LLMs will go down as a barely remembered brief period in history, that seems quaint and dumb to anyone who remembers or is told about it. (I don't know what we'll be doing instead, I just think text prompts feel dumb.)

kotaKat 1y ago

Enough AI copilots and AI HUDs, we need “none of this bullshit forced upon us”.

julik 1y ago

So... Google Glass?

cmrdporcupine 1y ago

100% agree with this. I want augmentation not automation. That said, CoPilot in the form of "autocomplete" is kind of that. I have been enjoying the hell out of Claude Code, but I'd feel much better about it if it wasn't a case of "here take this pile of diffs" and had a mode Socratic modality. A tool that works with me instead of for me, because I'm going to have to review everything it does anyways.

aleksituk 1y ago

I like HUDs as an analogy for other forms of content/experience enrichment too e.g. what's a HUD for other forms of interaction than just looking at screens (like audio, or multimodal). Can we embed useful data behind the content we produce/consume or derive personalised versions for the user? -> Text/audio/video tailored for me and my interests? (ie. not just content recommended for me, but the content itself is tailored for me in terms of the insights and practical applications) -> Podcasts I can interact with? -> Audiobooks I can ask questions about? Richer interaction that allows for more back&forth and key insights for the user and use case.

FrustratedMonky 1y ago

I think the OpenAI left panel/ right panel code, seems to fit this.

Havoc 1y ago

Some sort of neural interface feels inevitable at this point

betterhealth12 1y ago

How about contextual AI insights as you're interacting with the world, wherever you are? Like a quality signal. Oh wait, this is something FB/Goog could never do because what would it say about the quality of information hosted...

soco 1y ago
AR for the regular guy is something which I remember installing on my mobile over ten years ago - with low expectations, and it never got any better than that, quite the contrary. Powered by AI or not, seems nobody has found (or looked for) a market for this.
- betterhealth12 1y ago
  Another decade, another attempt? We are aiming for this with our work at Credible (https://mycredible.ai https://mycredible.ai and our chrome extension) - product is still early, I'd be curious to get your first thoughts or reaction.

cat-whisperer 1y ago

I'm thinking AR glasses with AI HUDs and we're basically there - my to-do list just got a whole lot more interesting.

monkpit 1y ago

I was recently thinking, if services offered sufficient api docs/specs you could really just provide an llm to the client and have any client build their own UI to fit their needs. Maybe not that you’d want to. But you could!

bitwize 1y ago

I want AI (machine learning statistical techniques) to convert my brainwaves into text so that I can think my programs into the computer. That would be a useful breakthrough in AI-assisted programming,

_jab 1y ago

There’s a lot of ideation for coding HUDs in the comments, but ironically I think the core feature of most coding copilots is already best described as a HUD: tab completion. And interestingly, that is indeed the feature I find most compelling from Cursor. I particularly love when I’m doing a small refactor, like changing a naming convention for a few variables, and after I make the first edit manually Cursor will jump in with tab suggestions for the rest. To me, that fully encapsulates the definition of a HUD. It’s a delightful experience, and it’s also why I think anyone who pushes the exclusively-copilot oriented Claude Code as a superior replacement is just wrong.

cleverwebble 1y ago
Agreed! I've spent the last few months using Claude Code and Cursor - experimenting with both. For simple tasks, both are pretty good (like identifying a bug given console output) - but when it comes to making a big change, like adding a brand new feature to existing code that requires changes to lots of files, writing tests, etc - it often will make at least a few mistakes I catch on review, and then prompting the model to fix those mistakes often causes it to fix things in strange ways. A few days ago, I had a bug I just couldn't figure out. I prompted Claude to diagnose and fix the issue - but after 5 minutes or so of it trying out different ideas, rerunning the test, and getting stuck just like I did - it just turned off the test and called it complete. If I wasn't watching what it was doing, I could have missed that it did that and deployed bad code. The last week or so, I've totally switched from relying on prompting to just writing the code myself and using tab complete to autocomplete like 80% of it. It is slower, but I have more control and honestly, it's much more enjoyable of an experience.
- jonchurch_ 1y ago
  Drop in a lint rule to fail on skipped tests. Ive added these at a previous job after finding that tests skipped during dev sometimes slipped through review and got merged.
Garlef 1y ago
I think there's a lot of room for even better UI. I'd love to have something that operates more at the codebase level. Autocomplete is very local. (Maybe "tab completion" when setting up a new package in a monorepo? Or make architectural patterns consistent across a whole project? Highlight areas in the codebase where the tests are weak? Or collect on the fly a full view of a path from FE to BE to DB?)

timeinput 1y ago

There is a giant confusing tech stack here, but opening up this simple plain text page resulted in like 100% CPU usage from from gnome-shell when visiting from firefox on a pretty non-customized Ubuntu 24 install. In the firefox task manager nothing really looked odd, but opening that tab and displaying it is insanely CPU intensive. Pausing the autoplaying video makes it seem like a sane web page in terms of CPU usage. I'm surprised how much CPU playing that video consumed.

itslennysfault 1y ago
Very odd. This must be some kind of Ubuntu issue. Out of curiosity I tried it in Chrome, Firefox and Safari on MacOS 15.5 and saw very little CPU usage and no difference with/without the video playing. I don't have a Ubuntu Desktop handy to confirm. Looking at the video it's nothing special. Just a fairly small mp4 using a native html5 video element to play it. Really no reason this should be causing issues.
- timeinput 1y ago
  Yeah, totally agreed. It was quite surprising to me. Who knows it may even be related to the random GPU I happen to have or ....

dreamcompiler 1y ago

The main reasons I don't have "smart" devices in my house: 1. They introduce new and fascinating failure modes that never happened with the old, dumb devices (e.g. if your router fails you lose the ability to control your lights). 2. They demand human attention at the slightest provocation (e.g. the microwave beeps loudly forever when your food is done, every app on your phone insists on interacting with you whenever the company would like to upsell you something, etc.) Item 2 above is what TFA is about. Yes you can often turn this shit off, but that's not the point. The point is you shouldn't have to. Useful technology should never call attention to itself in the manner of someone with narcissistic personality disorder. But what about emergency situations? Glad you asked. Many airplane crashes in modern aircraft have happened because of "warning buzzer overload" which happens when one important system on the aircraft fails and then causes a cascade of secondary warnings, while giving the pilot no insight as to the root cause. A true AI assistant would reason about such situations and guide the pilot toward the root solution. A true coding assistant would do the same kind of reasoning about program errors and suppress multipage error dumps in favor of flagging the root issue.

jFriedensreich 1y ago

My aha moment into this direction happened when i got XR glasses and tried embracing voice agents to see where this leads. Did not expect the HUD aspect be so cool of how this setup works. At first i tried using the glasses as a replacement for my main screen and using voice agents as a replacement to the sidebars and chat windows we are used to, which worked ok but then i started using the main screen and main agent interfaces again but kept the glasses with a second screen on a different visual focal plane connected to the voice agents. Dark mode gets a totally new importance there because black is transparent in XR which is important to work well without obscuring the main screen. I can switch between screens by shifting eye focus and have a sense what the other screen is doing a far better than just how a physical second screen is in the visual periphery. When i need to combine images more i move the focal planes closer together so i can see both screens better at the same time. The voice agents can answer side questions, start research that is needed for the next step of the main workflow or fix minor issues that are not important enough to interrupt the main workflow. Its so obvious how HUD, voice ai, XR and agents can grow together into this new computing environment but i am afraid what happens once android and ios shape that reality. I want this to be part of the web.

kroaton 1y ago
What glasses are you using?
- jFriedensreich 1y ago
  viture pro xr but connected just as external display to my macbook. the experimental hud software is build from scratch for my usecase using web presentation api and a custom svelte framework that does stereoscopic rendering with css matrix transformations.

Einenlum 1y ago

Makes me think that I would love to have an AI tool that not only allows autocomplete but also provides suggestions while I write code. "Have you thought about using useCallback here?" "This feature seems already provided by library X and Y" "It seems like this code already exists in function Foo. Maybe a small refactoring would be helpful"

throwaway12345t 1y ago

This is now (more) dated. Copilots as an interface are dated. The current initiative is full agents with human in the loop at the very start, occasionally the middle and the end. Huds are just good UI, something copilots can natively exist as part of in the form of contextual insights and alerts. We’re moving on to agency where it’s everything else vs an entirely different entity taking the action of flying the plane from take off to landing.

ColonelPhantom 1y ago
I don't think you understood what a "copilot" and a "HUD" is. Confusingly, GitHub Copilot (the original one that suggests completions) is basically a HUD. On the other hand, agents that you give a task are clearly copilots that work via a natural language interface. The article also mentions that agents are copilots: > Here’s another personal example from AI coding. Let’s say you want to fix a bug. The obvious “copilot” way is to open an agent chat and ask it to do the fix.

LeoPanthera 1y ago

All I have ever wanted in my life is a heads-up display for my eyes that puts floating icons over other people telling me their name, birthday, the names of their kids, and what we talked about the last time we met.

maksimur 1y ago
You basically want a cognitive aid device?
- LeoPanthera 1y ago
  Yes please.

bhaktatejas922 1y ago

doing AI code edits on a JS web app is probably the way to do it. probably using something like groq kimi k2 + morph for fast edits now we need a web framework with faster reload times

galaxyLogic 1y ago

Heads-Up-Display is of course useful if it gives you useful information. But you can't tell that display "Ok fly me to Rome". That's what "co-pilot" is about. It can do things, rather than just display information it decides you might find useful.

amir-h 1y ago

I love this analogy. I've been particularly feeling like this regarding AI code reviewers recently - I don't want a copilot that will do their own review, I want a hud that will make it easier for me to understand the change. I've been toying with crafting such a code review tool as a side project recently: https://useglide.ai https://useglide.ai

revskill 1y ago

Agree. Current copilot bullshits lost their context, ahhh no, they know nothing bout context, such a joke.

djtriptych 1y ago

I'm pretty sure this is want I want to work on for the next decade if anyone's looking for a world class UI engineer.

WASDAai 1y ago

You are right and i am working on it for months. It is just not easy to make. Especially if your team is small

DonHopkins 1y ago

Ben Shneiderman wrote "Beyond Intelligent Machines" published in the Jan 1993 IEEE Software, and the subject of this hn discussion: Beyond Intelligent Machines: Just Do It (1993) (umd.edu) https://news.ycombinator.com/item?id=22742100 https://news.ycombinator.com/item?id=22742100 Mark Weiser and Ben Shneiderman and I worked at the University of Maryland College Park before he went to PARC, and they have a similar take on agents versus augmentation. I think Douglass Engelbart, Maurice Wilkes, and Joseph Weizenbaum would agree with many of their points. https://www.cs.umd.edu/~ben/papers/Shneiderman1993Beyond.pdf https://www.cs.umd.edu/~ben/papers/Shneiderman1993Beyond.pdf Easier to read text version: http://www.cs.umd.edu/hcil/trs/93-03/93-03.html http://www.cs.umd.edu/hcil/trs/93-03/93-03.html Beyond Intelligent Machines, IEEE Software, January 1993 https://news.ycombinator.com/item?id=22744573 https://news.ycombinator.com/item?id=22744573 DonHopkins on April 1, 2020 | parent | context | favorite | on: Beyond Intelligent Machines: Just Do It (1993) I love this article! I worked with Ben Shneiderman at HCIL, and when he sent me a copy of this article to review, I was inspired by the "Dynamic Queries" of his "Dynamic Home Finder" demo, to implement something like that for SimCity (the Frob-O-Matic window). [Video and transcript below.] Here are some of the main points of his article: Don't label machines as "intelligent". It limits the imagination. We should have greater ambitions. Enable humans to accomplish tasks that weren't before possible, instead of trying to enable machines to accomplish tasks people can already do. Predictability and control are desirable qualities. Give users the feeling of mastery, competence, and understanding, sense of accomplishment. The "intelligent machine" label limits or even eliminates human responsibility. If you treat machines like people, you're likely to end up treating people like machines. Ben Shneiderman called his lab HCIL instead of CHIL, to explicitly put Humans before Computers. Here's his description of "Dynamic Queries" and the "Dynamic Home Finder": >Dynamic Queries. These animations let you rapidly adjust query parameters and immediately display updated result sets, which makes them very effective when a visual environment like a map, calendar, or schematic diagram is available. The immediate display of results lets users more easily develop intuitions, discover patterns, spot trends, find exceptions, and see anomalies. >Figure 2 shows a screen from Dynamic HomeFinder, a prototype interface for real-estate agents that uses dynamic queries, written by Christopher Williamson of UM. Users can adjust the cost, number of bedrooms, and location of the A and B markers, among other characteristics, and points of light appear on a map to indicate a home that matches their specifications. Clicking on a point of light brings up a home description or image. >Users of Dynamic HomeFinder can execute up to 100 queries per second (rather than one query per 100 seconds as is typical in a database query language), producing a revealing animated view of where high- or low-price homes are found-and there are no syntax errors. >Our empirical study of 18 users showed Dynamic HomeFinder to be more effective than a natural-language interface using Q&A from Symantec (C. Williamson and B. Shneiderman, The Dynamic HomeFinder: Evaluating Dynamic Queries in a Real-Estate Information Exploration System," Proc. SIG Information Retrieval, ACM Press, 1992, pp. 338-346). https://www.cs.umd.edu/users/ben/papers/Williamson1992dynamic.pdf https://www.cs.umd.edu/users/ben/papers/Williamson1992dynami... That inspired me to implement a version of the "Dynamic Home Finder" in SimCity: Multi Player SimCityNet for X11 on Linux (Dynamic Home Finder / Frob-O-Matic demo at 3:35). https://www.youtube.com/watch?v=_fVl4dGwUrA&t=3m35s https://www.youtube.com/watch?v=_fVl4dGwUrA&t=3m35s Transcript: Here's an interesting thing inspired by Ben Shneiderman. We can look at the ... This is the "Dynamic Zone Filter". So we are going to set this to be dynamic. Now it's going to show all the zones, but it's not going to show the ones that don't pass this filter. So this filter is currently all the way open. Now we're going to change population density. This is a two-ended slider. This is the segment of the population density, from zero to 81. So everything else will disappear. Say I'm looking for a home. I want low population density. But I want high rate of growth. And now, these are the places that have low population density and high rate of growth. And then you can just interactively ... So each of these filters out some of the places. So you can look at ... I don't want high traffic density. I don't want any pollution. I don't want any crime. I want land value to be high. I'm getting pretty picky. So maybe I'll deal with more people. Lower rate of growth. I'm too picky about pollution. I'm too picky about land value. That's it. So basically, Ben Shneiderman demonstrated this as the "Dynamic Home Finder", and I realized that SimCity has all these layers of information that it can draw on, as fictitious as they are, to do that kind of real time, interactive, smooth database query. It's just a much higher bandwidth way to query a database than is conventionally used. Anyway, that was the dynamic zone finder. Also: Here's some email I wrote to Ben about this article, after re-reading the article again in 2009: Date: 28/02/2009 18:51 Subject: Re-reading "Beyond Intelligent Machines" A long time ago, you sent us (me, Brad Myers, Jack Callahan and Mark Weiser) a preview of your IEEE Software article "Beyond Intelligent Machines", which I ran across and have re-read. It's still delightful, inspiring and relevant today. Speech synthesis and recognition has come a long way, to the point where the "talking car" scenario is quite common (TomToms that speak and recognize street names and addresses). But I think the development of user-customizable user interfaces has been dreadfully stalled since HyperCard died. An "Interactive Learning Environment" is a great description of what I'm developing SimCity into. Remember when you visited CMU and I showed you the version of SimCity with the dynamic query feature, an "homage" to your Dynamic Home Finder? You could dial a series of filters on spatial properties like population density, pollution, traffic, land value, police coverage, etc. That code is now open source, and I'm redeveloping it! It's now very easy to write Python code to dynamically query and filter the map and data layers, and visualize with transparent colored tiles and pixel overlays, depending on arbitrary Python functions over the state of the map and its data layers. And it's also possible to script agents in Python! Here's a picture of SimCity with the traffic overlay enabled (showing yellow/orange/red haze over high traffic areas), with some PacMan agents, which are programmed to follow roads, go towards high traffic and eat the cars! Notice the clear road behind each PacMan! http://www.donhopkins.com/home/images/SimCityPacMan.png http://www.donhopkins.com/home/images/SimCityPacMan.png -Don Micropolis Online (SimCity) Web Demo: https://www.youtube.com/watch?v=8snnqQSI0GE https://www.youtube.com/watch?v=8snnqQSI0GE A demo of the open source Micropolis Online game (based on the original SimCity Classic source code from Maxis), running on a web server, written in C++ and Python, and displaying in a web browser, written in OpenLaszlo and JavaScript, running in the Flash player. Developed by Don Hopkins. Source Code: https://github.com/SimHacker/micropolis https://github.com/SimHacker/micropolis HAR 2009 talk: Constructionist Educational Open Source https://medium.com/@donhopkins/har-2009-lightning-talk-transcript-constructionist-educational-open-source-simcity-by-don-3a9e010bf305 https://medium.com/@donhopkins/har-2009-lightning-talk-trans...

Bellary 1y ago

[dead]

careful_ai 1y ago

[dead]