5 ms·
I'm very curious if a toggle would be useful that would display a heatmap of a source file showing how surprising each token is to the model. Red tokens are mor
by furyofantares 1y ago
I'm very curious if a toggle would be useful that would display a heatmap of a source file showing how surprising each token is to the model. Red tokens are more likely to be errors, bad names, or wrong comments.
- WithinReason 1y agopreviously undefined variable and function names would be red as well
- Too 1y agoAll editors do this already.
- WithinReason 1y agoNo, I mean when an LLM encounters a previously unseen name it doesn't expect it so it would be red, even though it's perfectly valid.
- jakelazaroff 1y agoI don't think that's necessarily true. I've definitely seen LLMs hallucinating variables they never defined.
- furyofantares 1y agoI'm imagining everything the LLM could produce (with a given top_k setting) would be shades of green to yellow. Just outside of that orange, and far outside red. LLMs generate new functions all the time, I'd guess these would be light green, maybe the first token in the name would be yellow and it would get brighter green as the name unfolds. The logits are probably all small when in the global scope where it's not clear what will be defined next. I'm not imagining mapping logits directly to heat, the ordering of tokens seems much more appropriate.
- fwip 1y agoIt would depend on how surprising the name is, right? The declaration of `int total` should be relatively unsurprising at the top of a function named `sum`, but probably more surprising in `reverseString`.
- digdugdirk 1y agoInteresting! I've often felt that we aren't fully utilizing the "low hanging fruit" from the early days of the LLM craze. This seems like one of those ideas.
- jama211 1y agoThat’s actually fantastic as an idea
- dclowd9901 1y agoThat's a really cool idea. Also the inverse, where suggestions from the AI were similarly heat mapped for confidence would be extremely useful.
- ijk 1y agoI want that in an editor. It's also a good way to check if your writing is too predictable or cliche. The perplexity calculation isn't difficult; just need to incorporate it into the editor interface.
- newswasboring 1y agoCan you elaborate on how would one do this calculation?
- irthomasthomas 1y agoimport openai, math, os, textwrap, json, sys query = 'Paris is the capital of' # short demo input os.environ['OPENAI_API_KEY'] # check key early client = openai.OpenAI() resp = client.chat.completions.create( model='gpt-3.5-turbo', messages=[{'role': 'user', 'content': query}], max_tokens=12, logprobs=True, top_logprobs=1 ) logprobs = [t.logprob for t in resp.choices[0].logprobs.content] perplexity = math.exp(-sum(logprobs) / len(logprobs)) print('Prompt: "', query, '"', sep='') print('\nCompletion:', resp.choices[0].message.content) print('\nToken count:', len(logprobs)) print('Perplexity:', round(perplexity, 2)) Output: Prompt: "Paris is the capital of" Completion: France. Token count: 2 Perplexity: 1.17 Meta: Out of three models: k2, qwen3-coder and opus4, only opus one-shot the correct formatting for this comment.
- cleverwebble 1y agoIf you want to generate a heatmap of existing text, you will have to take a different approach here. The naive solution I could come up with would be really expensive with openai, but if you have an open source model, you can write up custom inference that goes one-token-at-a-time through the text, and on each token you look up the difference in logprobs between the token that the LLM predicted vs what was actually there, and use that to color the token. The downside I imagine to this approach is it would probably tend to highlight the beginning of bad code, and not the entire block - because once you commit to a mistake, the model will generally roll with it - ie, a 'hallucination' - so logprobs of tokens after the bug happened might only be slightly higher than normal. Another option might be to use a diffusion based model, adding some noise to the input and having it iterate a few times through, then measuring the parts of the text that changed the most. I have only a light theory understanding of these models though, so I'm not sure how well that would work
- nextaccountic 1y agoEven if something is surprising just because it's a novel algorithm, it warrants better documentation - but commenting the code explaining how it works will make the code itself less surprising! In short, it's probably possible (and it's maybe a good engineering practice) to structure the source such as no specific part is really surprising It reminds me how LLMs finally made people to care about having good documentation - if not for other people, for the AIs to read and understand the system
- Kichererbsen 1y agoI often find myself leaving review comments on pull requests where I was surprised. I'll state as much: This surprised me - I was expecting XYZ at this point. Or I wasn't expecting X to be in charge of Y.
- _kb 1y agoWTFs/minute is a good metric for code quality. Now your pair expressing that can be an LLM. https://blog.codinghorror.com/whos-your-coding-buddy/ https://blog.codinghorror.com/whos-your-coding-buddy/
- federiconafria 1y agoI like to say that the reviewer is always right in that sense, if something is surprising, confusing, unexpected. Since I've been looking at the code for hours, I don't have a valid perspective anymore.
- deleted 1y ago[deleted]
- philipwhiuk 1y ago> It reminds me how LLMs finally made people to care about having good documentation - if not for other people, for the AIs to read and understand the system Honestly I've mostly seen the opposite - impenetrable code translated to English by AI
- 1y ago
- teoremma 1y agoWe explored this exact idea in our recent paper https://arxiv.org/abs/2505.22906 https://arxiv.org/abs/2505.22906 Turns out this kind of UI is not only useful to spot bugs, but also allows users to discover implementation choices and design decisions that are obscured by traditional assistant interfaces. Very exciting research direction!
- reneherse 1y agoVery exciting indeed. I will definitely do a deep dive into this paper, as my current work is exploring layers of affordances such as these in workflows beyond coding.
- zacmps 1y agoI've wanted someone to write an extension utilising this idea since GPT-3 came out. Is it available to use anywhere?
- smolder 1y agoNope, not surprising. Parent changed their text but they are just as wrong.
- smolder 1y ago[flagged]
- vidarh 1y agoYou're getting downvoted - and flagged - because you're repeatedly breaking the HN guidelines. You may want to consider whether that's a path you want to continue.
- smolder 1y agoNot sure you understand what has been going on but thanks for your concern.
- vidarh 1y agoWhat I understand is that if this has already gotten two of your comments killed, and will eventually get your account banned if you keep it up. I'm only bothering with this at all because I see from your profile that you make reasonable comments quite regularly.
- smolder 1y agoConsider that I'm being reasonable.
- sokoloff 1y agoIt’s frequently possible to disagree while still adding to the discussion. The comments of yours that I see downvoted are falling well short of that mark, particularly the ones where you accuse a decade-plus-old account with a recent comment history of being quite skeptical or even anti-LLM for coding of being a shill for an AI company. Consider that your words are not being experienced as reasonable by readers.
- smolder 1y ago[flagged]
- smolder 1y ago[flagged]
- atoav 1y agoYou're adding nothing of substance. If you have a point about the subject itself make it and present the receipts, then the rest of us can decide if we can follow your observation. Without even knowing what the lines of the supposed conflict ought to be about: All I see here are baseless accusations from your side that make you quite frankly look a little bit unhinged. Please discuss your issues based on the merit of ideas not based on accusations and persons.
- smolder 1y ago[flagged]
- furyofantares 1y agoI read all your dead replies and it's a little wild that you think I'm an OpenAI employee (I'm not) trying to do damage control (is the article damaging to OpenAI?) by hijacking the comments (is my comment not a HUD-like idea?). I don't really know where that's coming from, I'm just a dude who connected the idea in the article to an old idea that I haven't seen tried yet. The only thing I truly don't appreciate is you made one comment saying the text of my post had changed. It didn't.
- willtemperley 1y agoThe trouble with anonymous downvoting is that it fuels this kind of paranoia.
- smolder 1y ago[flagged]
- rambambram 1y agoCould you elaborate, please?
- smolder 1y ago[flagged]
- geon 1y agoSounds great. I'd like to see more contextually meaningful refactoring tools. Like "Remove this dependency" or "Externalize this code with a callback". And refactoring shouldn't be done by generatively rewriting the code, but as a series of guaranteed equivalent transformations of the AST, each of which should be committed separately. The AI should be used to analyse the value of the transformation and filter out asinine suggestions, not to write code in itself.
- 8n4vidtmkvmk 1y agoIf it can't guarantee exact equivalence, it should at least rerun the relevant tests in the background and make sure they still pass before making suggestions.
- geon 1y agoIt would be awesome if an ai could speculatively refactor sections of code to an exact equivalent where inconsistency stands out more. That way you could find bugs and make actual features more understandable.
- GuB-42 1y agoThis! That's what I wanted since LLMs learned how to code. And in fact, I think I saw a paper / blog post that showed exactly this, and then... nothing. For the last few years, the tech world became crazy with code generation, with forks of VSCode hooked to LLMs worth billions of dollars and all that. But AI-based code analysis is remarkably poor. The only thing I have seen resembling this is bug report generators, which is I believe is one of the worst approach. The idea you have, that I also had and I am sure many thousands of other people had seem so obvious, why is no one talking about it? Is there something wrong with it? The thing is, using such a feature requires a brain between the keyboard and the chair. A "surprising" token can mean many things: a bug, but also a unique feature, anyways, something you should pay attention to. Too much "green" should also be seen as a signal. Maybe you reinvented the wheel and you should use a library instead, or maybe you failed to take into account a use case specific to your application. Maybe such tools don't make good marketing. You need to be a competent programmer to use them. It won't help you write more lines faster. It doesn't fit the fantasy of making anyone into a programmer with no effort (hint: learning a programming language is not the hard part). It doesn't generate the busywork of AI 1 introducing bugs for AI 2 to create tickets for.
- furyofantares 1y ago> The idea you have, that I also had and I am sure many thousands of other people had seem so obvious, why is no one talking about it? Is there something wrong with it? I expect it definitely requires some iteration, I don't think you can just map logits to heat, you get a lot of noise that way.
- marcosdumay 1y agoJust to point... > Is there something wrong with it? > Maybe such tools don't make good marketing. You had the answer the entire time :) Features that require a brain between the AI and key-presses just don't sell. Don't expect to see them for sale. (But we can still get them for free.)
- brookst 1y agoI don’t think I understand your point. Are you saying that people of a certain competence level lose interest in force-multiplying tools? I don’t think you can be saying that because there’s so much contrary evidence. So what are you saying?
- tionis 1y agoThat's actually something I implemented for a university project a few weeks ago. My professor also did some research into how this can be used for more advanced UIs. I'm sure it's a very common idea.
- digdugdirk 1y agoDo you have a link to the code? I'm curious how you implemented it. I'd also be really intrigued to see that research - does your professor have any published papers or something for those UIs?
- layer8 1y agoYou know what happens when a measure becomes a target, though.
- nojs 1y agoThere is some argument to be made here about entropy, compression, and how if there are no surprises, the program communicates no new information. Interestingly, frequency of "surprising" sentences is one of the ways quality of AI novels is judged: https://arxiv.org/abs/2411.02316 https://arxiv.org/abs/2411.02316