6 ms·
Another cherry-picked example of an LLM doing something amazing, written about with a heavy dose of anthropomorphism. It's easy to get LLMs to do seemingly ama
by afro88 2y ago
Another cherry-picked example of an LLM doing something amazing, written about with a heavy dose of anthropomorphism.
It's easy to get LLMs to do seemingly amazing things. It's incredibly hard to build something where it does this amazing thing consistently and accurately for all reasonable inputs.
> Analyzing authentication system files:
> - Core token validation logic
> - Session management
> - Related middleware
This hard coded string is doing some very heavy lifting. This isn't anything special until this string is also generated accurately and consistently for any reasonable PR.
OP if you are reading, the first thing you should do is get a variety of codebases with a variety of real world PRs and set up some evals. This isn't special until evals show it producing consistent results.
- namanyayg 2y agoThat's exactly what I want to do next. Any tips on how should I get codebases and real world PRs? Are the ones on popular open source repos on GitHub sufficient? I worry that they don't really accurately reflect real world closed source experience because of the inherent selection bias. Secondly, after getting all this, how do I evaluate which method gave better results? Should it be done by a human, or should I just plug an LLM to check?
- deleted 2y ago[deleted]
- jarebear6expepj 2y ago“…should it be done by a human?” Sigh.
- namanyayg 2y agoI'll do it personally in the beginning but was thinking about doing it on scale
- deleted 2y ago[deleted]
- JTyQZSnP3cQGa8B 2y ago> doing it on scale Like cloud-scale, no-code scale, or NoSQL scale? You are confused, which shows that, maybe, you should not be using such tools with the experience that you don't have.
- zBard 2y ago‘Like cloud-scale, no-code scale or NoSQL scale’. That is the dumbest statement I have heard this week. You should perhaps refrain from commenting, at-least until you gain the modicum of intelligence that you currently don’t have.
- smusamashah 2y agoVery first thing you can tell us (or try if you haven't) is that if you re-prompt, does it give the same answer? Second can you get it to generate (consistently and repeatedly) the text that gp pointed out? Don't need to switch to a different repo for quick test, just make it reproable on your current repo.
- namanyayg 2y agoNot the exact text, but still decent quality. I'll play around with temperature and prompts a bit.
- QuadmasterXLII 2y agoIf you could get an LLM to check, you could just spam solutions with any assortment of models and then use your checker to pick the best.
- Yoric 2y agoExcept LLMs are usually really bad at checking. Perhaps if they're some day augmented by formal methods, that might change.
- QuadmasterXLII 2y agoSorry, I was unclear- my statement was supposed to be proof that LLMs are bad at checking, because the approach I listed is well known to not work
- Yoric 2y agoGot it, now it makes sense :) I may accidentally have been inspired from your message when I wrote the following piece, yesterday: https://yoric.github.io/post/formal-ai/ https://yoric.github.io/post/formal-ai/
- throwup238 2y ago> I worry that they don't really accurately reflect real world closed source experience because of the inherent selection bias. As opposed to what, yet another beginner React app? That’s what everyone seems to be testing with but none of the projects I’ve seen are reflective of a production codebase that’s years old and has been touched by a dozen developers. Throw it at a complicated non-frontend mixed language repo like cxx-qt [1] or something, preferably where the training data doesn’t include the latest API. [1] https://github.com/KDAB/cxx-qt https://github.com/KDAB/cxx-qt
- lukan 2y ago"preferably where the training data doesn’t include the latest API" That is the reason LLM's in their current shape are pretty useless to me for most tasks. They happily mix different versions of popular frameworks, so I have to do so much manual work to fix it, I rather do all by myself then. Pure (common) math problems, or other domains where the tech did not change so much, like bash scripts or regex are where I can use them. But my actual code? Not really. The LLM would need to be trained only on the API version I use and that is not a thing yet, as far as I am aware.
- owenpalmer 2y agoHave you tried RAG on the docs?
- lukan 2y agoNot yet, I can imagine it can improve things, but was too sceptical that it is worth the effort. Have you had good succes with it?
- JTyQZSnP3cQGa8B 2y ago[flagged]
- tyrust 2y agoThey're asking about viewing existing PRs, not creating new ones. No need to be so hostile.
- fuzzythinker 2y agohttps://www.kaggle.com/competitions/konwinski-prize https://www.kaggle.com/competitions/konwinski-prize
- rHAdG12327 2y agoGiven how eager Microsoft is to steal other people's code, perhaps the leaked Windows source code would be an option. Or perhaps Microsoft will let you train on their internal issue tracker.
- InkCanon 2y agoI also think there's some exaggeration. Annotating files with a feature tag system is both manual and not scabale. Custom prompting for each commit or feature a lot more so. You do a decent bit of specialized work here. And I think he left out the most important part, was the answer actually right? The real value of any good dev at all is that he can provide reasonably accurate analysis with logic and examples. "Could have an error" is more like a compiler warning than the output of a good engineer. Side note: "broke the benchmark script?" If you have an automated way to qualitatively evaluate the output of an LLM in a reasonably broad context like code reading, that's far bigger a story.
- ninetyninenine 2y agoThis post talks as if the results are a worthless pile of trash while obeying the HN rules of not directly insulting the results. I agree with everything under the first paragraph. Let me spell it out for you. These results. Are. Not. Worthless. Certainly what you said is correct on what he “should” do to get additional data, but your tonality of implying that the results are utter trash and falsely anthropomorphizing something is wrong. Why is it wrong? Imagine Einstein got most things wrong in his life. Most things but he did discover special and general relativity. It’s just everything else was wrong. Relativity is still worth something. The results are still worthwhile. We have an example of an LLM hallucinating. Then we have another example of additional contextual data causing the LLM to stop hallucinating. This is a data point leaving a clue about hallucinations and stopping hallucinations. It’s imperfect but a valuable clue. My guess is that there’s a million causal factors that cause an LLM to hallucinate and he’s found one. If he does what he did a multitude of times for different topics and different problems where contextual data stops an hallucination, with enough data and categorization of said data we may be able to output statistical data and have insight into what’s going on from a statistical perspective. This is just like how we analyze other things that produce fuzzy data like humans. Oh no! Am I anthropomorphizing again?? Does that action make everything I said wrong? No, it doesn’t. Humans produce correct data when given context. It is reasonable to assume in many cases LLMs will do the same. I wrote this post because I agree with everything you said but not your tone which implies that what OP did is utterly trivial.
- godelski 2y agoThey didn't say worthless, they said amazing. Their comment is "do it consistently, then I'll buy your explanation"
- ninetyninenine 2y agolol “seemingly amazing” means not amazing at all. He didn’t literally say it but the comment implies it is worthless as does yours. Humans dont “buy it” when they think something is worthless. The tonality is bent this way. He could have said, “this is amazingly useful data but we need more” but of course it doesn’t read like this at all thanks to the first paragraph. Let’s not hallucinate it into something it’s not with wordplay. The comment is highly negative.
- nuancebydefault 2y agoStill, the findings in the article are very valuable. The fact that directing the "thought" process of the LLM by this kind of prompting, yields much better results, is useful. The comparison to how a senior dev would approach the assignment, as a metaphor explaining the mechanism, makes perfect sense to me.
- mensetmanusman 2y agoNo need for the pessimism, these are new tools that humans have invented. We are groking how to utilize them.
- ramblerman 2y agoOP brought a rational argument, you didn't. It sounds like you are defending your optimism with emotion. > We are groking how to utilize them. Indeed.
- mensetmanusman 2y agoIt’s tautologically the case that these are new tools under constant improvement. The fact that these tools have extremely weird and new to the world interfacial quirks is what the discussion is about…
- asah 2y agoIt's incredibly hard to get __ HUMANS __ to do this amazing thing consistently and accurately for all reasonable inputs.
- iaseiadit 2y agoSome humans can do it consistently, other humans can't. Versus how no publicly-available AI can do it consistently (yet). Although it seems like a matter of time at this point, and then work as we know it changes dramatically.
- overgard 2y agoTo be fair, most senior developers don't have any incentive to put this amount of analysis into a working codebase. When the system is working, nobody really wants to spend time they could be working on something interesting trying to find bugs in old code. Plus there's the social consideration that your colleagues might not like you a lot if you spend all your time picking their (working) code apart while not doing any of your tasks. Usually this kind of analysis would come from someone specifically brought in to find issues, like an auditor or a pen-tester.
- kotlip 2y agoThe right incentives would motivate bug hunting, it entirely depends on company management. Most competent senior devs I’ve worked with spend a great deal of time carefully reading through PRs that involve critical changes. In either case, the question is not whether humans tend to act a certain way, but whether they are capable of skillfully performing a task.
- smrtinsert 2y agoUnfortunately some senior devs like myself do care. Too bad no one else does. Code reviews become quick after a while, you brain adapts to being to review code deeply and quickly
- talldayo 2y agoHumans are fully capable of protracted, triple-checked scrutiny if the incentives align just right. Given the same conditions, you cannot ever compel an AI to stop being wrong or consistently communicate what it doesn't understand.
- ryaniscool 2y agoWhat I want to know is how accurate was the comment? I've found AI to frequently suggest plausible changes. Like they use enough info and context to look like excellent suggestions on the surface but you realize with some digging it was so completely wrong.
- zbyforgotp 2y agoThis is exactly the behaviour that I would expect from a human teleported into a project and given a similar task. After some time humans would gather some background info needed to be more productive and we need to find out how to copy that.
- ryaniscool 2y agoYou're saying a human would also offer bullsh*t suggestions that seem right but are in fact wrong? Humans who make lots of mistakes with confidence that they aren't mistakes usually get fired or steered into a position where they can do the least amount of damage. It's not that AI needs more background info for this type of of thing. It needs the ability to iteratively check it's own work and make corrections. This is what humans do better.
- zbyforgotp 2y agoOk, maybe that was exaggerated - there are differences, but they are shrinking.
- Der_Einzige 2y agoThe people who claim it's that hard to do these things have never heard of or used constrained/structured generation, and it shows big time! Most other related issues of models these days are due to the tokenizer or poor choice of sampler settings which is a cheap shot on models.
- j45 2y agoWhat i'm learning is just because something might be hard to you or I, doesn't mean it's not possible or not working. LLMs can generlaly only do what they have data on, either in training, or instructions via prompting it seems. Keeping instructions reliable, is increasing and testing, appears to benefit from LLMops tools like Agenta, etc. It seems to me like LLMs are reasonably well suited for things that code can't do easily as well. You can find models on Hugging face that are great at categorizing and applying labels and categorization, instead of trying to get a generalized assistant model to do it. I'm more and more looking at tools like OpenRouter to allow doing each step with the model that does it best, almost functionally where needed to increase stability. For now, it seems to be one way to improve reliability dramatically, happy to learn about what others are finding too. It seems like a pretty nascent area still where existing tooling in other areas of tech is still figuring itself out in the LLM space.