20 ms·
CriticGPT: Finding GPT-4's mistakes with GPT-4
- mkmk 2y agoIt seems more and more that the solution to AI's quality problems is... more AI. Does Anthropic do something like this as well, or is there another reason Claude Sonnet 3.5 is so much better at coding than GPT-4o?
- GaggiX 2y ago>or is there another reason Claude Sonnet 3.5 is so much better at coding than GPT-4o? It's impossible to say because these models are proprietary.
- ru552 2y agoAnthropic has attributed Sonnet 3.5's model improvement to better training data. "Which data specifically? Gerstenhaber wouldn’t disclose, but he implied that Claude 3.5 Sonnet draws much of its strength from these training sets."[0] [0]https://techcrunch.com/2024/06/20/anthropic-claims-its-latest-model-is-best-in-class/ https://techcrunch.com/2024/06/20/anthropic-claims-its-lates...
- swyx 2y agoand water is wet
- jasonjmcghee 2y agoMy guess, which could be completely wrong, Anthropic spent more resources on interpretability and it's paying off. I remember when I first started using activation maps when building image classification models and it was like what on earth was I doing before this... just blindly trusting the loss. How do you discover biases and issues with training data without interpretability?
- Kiro 2y agoIs it really that much better? I'm really happy with GPT-4o's coding capabilities and very seldom experience problems with hallucinations or incorrect responses, so I'm intrigued by how much better it can actually be.
- p1esk 2y agoIn my experience Sonnet 3.5 is about the same as 4o for coding. Sometimes one provides a better solution, sometimes the other. Both are pretty good.
- surfingdino 2y ago> It seems more and more that the solution to AI's quality problems is... more AI. This reminds me of the passage found in the description of the fuckitpy module: "This module is like violence: if it doesn't work, you just need more of it."
- soloist11 2y agoHow do they know the critic did not make a mistake? Do they have a critic for the critic?
- OlleTO 2y agoIt's critics all the way down
- deleted 2y ago[deleted]
- azulster 2y agoit's literally just the oracle problem all over again
- esafak 2y agoIt's called iteration. Humans do the same thing.
- soloist11 2y agoAre you sure it's not called recursion?
- citizen_friend 2y agoIt’s not a human, and we shouldn’t assume it will have traits we do without evidence. Iteration also is when your brain meets the external world and corrects. This is a closed system.
- vidarh 2y agoWe are not assuming that. The iteration happens by taking the report and passing it to another reviewer who reviews the first review. Their comparison is between a human reviewer passing reports to a human reviewer vs. CriticGPT -> human reviewer vs. CriticGPT+human reviewer -> human reviewer.
- 2y ago
- sdenton4 2y agoLooks like the hallucination rate doesn't improve significantly, but I suppose it's still a win if it helps humans review things faster? Though I could imagine reliance on the tool leading to missing less obvious problems.
- tombert 2y agoWhile it's a net good, it would kind of kill one of the most valuable parts of ChatGPT for me, which is critiquing its output myself. If I ask it a question, I try not to trust it immediately, and I independently look the answer up and I argue with it. In turn, it actually is one of my favorite learning tools, because it kind of forces me to figure out why it's wrong and explain it.
- goostavos 2y agoUnexpectedly, I kind of agree. I've found GPT to be a great tutor for things I'm trying to learn. It being somewhat unreliable / prone to confidently lying embeds a certain amount of useful skepticism and questioning of all the information, which in turn leads to an overall better understanding. Fighting with the AI's wrongness out of spike is an unexpectedly good motivator.
- ExtremisAndy 2y agoWow, I’ve never thought about that, but you’re right! It really has trained me to be skeptical of what I’m being taught and confirm the veracity of it with multiple sources. A bit time-consuming, of course, but generally a good way to go about educating yourself!
- tombert 2y agoI genuinely think that arguing with it has been almost a secret weapon for me with my grad school work. I'll ask it a question about temporal logic or something, it'll say something that sounds accurate but is ultimately wrong or misleading after looking through traditional documentation, and I can fight with it, and see if it refines it to something correct, which I can then check again, etc. I keep doing this for a bunch of iterations and I end up with a pretty good understanding of the topic. I guess at some level this is almost what "prompt engineering" is (though I really hate that term), but I use it as a learning tool and I do think it's been really good at helping me cement concepts in my brain.
- GiorgioG 2y agoAll these LLMs make up too much stuff, I don't see how that can be fixed.
- elwell 2y ago> All these LLMs make up too much stuff, I don't see how that can be fixed. All these humans make up too much stuff, I don't see how that can be fixed.
- testfrequency 2y agoI know you’re trying to be edgy here, but if I was deciding between searching online and finding a source vs trying to shortcut and use GPT, but GPT decides to hallucinate and make something up - that’s the deceiving part. The biggest issue is how confidently wrong GPT enjoys being. You can press GPT in either right or wrong direction and it will concede with minimal effort, which is also an issue. It’s just really bad russian roulette nerdspining until someone gets tired.
- sva_ 2y agoI wouldn't call it deceiving. In order to be motivated to deceive someone, you'd need agency and some benefit out of it
- testfrequency 2y agoIsn’t that GPT Plus? Trick you into thinking you have found your new friend and they understand everything? Surely OpenAI would like people to use their GPT over a Google search. How do you think leadership at OpenAI would respond to that?
- advael 2y ago1. Deception describes a result, not a motivation. If someone has been led to believe something that isn't true, they have been deceived, and this doesn't require any other agents 2. While I agree that it's a stretch to call ChatGPT agentic, it's nonetheless "motivated" in the sense that it's learned based on an objective function, which we can model as a causal factor behind its behavior, which might improve our understanding of that behavior. I think it's relatively intuitive and not deeply incorrect to say that that a learned objective of generating plausible prose can be a causal factor which has led to a tendency to generate prose which often deceives people, and I see little value in getting nitpicky about agentic assumptions in colloquial language when a vast swath of the lexicon and grammar of human languages writ large does so essentially by default. "The rain got me wet!" doesn't assume that the rain has agency
- jmount 2y agoSince the whole thing is behind an API- exposing the works adds little value. If the corrections worked at an acceptable rate, one would just want them applied at the source.
- renewiltord 2y ago> If the corrections worked at an acceptable rate, one would just want them applied at the source. What do you mean? The model is for improving their RLHF trainers performance. RLHF does get applied "at the source" so to speak. It's a modification on the model behind the API. Perhaps if you were to say what you think this thing is for and then share why you think it's not "at the source".
- Panoramix 2y agoNot OP but the screenshot in the article pretty much shows something that it's not at the source. You'd like to get the "correct" answer straight away, not watch a discussion between two bots.
- IanCal 2y agoYes, but this is about helping the people who are training the model.
- ertgbnm 2y agoYou are missing the point of the model in the first place. By having higher quality RLHF datasets, you get a higher quality final model. CriticGPT is not a product, but a tool to make GPT-4 and future models better.
- uththtjf 2y ago[dead]
- teaearlgraycold 2y agoSounds like a combination of a GAN[1] and RLHF. Not surprising that this works. [1] - https://en.wikipedia.org/wiki/Generative_adversarial_network https://en.wikipedia.org/wiki/Generative_adversarial_network
- megaman821 2y agoI wonder if you could apply this to training data. Like here is an example of a common mistake and why that mistake could be made, or here is a statement made in jest and why it could be found funny.
- wcoenen 2y agoThis is about RLHF training. But I've wondered if something similar could be used to automatically judge the quality of the data that is used in pre-training, and then spend more compute on the good stuff. Or throw out really bad stuff even before building the tokenizer, to avoid those "glitch token" problems. Etc.
- smsx 2y agoYup, check out Rho-1 by microsoft research.
- integral_1699 2y agoI've been using this approach myself, albeit manually, with ChatGPT. I first ask my question, then open a new chat and ask it to find flaws with the previous answer. Quite often, it does improve the end result.
- victor9000 2y agoThis gets at the absolute torrent of LLM diarrhea that people are adding to PRs these days. The worst of it seems to come from junior and first time senior devs who think more is more when it comes to LoC. PR review has become a nightmare at my work where juniors are now producing these magnificent PRs with dynamic programming, esoteric caching, database triggers, you name it. People are using LLMs to produce code far beyond their abilities, wisdom, or understanding, producing an absolute clusterfuck of bugs and edge cases. Anyone else dealing with something similar? How are you handling it?
- Zee2 2y agoMy company simply prohibits any AI generated code. Seems to work rather well.
- ffsm8 2y agoMy employer went all in and pays for both enterprise subscriptions (github Copilot+ chatgpt enterprise, which is just a company branded version of the regular interface) We've even been getting "prompt engineering" meeting invites of 3+ hours to get an introduction into their usage. 100-150 participants each time I joined It's amazing how much they're valuing it. from my experience it's usually a negative productivity multiplier (x0.7 vs x1 without either)
- kridsdale3 2y agoSounds like they plan to lay half of you off once metrics stabilize.
- deleted 2y ago[deleted]
- gnicholas 2y agoHow is this enforced? I'm not saying it isn't a good idea, just that it seems like it would be tricky to enforce. Separately, it could result in employees uploading code to a non-privacy-respecting AI, whereas if employees were allowed to use a particular AI then the company could better control privacy/security concerns.
- rodoxcasta 2y ago> Additionally, when people use CriticGPT, the AI augments their skills, resulting in more comprehensive critiques than when people work alone, and fewer hallucinated bugs than when the model works alone. But, as per the first graphic, CriticGPT alone has better comprehensiveness than CriticGPT+Human? Is that right?
- neom 2y agoI was curious about the authors, did some digging, they've published some cool stuff: Improving alignment of dialogue agents via targeted human judgements - https://arxiv.org/abs/2209.14375 https://arxiv.org/abs/2209.14375 Teaching language models to support answers with verified quotes - https://arxiv.org/abs/2203.11147 https://arxiv.org/abs/2203.11147
- VWWHFSfQ 2y agoIt's an interesting dichotomy happening in the EU vs. USA in terms of how these kinds of phenomena are discovered, presented, analyzed, and approached. The EU seems to be very much toward a regulate early, safety-first approach. Where USA is very much toward unregulated, move fast, break things, assess the damage, regulate later. I don't know which is better or worse.
- ipaddr 2y agoRegulate before you understand the problem seems like a poor approach
- dns_snek 2y agoWhich parts of the EU AI act do you disagree with? It primarily focuses on areas of use that present high or unacceptable risks.
- l5870uoo9y 2y agoAs a European, after decades of regulations and fines without much to show, nobody in the industry believes the EU the capable of creating tech ecosystem. Perhaps even that the EU is part of the problem and that individual countries could independently move much faster.
- Twixes 2y agoThe difference in tech outcomes can definitely be attributed to _European_ conditions, but regulation is extremely country-specific – the _EU_ is not the detriment (e.g. Germany might be notably bureaucratic, but the culture in Sweden differs, and so do frameworks in Poland). EU institutions are out to get the US giants, but startups or scaleups? Out of sight. There really is comparatively little reward for shooting for the moon though – the fragmented stock markets don't provide great exit opportunities, so less money goes into funding ambitious companies. Then, scaling throughout all of Europe is notably hard, with dozens of languages, cultures, and legal frameworks to navigate. Some of these cultures are more risk-averse, and that's not easy to change. Not to mention English being the _lingua franca_ of business and tech. I would love Europe to reach the States' level of tech strength, but these are all really hard problems.
- jimmytucson 2y agoWhat's the difference between CriticGPT and ChatGPT with a prompt that says "You are a software engineer, your job is to review this code and point out bugs, here is what the code is supposed to do: {the original prompt}, here is the code {original response}, review the code," etc.
- ipaddr 2y ago$20 a month
- advael 2y agoThis is a really bizarre thing to do honestly It's plausible that there are potential avenues for improving language models through adversarial learning. GANs and Actor-Critic models have done a good job in narrow-domain generative applications and task learning, and I can make a strong theoretical argument that you can do something that looks like priority learning via adversarial equilibria But why in the world are you trying to present this as a human-in-the-loop system? This makes no sense to me. You take an error-prone generative language model and then present another instance of an error-prone generative language model to "critique" it for the benefit of... a human observer? The very best case here is that this wastes a bunch of heat and time for what can only be a pretty nebulous potential gain to the human's understanding Is this some weird gambit to get people to trust these models more? Is it OpenAI losing the plot completely because they're unwilling to go back to open-sourcing their models but addicted to the publicity of releasing public-facing interfaces to them? This doesn't make sense to me as a research angle or as a product I can really see the Microsoft influence here
- kenjackson 2y agoIt's for their RLHF pipeline to improve labeling. Honestly, this seems super reasonable to me. I don't get why you think this is such a bad idea for this purpose...
- advael 2y agoRLHF to me seems more as a PR play than anything else, but inasmuch as it does anything useful, adding a second LLM to influence the human that's influencing the LLM doesn't solve any of the fundamental problems of either system. If anything it muddies the waters more, because we have already seen that humans are probably too credulous of the information presented to them by these models. If you want adversarial learning, there are far more efficient ways to do it. If you want human auditing, the best case here is that the second LLM doesn't influence the human's decisions at all (because any influence reduces the degree to which this is independent feedback)
- vhiremath4 2y ago
- rvz 2y agoAnd both can still be wrong as they have no understanding of the mistake.
- lowyek 2y agoI find it fascinating that while in other fields you see lot of theorums/results much before practical results are found. But in this forefront of innovation - I have hardly seen any paper discussing hallucinations and lowerbound/upperbound on that. Or may be I didn't open hacker news on that right day when it was published. Would love to understand the hallucination phenomena more deeply and the mathematics behind it.
- amelius 2y agoI don't see many deep theorems in the field of psychology either.
- deleted 2y ago[deleted]
- dennisy 2y agoNot sure if there is a great deal of maths to understand. The output of an LLM is stochastic by nature, and will read syntactical perfect, AKA a hallucination. No real way to mathematically prove this, considering there is also no way to know if the training data also had this “hallucination” inside of it.
- ben_w 2y agoI think mathematical proof is the wrong framework, in the same way that chemistry is the wrong framework for precisely quantifying and explaining how LSD causes humans to hallucinate (you can point to which receptors it binds with, but AFAICT not much more than that). Investigate it with the tools of psychologically, as suited for use on a new non-human creature we've never encountered before.
- deleted 2y ago[deleted]
- beernet 2y agoHow are 'hallucinations' a phenomenon? I have trouble with the term 'hallucination' and believe it sets the wrong narratuve. It suggests something negative or unexpected, which it absolutely is not. Language models aim at, as their name implies, modeling language. Not facts or anything alike. This is per design and you certainly don't have to be an AI researcher to grasp that. That being said, people new to the field tend to believe that these models are fact machines. In fact, they are the complete opposite.
- bluelightning2k 2y agoEvaluators with CriticGPT outperform those without 60% of the time. So, slightly better than random chance. I guess a win is a win but I would have thought this would be higher. I'd kind have assume that just asking GPT itself if it's sure would be this kind of lift.
- torginus 2y agoNot sure what this means, but if someone asked me to critique iOS code for example, I wouldn't be much of a help since I don't know the first thing about it , other than some generic best practices. I'm sure ChatGPT would outperform me, and I could only aid it in very limited ways. That doesn't mean an expert iOS programmer wouldn't run circles around it.
- pama 2y agoI’m not sure why 60 vs 40 is slightly better than random chance. A person using this system has a 50% higher success rate than those not using it. I wouldnt call this a slight better result.
- smrq 2y agoThat's just... obviously incorrect. It's 10% higher. (60%, as opposed to the expected 50% from random chance.)
- pama 2y agoI’m not sure what you mean. You can see the plots if you prefer, or think of it this way: out of a total of 100 trials, one team gets 40 and the other gets 60 = 40 + 40 * 50% If you want to think of a 75% win rate as a more extreme example: you could say 25% above random or you could say one team wins 3 times as many cases as the other. Both are equivalent but I think that the second way conveys the strength of the difference much better. The results in this work are statistically significant and substantial.
- croes 2y agoNext step Critic²GPT, a model based on GPT-4, writes critiques of CriticGPT responses.
- upwardbound 2y agoFor those new to the field of AGI safety: this is an implementation of Paul Christiano's alignment procedure proposal called Iterated Amplification from 6 years ago. https://www.alignmentforum.org/s/EmDuGeRw749sD3GKd https://www.alignmentforum.org/s/EmDuGeRw749sD3GKd According to his website he previously ran the language model alignment team at OpenAI. https://paulfchristiano.com/ https://paulfchristiano.com/ It's wonderful to see his idea coming to fruition! I'm honestly a bit skeptical of the idea myself (it's like proposing to stabilize the stack of "turtles all the way down" by adding more turtles - as is insightfully pointed out in this other comment https://news.ycombinator.com/item?id=40817017 https://news.ycombinator.com/item?id=40817017) but every innovative idea is worth a try, in a field as time-critical and urgent as AGI safety. For a good summary of technical approaches to AGI safety, start with the Future of Life Institute AI Alignment Podcast, especially these two episodes which serve as an overview of the field: - https://futureoflife.org/podcast/an-overview-of-technical-ai-alignment-with-rohin-shah-part-1/ https://futureoflife.org/podcast/an-overview-of-technical-ai... - https://futureoflife.org/podcast/an-overview-of-technical-ai-alignment-with-rohin-shah-part-2/ https://futureoflife.org/podcast/an-overview-of-technical-ai... In both of those episodes, Cristiano's publication series on Iterated Amplification is link #3 in the list of recommended reading.
- pama 2y agoI am not sure why you didnt see the citations to 3 different papers from Cristiano. A simple search in the linked PDF suffices: citations 12, 19, and 31.
- upwardbound 2y agoThank you for providing the reference numbers, this is helpful! I'll update my GP comment.
- TeMPOraL 2y ago> it's like proposing to stabilize the stack of "turtles all the way down" by adding more turtles That's completely fine. Say each layer uses the same amount of turtles, but half the size of the layer above. Even allowing for arbitrarily small turtles, the total height of the infinite stack will be just 2x the height of the first layer. Point being, some series converge to a finite result, including some defined recursively. And in practice, we can usually cut the recursion after first couple steps, as the infinitely long remainder has negligible impact on the final result.
- deleted 2y ago[deleted]
- rob74 2y agoYou build an LLM that makes mistakes. Then you build another LLM to find the mistakes the first LLM makes, and, surprise: > CriticGPT’s suggestions are not always correct Now you have two problems...
- tim333 2y agoBit like human fact checking really.