13 ms·

GPT-4 and professional benchmarks: the wrong answer to the wrong question

340 points by agomez314 4y ago

ChancyChance 4y ago
"10/10 pre-2021, 0/10 post-2021" I guffawed. Hasn't there been ANY update to the Turing test since it was proposed? I suspect the answer to the benchmarking issue was address by philosophers long before we got here.
- sebzim4500 4y ago
  I think people completely lost interest in the Turing test as soon as it looked like something might actually be able to pass it.
  - schrodinger 4y ago
    The Turing test has now become "anything ChatGPT cannot solve yet" :)
- enono 4y ago
  Why is your username green?!
  - schrodinger 4y ago
    New account
    - YeGoblynQueenne 4y ago
      Yes, but why green? In particular?
      - wrycoder 4y ago
        Greenhorn. Look it up.
        YeGoblynQueenne 4y ago
        Who says that's the association? Citation please.
        wrycoder 4y ago
        LOL you’d have to ask PG, probably. But, you asked for a plausible reason behind an arbitrary choice. What color would you choose?
        YeGoblynQueenne 4y ago
        Green.
        wrycoder 4y ago
        +1 !
thwayunion 4y ago
Absolutely correct. We already know this is about self-driving cars. Passing a driver's test was already possible in 2015 or so, but SDCs clearly aren't ready for L5 deployment even today. There are also a lot of excellent examples of failure modes in object detection benchmarks. Tests, such as driver's tests or standardized exams, are designed for humans. They make a lot of entirely implicit assumptions about failure modes and gaps in knowledge that are uniquely human. Automated systems work differently. They don't fail in the same way that humans fail, and therefore need different benchmarks. Designing good benchmarks that probe GPT systems for common failure modes and weaknesses is actually quite difficult. Much more difficult than designing or training these systems, IME.
- zer00eyz 4y ago
  > good benchmarks ... failure modes and weaknesses is actually quite difficult. Much more difficult than designing or training these systems Is it? Based on the restrictions placed on the systems we see today and the way people are breaking it, I would say that some failure modes are known.
  - thwayunion 4y ago
    A good benchmark is not simply a set of unit tests. What you want in a benchmark is a set of things you can use to measure general improvement; doing better should decrease the propensity of a particular failure mode. Doing this in a way that generalizes beyond specific sub-problems, or even specific inputs in the benchmark suite, is difficult. Building a benchmark suite that's large and comprehensive enough that generalization isn't necessary is also a challenge. Think about an analogy to software security. Exploiting a SQL injection attack in insecure code is easy. Coming up with a set of unit tests that ensures an entire black box software system is free of SQL injection attacks is quite a bit more difficult. Red teaming vs blue teaming, except the blue team doesn't get source code in this case. So the security guarantee has to come from unit tests alone, not systematic design decisions. Just like in software security, knowing that you've systematically eliminated a problem is much more difficult than finding one instance of the problem.
  - 4y ago
jvanderbot 4y ago
Memorization is absolutely the most valuable part of GPT, for me. I can get natural language responses to documentation, basic scripting / sysadmin, and API questions much more easily than searching other ways. While this is an academic interest point, and rightly tamps down on hype around replacing humans, it doesn't dissuade what I think are most peoples' basic use case: "I don't know or don't remember how to do X, can you show me?" This is finally a good enough "knowledge reference engine" that I can see being useful to those very people it is over hyped to replace.
- vidarh 4y ago
  And asking higher level questions than what you'd otherwise look up. E.g. I've had ChatGPT write forms, write API calls, put together skeletons for all kinds of things that I can easily verify and fix when it gets details from but that are time consuming to do manually. I've held back and been sceptical but I'm at the point where I'm preparing to integrate models all over the place because there are plenty of places where you can add sufficient checks that doing mostly ok much of the time is sufficient to already provide substantial time savings.
  - zer00eyz 4y ago
    > I've held back and been sceptical but I'm at the point where I'm preparing to integrate models all over the place because there are plenty of places where you can add sufficient checks that doing mostly ok much of the time is sufficient to already provide substantial time savings. Im an old engineer. Simply put NO. If you don't understand it don't check it in. You are just getting code to cut and paste at a higher frequency and volume. At some point in time the fire will be burning around you and you won't have the tools to deal with it. Nothing about mostly, much and sufficient ever ends well when it has been done in the name of saving time.
    - vidarh 4y ago
      Nobody suggested checking in anything you don't understand. On the contrary. So maybe try reading again.
madsbuch 4y ago
It seems like many people focus on reasoning capabilities of the GPT models. The me the real value is in the industrial scale pattern recognition capabilities. I can indicate something I vaguely know or ask it to expand on a concept for further research. Within the last hours I have used it to kick-start my research on the AT1 bond and why Credit Suisse let them default and it helped me recall that it was the GenServer pattern I was looking for in Elixir when you have a facade that calls to an independent process.
- czbond 4y ago
  > The me the real value is in the industrial scale pattern recognition capabilities. Absolutely! It has perplexed me why your point is not being discussed more around GPT
- soared 4y ago
  How do you know that the research you’ve conducted is accurate, rather than just precise?
  - YetAnotherNick 4y ago
    For most things, verification is far easier than getting the answer. The same is the case with using stack overflow, where I think that at least half the answer doesn't answer my query, but once I have the potential solution, it is easy to look for the documentation of the key function call etc. Or purely by running it if it is simple and doesn't seem dangerous.
  - madsbuch 4y ago
    I don't And I don't care. As I wrote in the initial comment: > ... kick-start my research ... I use it in conjunction with search engines.
  - pixl97 4y ago
    How do you know when you go to google that your research is accurate?
- rmckayfleming 4y ago
helsinkiandrew 4y ago
> This strongly suggests that the model is able to memorize solutions from its training set I'm not sure why this is a problem - surely in systems like chatGPT we want the specifics that was in the training set not a generalization. It's not learning/reasoning from the training data its 'cleverly regurgitating' things its seen.
- jryb 4y ago
  For some applications, yes, but it comes at the cost of not knowing how powerful ChatGPT really is. So the claim from OpenAI that ChatGPT 4 can pass the bar exam are deceptive since it will likely fail any future bar exam.
  - helsinkiandrew 4y ago
    Agreed, but apart from the novelty factor I'm not sure what the practical use is of ChatGPT passing the bar exam. In fact I think it's a good thing that it can't do well in any future tests as its likely to be used by the unscrupulous
blurbleblurble 4y ago
I disagree that these are "the wrong questions", but I do think we need to try and be nuanced about what these kinds of results actually mean. The potential for these tools to impact labor markets is huge, no matter what they're "actually" or "essentially" capable of. I'm a little tired of the arguments that the large language models are just regurgitating memorized output, I think it's now clear that higher level capabilities are emerging in these models and we need to take this seriously as a social/economic/political challenge. This is "industrial revolution" level technology.
- tarruda 4y ago
  > I think it's now clear that higher level capabilities are emerging in these models and we need to take this seriously as a social/economic/political challenge. It is a hard truth to face. I admit I always feel a little bit of happiness when someone shows me a stupid error ChatGPT made, as if it would somehow invalidate all the awesome things it can do and the impact it will certainly have on all of us. What does it matter if ChatGPT is conscious or not when it can clearly automate a lot of work we previously considered to be creative?. Since last year I started to seriously take a look at AI and started learning about LLMs. Until a few days ago I hadn't bought the explanation that these things are just predicting the next word, but I accepted it once I started running the Alpaca/Llama locally on my computer. The concept of predicting words based on statistics seems simple, but clearly complex behavior emerges from it. Maybe our own intelligence emerges from simple primitives too?
  - cubefox 4y ago
    One possible such simple primitive is predictive coding, where the brain is hypothesized to predict experience rather than text: https://slatestarcodex.com/2017/09/05/book-review-surfing-uncertainty/ https://slatestarcodex.com/2017/09/05/book-review-surfing-un...
- quantiq 4y ago
  >I'm a little tired of the arguments that the large language models are just regurgitating memorized output The arguments are valid and you haven’t provided a single counterpoint. Data leakage is a well known problem in machine learning and OpenAI has seemingly done very little to mitigate against it.
deleted 4y ago
[deleted]
foroak 4y ago
Astonishing that a substack called "AI Snake Oil" would come to this conclusion...
- PaulDavisThe1st 4y ago
  Only about as astonishing as that a company called OpenAI would not ...
VeninVidiaVicii 4y ago
Great read. Adding a bit more — I think these are hinting at the reasons GPT seems to be getting worse. Our expectations realign after the failure modes become excruciatingly obvious after some use. For instance, asking it about differences in pronunciation between European and Brazilian Portuguese, even GPT-4 gives utter nonsense. Even after very carefully explaining the kinds of things I am looking for, it will very carefully repeat what I said, then extrapolate into totally misguided tips like “a” is pronounced “i”. Though, I think it does make sense that GPT would have problems with pronunciation.
- kian 4y ago
  In what format were you asking for the pronunciations?
dcolkitt 4y ago
GPT is a very impressive technical achievement. But that technical achievement is more in the field of compression rather than intelligence.
- sebzim4500 4y ago
  I don't think compression and intelligence can be disentangled in this way.
  - cjbprime 4y ago
    Agree: compression is straightforwardly prediction. Prediction is intelligence when you're able to do it well across disparate novel tasks. (Of course definitions are idiosyncratic, but I'm curious if anyone disagrees with these ones.)
    - cubefox 4y ago
      It seems that intelligence also is about explanation, apart from prediction. Humans not just try to predict future evidence from current evidence, or use the current evidence to confirm given hypotheses, but they also try to find an hypothesis which best explains this evidence. It's not quite clear how explanation would relate to compression.
      - kfarr 4y ago
        Explanation is decompression, using a limited set of tokens to model a more complex output
      - IIAOPSW 4y ago
        Explanation is when the number of bits in your model is smaller than the number of bits in the system. To understand is to have a compression good enough to store in your working memory.
    - Matumio 4y ago
      > Prediction is intelligence Depends on your definition of "intelligence". The big missing part is the ability to explore, try new things, to act (enactivism). Basically to become part of the environment, instead of being a sealed box with frozen weights. By predicting characters, the system had to master, digest, maybe even understand, all the cultural human knowledge it got in text form. Now let's aim for the process that generated this knowledge in the first place.
xivzgrev 4y ago
That was a really interesting point on cross contamination. It perfectly answered questions before a particular date and couldn’t at all do well on questions after. So it’s just really good at memorizing answers and regurgitating. Actually that might not be so different than human test takers.
code_lettuce 4y ago
Refreshing to see a nuanced opinion piece on the recent developments in AI.
wdefoor 4y ago
OpenAI didn’t conduct the bar exam study, Casetext and Stanford did (gotta read those footnotes). The questions were from after the knowledge cutoff and passed the contamination check.
- programmarchy 4y ago
  Sourcing and contamination is covered in the Appendices in the OpenAI paper, which is quoted by this article, and used to critique the method used to detect contamination. > Because of OpenAI’s lack of transparency, we can’t answer the contamination question with certainty. But what’s certain is that OpenAI’s method to detect contamination is superficial and sloppy: > > “We measure cross-contamination between our evaluation dataset and the pre-training data using substring match. Both evaluation and training data are processed by removing all spaces and symbols, keeping only characters (including numbers). For each evaluation example, we randomly select three substrings of 50 characters (or use the entire example if it’s less than 50 characters). A match is identified if any of the three sampled evaluation substrings is a substring of the processed training example. This yields a list of contaminated examples. We discard these and rerun to get uncontaminated scores.” > This is a brittle method. If a test problem were present in the training set with names and numbers changed, it wouldn’t be detected. Less flaky methods are readily available, such as embedding distances. > If OpenAI were to use a distance-based method, how similar is too similar? There is no objective answer to this question. So even something as seemingly straightforward as performance on a multiple-choice standardized test is fraught with subjective decisions.
- calf 4y ago
  The main issue is the inapplicability of a test designed for humans, because a LLM's cognition is very different. Contamination presumes thay the style of tests are applicable.
sebzim4500 4y ago
Clearly contaminated benchmarks are not very useful, but I do not understand the assertion that we should care about "Qualitative studies of professionals using AI" over "Comparison on real world tasks". I've looked through these benchmarks in details, and I've come to the conclusion that real world performance is all that matters. Everything else is either incredibly subjective or designed to beat a particular prior model.
pontus 4y ago
While everyone is debating whether this is impressive or dumb, if it's a leap forward in technology or just a rehashing of old ideas with more data, if we should really care that much about it passing the bar exam or if it's all just a parlor trick, people around the world are starting to use this as a tool and getting real results, becoming more productive, and building stuff... Seems like the proof is in the pudding!
- sebzim4500 4y ago
  Yeah but they aren't using it for the same stuff that they would use a lawyer who got the same result on the bar exam. I think it is fair to say that LLMs have an unfair advantage over humans on these exams, and we should take that into account when trying to assess them.
- int_19h 4y ago
  One of the major reasons why this all is heavily debated, including by those not in the field at all, is because if those things are really capable of human-like reasoning, it leads to answers for some commonly asked philosophical questions on the nature of human conscience, intellect etc that many people find difficult to accept.
  - pontus 4y ago
    Yeah, from a philosophical perspective these are interesting questions to ponder, but my impression of these comments is less that people are pondering the depth of consciousness and more that they're trying to be contrarian / naysayers.
m3kw9 4y ago
Gpt is very good for utility type work like snippets of code, command line interface parameters and some quick explanations. It’s zero to one in that regard but anything more complex or high stakes, you really need verify a lot which diminishes that use case people are hyping about
jeswin 4y ago
In an example [1] referenced in the article, ChatGPT (maybe v3.5) did a poor job of answering the following question. Boll is a new company founded by three recent MBAs. The company plans to buy used automobiles at auctions and then sell them at four times the purchase price direct to consumers (i.e., a car that is purchased for $1,000 would be sold at $4000). Based on an analysis of similar companies, the owners of the company expect to turn their inventory 1.5 times per year. To achieve an annual revenue of $3,000,000, how much inventory, expressed in dollars, should the company expect its operation to require? How does GPT4 fare? ------ Let's break down the problem step by step. The company wants to achieve an annual revenue of $3,000,000. They plan to sell cars at 4 times the purchase price. They expect to turn their inventory 1.5 times per year. First, we need to find out how much revenue they will generate per inventory turnover. To do this, we'll divide the annual revenue by the expected number of inventory turnovers per year: Annual revenue / Inventory turnover rate = Revenue per inventory turnover $3,000,000 / 1.5 = $2,000,000 Now, we know that they plan to sell the cars at 4 times the purchase price. So, we need to find out the purchase price of the cars that will generate $2,000,000 in revenue per inventory turnover: Revenue per inventory turnover / Markup factor = Purchase price of cars $2,000,000 / 4 = $500,000 Therefore, Boll should expect its operation to require $500,000 worth of inventory to achieve an annual revenue of $3,000,000. ------ That V4 is able to do this is in itself just as important as everything said in the article. We live in interesting times. [1]: https://aiguide.substack.com/p/did-chatgpt-really-pass-graduate https://aiguide.substack.com/p/did-chatgpt-really-pass-gradu...
- M4v3R 4y ago
  What's even more interesting, I gave GPT-4 a follow-up instruction: > Please provide a single mathematical equation that could be used to solve this problem. And it gave me the following answer: > Sure, let's represent the required inventory as I, the desired annual revenue as R, the markup factor as M, and the inventory turnover as T. We can create a single equation to solve this problem: > I = (R / M) / T > In this problem, R = $3,000,000, M = 4, and T = 1.5. Plugging these values into the equation will give you the required inventory (I). To me that a language model can do this is simply mind-blowing.
- the_af 4y ago
jrochkind1 4y ago
Just realized that this is yet another example of the category of "imperfect measurement" that HN likes to talk about so much -- Goodhart's law and such. The bar exam is a proxy for actually being a competent lawyer. But it's an imperfect proxy. It seems obvious that it will be possible for a computer program to target getting good on the bar exam, and be good at the bar exam without being a competent lawyer. It may also be possible for a human to? But hard enough that it's still a reasonable proxy when it's humans?
- thwayunion 4y ago
  More-over, in most states, passing that bar isn't enough to practice. In most states, the bar exam is just one component. Humans also need to pass several demanding courses in law school, and to get into law school they need to get a BA/BS degree, which again requires passing several demanding courses and writing various essays. That's just to be allowed to practice law. Getting an actual job as a lawyer often means internships of one form or another, and then spending some time at the beginning of one's career as a de facto apprentice to a seasoned lawyer. These sorts of exams play a very particular role in evaluation of humans.
- Verdex 4y ago
  Yeah, a good thought experiment (or possibly even a good experiment to just straight up run) is to try and decide what's the simplest computer program that could possibly pass the bar exam. And then ask people if they would feel comfortable with that program being their lawyer. So the most obvious solution is to steal the answers and then have the computer blindly paste them into the test, although critics might say that this is 'obviously' not the same as was the language model is doing. I suspect you could pass the bar exam with a bunch of regexes that have an associated formattable string and/or answer result. If someone actually did this, I wonder if people would still be excited about language model techniques passing the bar exam.
  - jrochkind1 4y ago
    I mean, stealing the answers (or even just the questions) would be cheating, of course stolen answers would make it a lot easier for a human to pass too. Nobody is surprised that if you cheat then the exam is no longer a good proxy measure for professional competency! Nobody expects or intends it to be a good proxy measure for those who have stolen the questions. I actually doubt you could write software to pass the bar exam with "a bunch of regexes that have an associated formattable string and/or answer result." I'm not even sure what that means, but I suspect you aren't familiar with bar exams. They are very hard for humans that are in fact familiar with the material; they can contain "trick" questions and require thinking about edge cases, etc. They generally include both essay questions and multiple-choice -- and it can be very tricky multiple choice. Here's just one of the first samples I found googling, I have trouble imagining "regexes with associated answers" doing anything useful here: https://barexam.virginia.gov/pdf/essays/2022%20February%20VA%20Bar%20Exam%20AM.pdf https://barexam.virginia.gov/pdf/essays/2022%20February%20VA... Here are some multiple choice examples: https://www.ncbex.org/pdfviewer/?file=%2Fdmsdocument%2F17 https://www.ncbex.org/pdfviewer/?file=%2Fdmsdocument%2F17 i'd be shocked. But feel free to spend a couple years trying to prove me wrong!
havkom 4y ago
My own experience is that for coding tasks which are probably very well represented in the training data set, such as generating a react page with some functional buttons, GPT-4 performs perfectly. When coming to more specialized tasks (with probably fewer samples in the data set for just that task), such as creating integration between systems with concurrency handling, it still performs surprisingly good but only “first draft” quality and the generated code contains bugs, misses important aspects and the code does not compile usually on the first try - even when using popular programming languages, popular libraries and effort is put in to describing the problem&expected results. Based on this, an excellent tool for developers, but not ready to replace them even though it is surprisingly good. For scaffolding tasks that junior programmers do - it could possibly cut down the need for them (in additio to providing valuable assistance to non-scaffolding standard tasks).
joshuanapoli 4y ago
> Surprisingly, Horace He pointed out that GPT-4 solved 10/10 pre-2021 problems and 0/10 recent problems in the easy category. The training data cutoff for GPT-4 is September 2021. This strongly suggests that the model is able to memorize solutions from its training set — or at least partly memorize them, enough that it can fill in what it can’t recall. Probably the latency of introducing fresh content into the model training will quickly decrease, as these models develop commercial success. It will be really nice when the AI can provide the most cutting edge solutions and up-to-date information relevant to your query. That will be really valuable, even if the AI cannot yet synthesize novel cutting-edge solutions to new problems.
kybernetikos 4y ago
I gave ChatGPT the four cards logic puzzle, lots of humans struggle with it, but chatGPT got it exactly right, so I was very impressed. Then I realised that the formulation I'd given it (the same as the original study) was almost certainly part of its training set. I made extremely minor changes to the way the question was phrased and it failed badly, not just getting the answer wrong but falling into incoherence, claiming that T was a vowel, or that 3 was an even number. The largeness of its training set can give an incorrect impression of its reasoning capabilities. It can apply simple logic to situations, even situations it hasn't seen, but the logic can't get much beyond the first couple of lectures in an introductory First-order Logic course before it starts to fall apart if it can't lean on its large training set data. The fact that it can do logic at all is impressive to me though, I'm interested to see how much deeper its genuine capability goes as we get more advanced models.
- zarzavat 4y ago
  GPT is a token prediction engine. It predicts what the next token is, and it does that very well. Its logical abilities are emergent and are limited by the design of the network. Transformers are constant-time computations: they compute a fixed number of steps and then they stop and produce a result. This is very different to how humans think, we can expend more time on a difficult task (sometimes years!), or give an instant answer to an easy task. And we have a conception of when a task is done, or when we have to think more.
  - kybernetikos 4y ago
    > This is very different to how humans think, we can expend more time on a difficult task (sometimes years!) When we do that, we maintain a chain of thought. It's absolutely possible to get ChatGPT (for instance) to maintain a chain of thought by asking it to plan steps and describe plans before following them. It can allow it to tackle more difficult problems with better results. I don't think we know enough yet about how humans think to be confident in saying that "This is very different to how humans think".
- joenot443 4y ago
  I'd never heard of that puzzle, seems like a great test for ChatGPT though. Wikipedia defines the problem as: You are shown a set of four cards placed on a table, each of which has a number on one side and a colored patch on the other side. The visible faces of the cards show 3, 8, red and brown. Which card(s) must you turn over in order to test the truth of the proposition that if a card shows an even number on one face, then its opposite face is red?
djoldman 4y ago
These are valuable call-outs. They are evidence that the definition of intelligence is at best, a set of ever-moving goalposts and at worst, vague enough to be useless. However. It seems the ML community has centered around the idea that LLMs are zero or few shot learners, meaning that despite only being trained on the task of predicting the next token, they do well on other specific, highly niche, tasks. This is surprising and important. Predictions of immediate profession replacement seem silly. As noted in TFA, there's more to a job than the license test.
examtopics 4y ago
Throwaway for obvious reasons. I’m part of a collective of people that takes exams for people remotely. I would exclusively rely on exam dumps; no longer is the case. ChatGPT, and specifically GPT-3.5-turbo, is not only able to consistently answer novel, new, and fresh exams accurately; I have yet to fail an exam for a client. I’m talking about brand new exams released in 2023, never before seen questions out in the wild. GPT-3.5 (and GPT-4, which I have not needed to test as much) is incredibly good at reasoning and novel exam questions. Take this with a grain of salt as you do all things on the internet. The exams I talk about are the highest levels of certifications possible. GMAT, to professional IT certs. All doable.
Ozzie_osman 4y ago
Looking at human labor, we have some generalists (eg college grad with a general major) who can do some broad range of tasks but can't do very specialized tasks, then experts who can do specialized tasks with very high accuracy (and are much more expensive). My guess is LLMs will proceed the same way. You will have general, base models like GPT4 (I'm assuming we will solve the hallucination problem), then folks will build highly specialized "expert" LLMs for specific domains. You could totally imagine a base LLM delegating to the expert LLMs using some agent/toolformer model, too.
- thwayunion 4y ago
  > I'm assuming we will solve the hallucination problem It's unclear what this would even mean, since "hallucination" carries a surprising number of different definitions and commentators are rarely precise about what they mean when they say hallucination. But, color me skeptical. We will never solve the problem of a token prediction engine being able to generate a sequence of tokens that the vast majority of humans interpret as not corresponding to a true statement. Perhaps in very particular and constrained domains we can build systems that, through a variety of mechanisms, are capable of providing trustworthy automation despite the ever-present risk of hallucination. Something like mathematical proofs checked by a computer are an obvious case where the model can hallucinate because the overall system can gate-keep truth. Doing this in any other domain will, of course, be more difficult. In other words: we may be able to mitigate and systemically manage the risk for some types of particular tasks, but the problem of generating untrue statements is fundamental to the technology and will always require effort to manage and mitigate. In that sense, the whole conversation around hallucination is reminiscent of the frame problem.
  - textninja 4y ago
    > But, color me skeptical That is not a creative color. > We will never solve the problem of a token prediction engine being able to generate a sequence of tokens that the vast majority of humans interpret as not corresponding to a true statement. I think we already solved that problem by making sure the vast majority of humans never agree about anything. It is true that we probably won’t ever get a machine trained on human output to ever be completely accurate (GIGO) but with the right systems and sensors we can at least get probabilistic accuracy. Let’s not forget how human consensus gets shaken up every so many centuries.
dukeofdoom 4y ago
Seems like the wrong answer to the wrong question my produce some sort of break through formula.
epups 4y ago
Regarding this contamination issue, I think the author has a point here but the evidence is just very weak - apparently on one specific test ChatGPT fails when answering recent questions. I wonder if in fact a certain amount of "contamination" is required. When you search for a given problem on Google, you are more likely to find a good answer if every term is meaningful. If you scramble your search and use synonyms, of course your search result will take a hit.
swyx 4y ago
> This is a brittle method. If a test problem were present in the training set with names and numbers changed, it wouldn’t be detected. Less flaky methods are readily available, such as embedding distances. well honestly i think this is a temporary problem for GPT-4. what you do is fuzz your benchmarks by rephrasing them with GPT itself. the same way the image AI people make their models robust to perturbations. you can generate 100 variations for every 1 "real" test. then train to pass those. you've just unlocked GPT-5.
- YeGoblynQueenne 4y ago
  You've just unlocked overfitting.
fwlr 4y ago
I commend them on pushing back on LLM hype and hope their book gets published in a timely manner… but damn I’m also glad that I am not the one writing it, since I fear many of its claims will go the way of IBM President Thomas Watson’s infamous 1940s quote that “there is a world market for about five computers”. The theme that LLMs reproduce knowledge from their training data rather than reason about it seems like one argument that will end up wrong pretty soon. When given the prompt “Which is heavier, one pound of feathers or two pounds of feathers?”, GPT3.5 gives a bizarre answer: “One pound of feathers and two pounds of feathers both weigh the same amount, which is two pounds.” Presumably this is because circa 2016-2017 there was a large internet discussion of the riddle “which weighs more, a pound of feathers or a pound of steel”, and text from this discussion found its way into the training data for the model. I see no reason why the training data would have changed to substantially exclude that discussion, and yet here is GPT 4: “Two pounds of feathers are heavier than one pound of feathers.” Improvements in the model appear to be improving the model’s ability to reason from the training data rather than merely reproduce it. The theme that AI won’t replace e.g. lawyers because it is more knowledge base than reasoning engine also reminds me of early opinions in computer chess discussions, which held that computers were more tactics solvers (short-term several move look-ahead to avoid forks and traps) than strategy planners (long-term construction of multi-piece attacks, protecting small advantages and growing them into large advantages over multiple dozens of moves). With the benefit of hindsight we saw that more of strategy was actually just tactics in disguise than we thought, and that increasing compute could produce real strategy capabilities besides. Separately, there is another theme I see in their writing, and also in some of them comments here: that humans passing standardized tests are doing something fundamentally different than LLMs passing standardized tests. The only thing that’s ‘uniquely human’ is being human, everything else is outputs from a black box. Arguments that ‘what’s inside the black box matters’ are risky, because the outputs gradually converge to indistinguishability; there’s no bright line to step off that train and pretty soon you end up like the person described in Boretti’s And Yet It Understands: “There is a species of denialist for whom no evidence whatever will convince them that a computer is doing anything other than shuffling symbols without understanding them, because “Concepts” and “Ideas” are exclusive to humans (they live in the Leibniz organ, presumably, where they pupate from the black bile). … [These people are] so committed to human chauvinism [that] they will soon start denying their own sentience because their brains are made of flesh and not Chomsky production rules.” https://borretti.me/article/and-yet-it-understands https://borretti.me/article/and-yet-it-understands
neilv 4y ago
> Surprisingly, Horace He pointed out that GPT-4 solved 10/10 pre-2021 problems and 0/10 recent problems in the easy category. The training data cutoff for GPT-4 is September 2021. This strongly suggests that the model is able to memorize solutions from its training set — or at least partly memorize them, enough that it can fill in what it can’t recall. That sounds like it might be relevant to open source laundering lawsuits.
qgin 4y ago
People who have a personal need to stay unimpressed with AI's progress will always find reasons to remain unimpressed.
whatshisface 4y ago
These kinds of arguments, negative as though they sound, signal a tremendous shift in AI capabilities. Ten years ago, the idea that an AI would be able to score 10% on the LSAT would have astounded most of us, and now we're arguing about whether the LSAT encompasses the full spectrum of abilities needed by a lawyer (it doesn't, but that's not the point!).
AtNightWeCode 4y ago
The problem with ChatGPT is that it is sometimes plain wrong. I don't know if there is a diff between 3.5 and 4. Probably not. A test is to try to get ChatGPT to solve simple math problems. It fails. One can even instruct it to apply simple methods like Newton Raphson, Runge Kutta or some Euler bla.
YeGoblynQueenne 4y ago
This is much better than the leakage article by the same authors, where they went on for ever without stopping to explain what "leakage" is. To this day, I have no idea what, exactly, they meant. Reading their article now, I see that they say leakage is "roughly the same as contamination".

340 points by agomez314 4y ago

"10/10 pre-2021, 0/10 post-2021" I guffawed. Hasn't there been ANY update to the Turing test since it was proposed? I suspect the answer to the benchmarking issue was address by philosophers long before we got here.

sebzim4500 4y ago
I think people completely lost interest in the Turing test as soon as it looked like something might actually be able to pass it.
- schrodinger 4y ago
  The Turing test has now become "anything ChatGPT cannot solve yet" :)
enono 4y ago
Why is your username green?!
- schrodinger 4y ago
  New account
  - YeGoblynQueenne 4y ago
    Yes, but why green? In particular?
    - wrycoder 4y ago
      Greenhorn. Look it up.
      - YeGoblynQueenne 4y ago
        Who says that's the association? Citation please.
        wrycoder 4y ago
        LOL you’d have to ask PG, probably. But, you asked for a plausible reason behind an arbitrary choice. What color would you choose?
        YeGoblynQueenne 4y ago
        Green.
        wrycoder 4y ago
        +1 !

thwayunion 4y ago

Absolutely correct. We already know this is about self-driving cars. Passing a driver's test was already possible in 2015 or so, but SDCs clearly aren't ready for L5 deployment even today. There are also a lot of excellent examples of failure modes in object detection benchmarks. Tests, such as driver's tests or standardized exams, are designed for humans. They make a lot of entirely implicit assumptions about failure modes and gaps in knowledge that are uniquely human. Automated systems work differently. They don't fail in the same way that humans fail, and therefore need different benchmarks. Designing good benchmarks that probe GPT systems for common failure modes and weaknesses is actually quite difficult. Much more difficult than designing or training these systems, IME.

zer00eyz 4y ago
> good benchmarks ... failure modes and weaknesses is actually quite difficult. Much more difficult than designing or training these systems Is it? Based on the restrictions placed on the systems we see today and the way people are breaking it, I would say that some failure modes are known.
- thwayunion 4y ago
  A good benchmark is not simply a set of unit tests. What you want in a benchmark is a set of things you can use to measure general improvement; doing better should decrease the propensity of a particular failure mode. Doing this in a way that generalizes beyond specific sub-problems, or even specific inputs in the benchmark suite, is difficult. Building a benchmark suite that's large and comprehensive enough that generalization isn't necessary is also a challenge. Think about an analogy to software security. Exploiting a SQL injection attack in insecure code is easy. Coming up with a set of unit tests that ensures an entire black box software system is free of SQL injection attacks is quite a bit more difficult. Red teaming vs blue teaming, except the blue team doesn't get source code in this case. So the security guarantee has to come from unit tests alone, not systematic design decisions. Just like in software security, knowing that you've systematically eliminated a problem is much more difficult than finding one instance of the problem.
- 4y ago

jvanderbot 4y ago

Memorization is absolutely the most valuable part of GPT, for me. I can get natural language responses to documentation, basic scripting / sysadmin, and API questions much more easily than searching other ways. While this is an academic interest point, and rightly tamps down on hype around replacing humans, it doesn't dissuade what I think are most peoples' basic use case: "I don't know or don't remember how to do X, can you show me?" This is finally a good enough "knowledge reference engine" that I can see being useful to those very people it is over hyped to replace.

vidarh 4y ago
And asking higher level questions than what you'd otherwise look up. E.g. I've had ChatGPT write forms, write API calls, put together skeletons for all kinds of things that I can easily verify and fix when it gets details from but that are time consuming to do manually. I've held back and been sceptical but I'm at the point where I'm preparing to integrate models all over the place because there are plenty of places where you can add sufficient checks that doing mostly ok much of the time is sufficient to already provide substantial time savings.
- zer00eyz 4y ago
  > I've held back and been sceptical but I'm at the point where I'm preparing to integrate models all over the place because there are plenty of places where you can add sufficient checks that doing mostly ok much of the time is sufficient to already provide substantial time savings. Im an old engineer. Simply put NO. If you don't understand it don't check it in. You are just getting code to cut and paste at a higher frequency and volume. At some point in time the fire will be burning around you and you won't have the tools to deal with it. Nothing about mostly, much and sufficient ever ends well when it has been done in the name of saving time.
  - vidarh 4y ago
    Nobody suggested checking in anything you don't understand. On the contrary. So maybe try reading again.

madsbuch 4y ago

It seems like many people focus on reasoning capabilities of the GPT models. The me the real value is in the industrial scale pattern recognition capabilities. I can indicate something I vaguely know or ask it to expand on a concept for further research. Within the last hours I have used it to kick-start my research on the AT1 bond and why Credit Suisse let them default and it helped me recall that it was the GenServer pattern I was looking for in Elixir when you have a facade that calls to an independent process.

czbond 4y ago
> The me the real value is in the industrial scale pattern recognition capabilities. Absolutely! It has perplexed me why your point is not being discussed more around GPT
soared 4y ago
How do you know that the research you’ve conducted is accurate, rather than just precise?
- YetAnotherNick 4y ago
  For most things, verification is far easier than getting the answer. The same is the case with using stack overflow, where I think that at least half the answer doesn't answer my query, but once I have the potential solution, it is easy to look for the documentation of the key function call etc. Or purely by running it if it is simple and doesn't seem dangerous.
- madsbuch 4y ago
  I don't And I don't care. As I wrote in the initial comment: > ... kick-start my research ... I use it in conjunction with search engines.
- pixl97 4y ago
  How do you know when you go to google that your research is accurate?
rmckayfleming 4y ago

helsinkiandrew 4y ago

> This strongly suggests that the model is able to memorize solutions from its training set I'm not sure why this is a problem - surely in systems like chatGPT we want the specifics that was in the training set not a generalization. It's not learning/reasoning from the training data its 'cleverly regurgitating' things its seen.

jryb 4y ago
For some applications, yes, but it comes at the cost of not knowing how powerful ChatGPT really is. So the claim from OpenAI that ChatGPT 4 can pass the bar exam are deceptive since it will likely fail any future bar exam.
- helsinkiandrew 4y ago
  Agreed, but apart from the novelty factor I'm not sure what the practical use is of ChatGPT passing the bar exam. In fact I think it's a good thing that it can't do well in any future tests as its likely to be used by the unscrupulous

blurbleblurble 4y ago

I disagree that these are "the wrong questions", but I do think we need to try and be nuanced about what these kinds of results actually mean. The potential for these tools to impact labor markets is huge, no matter what they're "actually" or "essentially" capable of. I'm a little tired of the arguments that the large language models are just regurgitating memorized output, I think it's now clear that higher level capabilities are emerging in these models and we need to take this seriously as a social/economic/political challenge. This is "industrial revolution" level technology.

tarruda 4y ago
> I think it's now clear that higher level capabilities are emerging in these models and we need to take this seriously as a social/economic/political challenge. It is a hard truth to face. I admit I always feel a little bit of happiness when someone shows me a stupid error ChatGPT made, as if it would somehow invalidate all the awesome things it can do and the impact it will certainly have on all of us. What does it matter if ChatGPT is conscious or not when it can clearly automate a lot of work we previously considered to be creative?. Since last year I started to seriously take a look at AI and started learning about LLMs. Until a few days ago I hadn't bought the explanation that these things are just predicting the next word, but I accepted it once I started running the Alpaca/Llama locally on my computer. The concept of predicting words based on statistics seems simple, but clearly complex behavior emerges from it. Maybe our own intelligence emerges from simple primitives too?
- cubefox 4y ago
  One possible such simple primitive is predictive coding, where the brain is hypothesized to predict experience rather than text: https://slatestarcodex.com/2017/09/05/book-review-surfing-uncertainty/ https://slatestarcodex.com/2017/09/05/book-review-surfing-un...
quantiq 4y ago
>I'm a little tired of the arguments that the large language models are just regurgitating memorized output The arguments are valid and you haven’t provided a single counterpoint. Data leakage is a well known problem in machine learning and OpenAI has seemingly done very little to mitigate against it.

deleted 4y ago

[deleted]

foroak 4y ago

Astonishing that a substack called "AI Snake Oil" would come to this conclusion...

PaulDavisThe1st 4y ago
Only about as astonishing as that a company called OpenAI would not ...

VeninVidiaVicii 4y ago

Great read. Adding a bit more — I think these are hinting at the reasons GPT seems to be getting worse. Our expectations realign after the failure modes become excruciatingly obvious after some use. For instance, asking it about differences in pronunciation between European and Brazilian Portuguese, even GPT-4 gives utter nonsense. Even after very carefully explaining the kinds of things I am looking for, it will very carefully repeat what I said, then extrapolate into totally misguided tips like “a” is pronounced “i”. Though, I think it does make sense that GPT would have problems with pronunciation.

kian 4y ago
In what format were you asking for the pronunciations?

dcolkitt 4y ago

GPT is a very impressive technical achievement. But that technical achievement is more in the field of compression rather than intelligence.

sebzim4500 4y ago
I don't think compression and intelligence can be disentangled in this way.
- cjbprime 4y ago
  Agree: compression is straightforwardly prediction. Prediction is intelligence when you're able to do it well across disparate novel tasks. (Of course definitions are idiosyncratic, but I'm curious if anyone disagrees with these ones.)
  - cubefox 4y ago
    It seems that intelligence also is about explanation, apart from prediction. Humans not just try to predict future evidence from current evidence, or use the current evidence to confirm given hypotheses, but they also try to find an hypothesis which best explains this evidence. It's not quite clear how explanation would relate to compression.
    - kfarr 4y ago
      Explanation is decompression, using a limited set of tokens to model a more complex output
    - IIAOPSW 4y ago
      Explanation is when the number of bits in your model is smaller than the number of bits in the system. To understand is to have a compression good enough to store in your working memory.
  - Matumio 4y ago
    > Prediction is intelligence Depends on your definition of "intelligence". The big missing part is the ability to explore, try new things, to act (enactivism). Basically to become part of the environment, instead of being a sealed box with frozen weights. By predicting characters, the system had to master, digest, maybe even understand, all the cultural human knowledge it got in text form. Now let's aim for the process that generated this knowledge in the first place.

xivzgrev 4y ago

That was a really interesting point on cross contamination. It perfectly answered questions before a particular date and couldn’t at all do well on questions after. So it’s just really good at memorizing answers and regurgitating. Actually that might not be so different than human test takers.

code_lettuce 4y ago

Refreshing to see a nuanced opinion piece on the recent developments in AI.

wdefoor 4y ago

OpenAI didn’t conduct the bar exam study, Casetext and Stanford did (gotta read those footnotes). The questions were from after the knowledge cutoff and passed the contamination check.

programmarchy 4y ago
Sourcing and contamination is covered in the Appendices in the OpenAI paper, which is quoted by this article, and used to critique the method used to detect contamination. > Because of OpenAI’s lack of transparency, we can’t answer the contamination question with certainty. But what’s certain is that OpenAI’s method to detect contamination is superficial and sloppy: > > “We measure cross-contamination between our evaluation dataset and the pre-training data using substring match. Both evaluation and training data are processed by removing all spaces and symbols, keeping only characters (including numbers). For each evaluation example, we randomly select three substrings of 50 characters (or use the entire example if it’s less than 50 characters). A match is identified if any of the three sampled evaluation substrings is a substring of the processed training example. This yields a list of contaminated examples. We discard these and rerun to get uncontaminated scores.” > This is a brittle method. If a test problem were present in the training set with names and numbers changed, it wouldn’t be detected. Less flaky methods are readily available, such as embedding distances. > If OpenAI were to use a distance-based method, how similar is too similar? There is no objective answer to this question. So even something as seemingly straightforward as performance on a multiple-choice standardized test is fraught with subjective decisions.
calf 4y ago
The main issue is the inapplicability of a test designed for humans, because a LLM's cognition is very different. Contamination presumes thay the style of tests are applicable.

sebzim4500 4y ago

Clearly contaminated benchmarks are not very useful, but I do not understand the assertion that we should care about "Qualitative studies of professionals using AI" over "Comparison on real world tasks". I've looked through these benchmarks in details, and I've come to the conclusion that real world performance is all that matters. Everything else is either incredibly subjective or designed to beat a particular prior model.

pontus 4y ago

While everyone is debating whether this is impressive or dumb, if it's a leap forward in technology or just a rehashing of old ideas with more data, if we should really care that much about it passing the bar exam or if it's all just a parlor trick, people around the world are starting to use this as a tool and getting real results, becoming more productive, and building stuff... Seems like the proof is in the pudding!

sebzim4500 4y ago
Yeah but they aren't using it for the same stuff that they would use a lawyer who got the same result on the bar exam. I think it is fair to say that LLMs have an unfair advantage over humans on these exams, and we should take that into account when trying to assess them.
int_19h 4y ago
One of the major reasons why this all is heavily debated, including by those not in the field at all, is because if those things are really capable of human-like reasoning, it leads to answers for some commonly asked philosophical questions on the nature of human conscience, intellect etc that many people find difficult to accept.
- pontus 4y ago
  Yeah, from a philosophical perspective these are interesting questions to ponder, but my impression of these comments is less that people are pondering the depth of consciousness and more that they're trying to be contrarian / naysayers.

m3kw9 4y ago

Gpt is very good for utility type work like snippets of code, command line interface parameters and some quick explanations. It’s zero to one in that regard but anything more complex or high stakes, you really need verify a lot which diminishes that use case people are hyping about

jeswin 4y ago

In an example [1] referenced in the article, ChatGPT (maybe v3.5) did a poor job of answering the following question. Boll is a new company founded by three recent MBAs. The company plans to buy used automobiles at auctions and then sell them at four times the purchase price direct to consumers (i.e., a car that is purchased for $1,000 would be sold at $4000). Based on an analysis of similar companies, the owners of the company expect to turn their inventory 1.5 times per year. To achieve an annual revenue of $3,000,000, how much inventory, expressed in dollars, should the company expect its operation to require? How does GPT4 fare? ------ Let's break down the problem step by step. The company wants to achieve an annual revenue of $3,000,000. They plan to sell cars at 4 times the purchase price. They expect to turn their inventory 1.5 times per year. First, we need to find out how much revenue they will generate per inventory turnover. To do this, we'll divide the annual revenue by the expected number of inventory turnovers per year: Annual revenue / Inventory turnover rate = Revenue per inventory turnover $3,000,000 / 1.5 = $2,000,000 Now, we know that they plan to sell the cars at 4 times the purchase price. So, we need to find out the purchase price of the cars that will generate $2,000,000 in revenue per inventory turnover: Revenue per inventory turnover / Markup factor = Purchase price of cars $2,000,000 / 4 = $500,000 Therefore, Boll should expect its operation to require $500,000 worth of inventory to achieve an annual revenue of $3,000,000. ------ That V4 is able to do this is in itself just as important as everything said in the article. We live in interesting times. [1]: https://aiguide.substack.com/p/did-chatgpt-really-pass-graduate https://aiguide.substack.com/p/did-chatgpt-really-pass-gradu...

M4v3R 4y ago
What's even more interesting, I gave GPT-4 a follow-up instruction: > Please provide a single mathematical equation that could be used to solve this problem. And it gave me the following answer: > Sure, let's represent the required inventory as I, the desired annual revenue as R, the markup factor as M, and the inventory turnover as T. We can create a single equation to solve this problem: > I = (R / M) / T > In this problem, R = $3,000,000, M = 4, and T = 1.5. Plugging these values into the equation will give you the required inventory (I). To me that a language model can do this is simply mind-blowing.
the_af 4y ago

jrochkind1 4y ago

Just realized that this is yet another example of the category of "imperfect measurement" that HN likes to talk about so much -- Goodhart's law and such. The bar exam is a proxy for actually being a competent lawyer. But it's an imperfect proxy. It seems obvious that it will be possible for a computer program to target getting good on the bar exam, and be good at the bar exam without being a competent lawyer. It may also be possible for a human to? But hard enough that it's still a reasonable proxy when it's humans?

thwayunion 4y ago
More-over, in most states, passing that bar isn't enough to practice. In most states, the bar exam is just one component. Humans also need to pass several demanding courses in law school, and to get into law school they need to get a BA/BS degree, which again requires passing several demanding courses and writing various essays. That's just to be allowed to practice law. Getting an actual job as a lawyer often means internships of one form or another, and then spending some time at the beginning of one's career as a de facto apprentice to a seasoned lawyer. These sorts of exams play a very particular role in evaluation of humans.
Verdex 4y ago
Yeah, a good thought experiment (or possibly even a good experiment to just straight up run) is to try and decide what's the simplest computer program that could possibly pass the bar exam. And then ask people if they would feel comfortable with that program being their lawyer. So the most obvious solution is to steal the answers and then have the computer blindly paste them into the test, although critics might say that this is 'obviously' not the same as was the language model is doing. I suspect you could pass the bar exam with a bunch of regexes that have an associated formattable string and/or answer result. If someone actually did this, I wonder if people would still be excited about language model techniques passing the bar exam.
- jrochkind1 4y ago
  I mean, stealing the answers (or even just the questions) would be cheating, of course stolen answers would make it a lot easier for a human to pass too. Nobody is surprised that if you cheat then the exam is no longer a good proxy measure for professional competency! Nobody expects or intends it to be a good proxy measure for those who have stolen the questions. I actually doubt you could write software to pass the bar exam with "a bunch of regexes that have an associated formattable string and/or answer result." I'm not even sure what that means, but I suspect you aren't familiar with bar exams. They are very hard for humans that are in fact familiar with the material; they can contain "trick" questions and require thinking about edge cases, etc. They generally include both essay questions and multiple-choice -- and it can be very tricky multiple choice. Here's just one of the first samples I found googling, I have trouble imagining "regexes with associated answers" doing anything useful here: https://barexam.virginia.gov/pdf/essays/2022%20February%20VA%20Bar%20Exam%20AM.pdf https://barexam.virginia.gov/pdf/essays/2022%20February%20VA... Here are some multiple choice examples: https://www.ncbex.org/pdfviewer/?file=%2Fdmsdocument%2F17 https://www.ncbex.org/pdfviewer/?file=%2Fdmsdocument%2F17 i'd be shocked. But feel free to spend a couple years trying to prove me wrong!

havkom 4y ago

My own experience is that for coding tasks which are probably very well represented in the training data set, such as generating a react page with some functional buttons, GPT-4 performs perfectly. When coming to more specialized tasks (with probably fewer samples in the data set for just that task), such as creating integration between systems with concurrency handling, it still performs surprisingly good but only “first draft” quality and the generated code contains bugs, misses important aspects and the code does not compile usually on the first try - even when using popular programming languages, popular libraries and effort is put in to describing the problem&expected results. Based on this, an excellent tool for developers, but not ready to replace them even though it is surprisingly good. For scaffolding tasks that junior programmers do - it could possibly cut down the need for them (in additio to providing valuable assistance to non-scaffolding standard tasks).

joshuanapoli 4y ago

> Surprisingly, Horace He pointed out that GPT-4 solved 10/10 pre-2021 problems and 0/10 recent problems in the easy category. The training data cutoff for GPT-4 is September 2021. This strongly suggests that the model is able to memorize solutions from its training set — or at least partly memorize them, enough that it can fill in what it can’t recall. Probably the latency of introducing fresh content into the model training will quickly decrease, as these models develop commercial success. It will be really nice when the AI can provide the most cutting edge solutions and up-to-date information relevant to your query. That will be really valuable, even if the AI cannot yet synthesize novel cutting-edge solutions to new problems.

kybernetikos 4y ago

I gave ChatGPT the four cards logic puzzle, lots of humans struggle with it, but chatGPT got it exactly right, so I was very impressed. Then I realised that the formulation I'd given it (the same as the original study) was almost certainly part of its training set. I made extremely minor changes to the way the question was phrased and it failed badly, not just getting the answer wrong but falling into incoherence, claiming that T was a vowel, or that 3 was an even number. The largeness of its training set can give an incorrect impression of its reasoning capabilities. It can apply simple logic to situations, even situations it hasn't seen, but the logic can't get much beyond the first couple of lectures in an introductory First-order Logic course before it starts to fall apart if it can't lean on its large training set data. The fact that it can do logic at all is impressive to me though, I'm interested to see how much deeper its genuine capability goes as we get more advanced models.

zarzavat 4y ago
GPT is a token prediction engine. It predicts what the next token is, and it does that very well. Its logical abilities are emergent and are limited by the design of the network. Transformers are constant-time computations: they compute a fixed number of steps and then they stop and produce a result. This is very different to how humans think, we can expend more time on a difficult task (sometimes years!), or give an instant answer to an easy task. And we have a conception of when a task is done, or when we have to think more.
- kybernetikos 4y ago
  > This is very different to how humans think, we can expend more time on a difficult task (sometimes years!) When we do that, we maintain a chain of thought. It's absolutely possible to get ChatGPT (for instance) to maintain a chain of thought by asking it to plan steps and describe plans before following them. It can allow it to tackle more difficult problems with better results. I don't think we know enough yet about how humans think to be confident in saying that "This is very different to how humans think".
joenot443 4y ago
I'd never heard of that puzzle, seems like a great test for ChatGPT though. Wikipedia defines the problem as: You are shown a set of four cards placed on a table, each of which has a number on one side and a colored patch on the other side. The visible faces of the cards show 3, 8, red and brown. Which card(s) must you turn over in order to test the truth of the proposition that if a card shows an even number on one face, then its opposite face is red?

djoldman 4y ago

These are valuable call-outs. They are evidence that the definition of intelligence is at best, a set of ever-moving goalposts and at worst, vague enough to be useless. However. It seems the ML community has centered around the idea that LLMs are zero or few shot learners, meaning that despite only being trained on the task of predicting the next token, they do well on other specific, highly niche, tasks. This is surprising and important. Predictions of immediate profession replacement seem silly. As noted in TFA, there's more to a job than the license test.

examtopics 4y ago

Throwaway for obvious reasons. I’m part of a collective of people that takes exams for people remotely. I would exclusively rely on exam dumps; no longer is the case. ChatGPT, and specifically GPT-3.5-turbo, is not only able to consistently answer novel, new, and fresh exams accurately; I have yet to fail an exam for a client. I’m talking about brand new exams released in 2023, never before seen questions out in the wild. GPT-3.5 (and GPT-4, which I have not needed to test as much) is incredibly good at reasoning and novel exam questions. Take this with a grain of salt as you do all things on the internet. The exams I talk about are the highest levels of certifications possible. GMAT, to professional IT certs. All doable.

Ozzie_osman 4y ago

Looking at human labor, we have some generalists (eg college grad with a general major) who can do some broad range of tasks but can't do very specialized tasks, then experts who can do specialized tasks with very high accuracy (and are much more expensive). My guess is LLMs will proceed the same way. You will have general, base models like GPT4 (I'm assuming we will solve the hallucination problem), then folks will build highly specialized "expert" LLMs for specific domains. You could totally imagine a base LLM delegating to the expert LLMs using some agent/toolformer model, too.

thwayunion 4y ago
> I'm assuming we will solve the hallucination problem It's unclear what this would even mean, since "hallucination" carries a surprising number of different definitions and commentators are rarely precise about what they mean when they say hallucination. But, color me skeptical. We will never solve the problem of a token prediction engine being able to generate a sequence of tokens that the vast majority of humans interpret as not corresponding to a true statement. Perhaps in very particular and constrained domains we can build systems that, through a variety of mechanisms, are capable of providing trustworthy automation despite the ever-present risk of hallucination. Something like mathematical proofs checked by a computer are an obvious case where the model can hallucinate because the overall system can gate-keep truth. Doing this in any other domain will, of course, be more difficult. In other words: we may be able to mitigate and systemically manage the risk for some types of particular tasks, but the problem of generating untrue statements is fundamental to the technology and will always require effort to manage and mitigate. In that sense, the whole conversation around hallucination is reminiscent of the frame problem.
- textninja 4y ago
  > But, color me skeptical That is not a creative color. > We will never solve the problem of a token prediction engine being able to generate a sequence of tokens that the vast majority of humans interpret as not corresponding to a true statement. I think we already solved that problem by making sure the vast majority of humans never agree about anything. It is true that we probably won’t ever get a machine trained on human output to ever be completely accurate (GIGO) but with the right systems and sensors we can at least get probabilistic accuracy. Let’s not forget how human consensus gets shaken up every so many centuries.

dukeofdoom 4y ago

Seems like the wrong answer to the wrong question my produce some sort of break through formula.

epups 4y ago

Regarding this contamination issue, I think the author has a point here but the evidence is just very weak - apparently on one specific test ChatGPT fails when answering recent questions. I wonder if in fact a certain amount of "contamination" is required. When you search for a given problem on Google, you are more likely to find a good answer if every term is meaningful. If you scramble your search and use synonyms, of course your search result will take a hit.

swyx 4y ago

> This is a brittle method. If a test problem were present in the training set with names and numbers changed, it wouldn’t be detected. Less flaky methods are readily available, such as embedding distances. well honestly i think this is a temporary problem for GPT-4. what you do is fuzz your benchmarks by rephrasing them with GPT itself. the same way the image AI people make their models robust to perturbations. you can generate 100 variations for every 1 "real" test. then train to pass those. you've just unlocked GPT-5.

YeGoblynQueenne 4y ago
You've just unlocked overfitting.

fwlr 4y ago

I commend them on pushing back on LLM hype and hope their book gets published in a timely manner… but damn I’m also glad that I am not the one writing it, since I fear many of its claims will go the way of IBM President Thomas Watson’s infamous 1940s quote that “there is a world market for about five computers”. The theme that LLMs reproduce knowledge from their training data rather than reason about it seems like one argument that will end up wrong pretty soon. When given the prompt “Which is heavier, one pound of feathers or two pounds of feathers?”, GPT3.5 gives a bizarre answer: “One pound of feathers and two pounds of feathers both weigh the same amount, which is two pounds.” Presumably this is because circa 2016-2017 there was a large internet discussion of the riddle “which weighs more, a pound of feathers or a pound of steel”, and text from this discussion found its way into the training data for the model. I see no reason why the training data would have changed to substantially exclude that discussion, and yet here is GPT 4: “Two pounds of feathers are heavier than one pound of feathers.” Improvements in the model appear to be improving the model’s ability to reason from the training data rather than merely reproduce it. The theme that AI won’t replace e.g. lawyers because it is more knowledge base than reasoning engine also reminds me of early opinions in computer chess discussions, which held that computers were more tactics solvers (short-term several move look-ahead to avoid forks and traps) than strategy planners (long-term construction of multi-piece attacks, protecting small advantages and growing them into large advantages over multiple dozens of moves). With the benefit of hindsight we saw that more of strategy was actually just tactics in disguise than we thought, and that increasing compute could produce real strategy capabilities besides. Separately, there is another theme I see in their writing, and also in some of them comments here: that humans passing standardized tests are doing something fundamentally different than LLMs passing standardized tests. The only thing that’s ‘uniquely human’ is being human, everything else is outputs from a black box. Arguments that ‘what’s inside the black box matters’ are risky, because the outputs gradually converge to indistinguishability; there’s no bright line to step off that train and pretty soon you end up like the person described in Boretti’s And Yet It Understands: “There is a species of denialist for whom no evidence whatever will convince them that a computer is doing anything other than shuffling symbols without understanding them, because “Concepts” and “Ideas” are exclusive to humans (they live in the Leibniz organ, presumably, where they pupate from the black bile). … [These people are] so committed to human chauvinism [that] they will soon start denying their own sentience because their brains are made of flesh and not Chomsky production rules.” https://borretti.me/article/and-yet-it-understands https://borretti.me/article/and-yet-it-understands

neilv 4y ago

qgin 4y ago

People who have a personal need to stay unimpressed with AI's progress will always find reasons to remain unimpressed.

whatshisface 4y ago

These kinds of arguments, negative as though they sound, signal a tremendous shift in AI capabilities. Ten years ago, the idea that an AI would be able to score 10% on the LSAT would have astounded most of us, and now we're arguing about whether the LSAT encompasses the full spectrum of abilities needed by a lawyer (it doesn't, but that's not the point!).

AtNightWeCode 4y ago

The problem with ChatGPT is that it is sometimes plain wrong. I don't know if there is a diff between 3.5 and 4. Probably not. A test is to try to get ChatGPT to solve simple math problems. It fails. One can even instruct it to apply simple methods like Newton Raphson, Runge Kutta or some Euler bla.

YeGoblynQueenne 4y ago

This is much better than the leakage article by the same authors, where they went on for ever without stopping to explain what "leakage" is. To this day, I have no idea what, exactly, they meant. Reading their article now, I see that they say leakage is "roughly the same as contamination".