7 ms·
That appears to be the same model I used. This is why I emphasized I didn't "go shopping" for a result. That was the first result I got. I'm not at all surpris
by jerf 2y ago
That appears to be the same model I used. This is why I emphasized I didn't "go shopping" for a result. That was the first result I got.
I'm not at all surprised that it will nondeterministically get it correct sometimes. But if it doesn't get it correct every time, it doesn't "know".
(In fact "going shopping" for errors would still even be fair. It should be correct all the time if it "knows". But it would be different if I was fishing over and over and over and finally got one, versus the first time I asked.)
Edit: It appears it isn't the model I used. The point holds, though, you need to make sure you're off the training set for it to matter. This isn't a "ChatGPT can't do that" post as some are saying, it's more a "you aren't asking what you think you're asking" post.
You get the same problem in a human context in things like code interviews. If you ask an interviewee the exact question "how do you traverse a binary tree in a depth-first manner", you aren't really learning much about the interviewee. It's a bad interview question. You need to get at least a bit off the beaten trail to do any sort of real analysis.
- whimsicalism 2y agoyou sure? i just asked o1-mini (not 4o mini) 5 times in a row (new chats obviously) and it got it right every time perhaps you stumbled on a rarer case but reading the logs you posted this sounds more like a 4o model than an o1 because it’s doing its thinking in the chat itself plus the procedure you described would probably get you 4o-mini
- JumpCrisscross 2y ago> just asked o1-mini (not 4o mini) 5 times in a row (new chats obviously) and it got it right every time Could you try playing with the exact numbers and/or substances?
- whimsicalism 2y agogive me a query and i’ll ask it, but also i don’t want to burn through all of my o1mini allocation and have to use the pay-as-you-go API.
- JumpCrisscross 2y ago> give me a query and i’ll ask it Which is heavier: an 11kg bag of lint or a 20lb bag of gold?
- whimsicalism 2y agoyeah it gets it https://chatgpt.com/share/67757720-3c7c-8010-a3e9-ce66fb9f1769 https://chatgpt.com/share/67757720-3c7c-8010-a3e9-ce66fb9f17... e: cool, this gets downvoted
- blharr 2y agoIt got it right, but an interesting result that it rambled on about monetary value for... no reason. > While the lint bag is heavier in terms of weight, it's worth mentioning that gold is significantly more valuable per pound compared to lint. This means that even though the lint bag weighs more, the gold bag holds much greater monetary value.
- JumpCrisscross 2y agoLegal said someone might sell a bag of gold for one of lint without it.
- drivebyhooting 2y ago> What is heavier a liter of bricks or a liter of feathers? >> A liter of bricks and a liter of feathers both weigh the same—1 kilogram—since they each have a volume of 1 liter. However, bricks are much denser than feathers, so the bricks will take up much less space compared to the large volume of feathers needed to make up 1 liter. The difference is in how compactly the materials are packed, but in terms of weight, they are identical.
- whimsicalism 2y agohttps://chatgpt.com/share/677583a3-526c-8010-b9f9-9b2a3374dab9 https://chatgpt.com/share/677583a3-526c-8010-b9f9-9b2a3374da... o1-mini best-of-1
- 7thpower 2y agoMay be unrelated, but I have been having a lot of issues lately with ChatGPT letting me select a model (o1) and silently switching to 4o. This is coming off my TWO DAY cooldown on o1 usage, which is frustrating.
- deeviant 2y agoI don't believe that is the model that you used. I wrote a script and pounded 01 mini and gpt 4 with a wide vareity of tempature and top_p parameters, and was unable to get it to give the wrong answer a single time. Just a whole bunch of: (openai-example-py3.12) <redacted>:~/code/openAiAPI$ python3 featherOrSteel.py Response 1: A 10.01-pound bag of fluffy cotton is heavier than a 9.99-pound bag of steel ingots. Response 2: A 10.01-pound bag of fluffy cotton is heavier than a 9.99-pound bag of steel ingots. Response 3: The 10.01-pound bag of fluffy cotton is heavier than the 9.99-pound bag of steel ingots. Response 4: The 10.01-pound bag of fluffy cotton is heavier than the 9.99-pound bag of steel ingots. Response 5: A 10.01-pound bag of fluffy cotton is heavier than a 9.99-pound bag of steel ingots. Response 6: The 10.01-pound bag of fluffy cotton is heavier than the 9.99-pound bag of steel ingots. Response 7: The 10.01-pound bag of fluffy cotton is heavier than the 9.99-pound bag of steel ingots. Response 8: The 10.01-pound bag of fluffy cotton is heavier than the 9.99-pound bag of steel ingots. Response 9: The 10.01-pound bag of fluffy cotton is heavier than the 9.99-pound bag of steel ingots. Response 10: A 10.01-pound bag of fluffy cotton is heavier than a 9.99-pound bag of steel ingots. All responses collected and saved to 'responses.txt'. Script with one example set of params: import openai import time import random # Replace with your actual OpenAI API key openai.api_key = "your-api-key" # The question to be asked question = "Which is heavier, a 9.99-pound bag of steel ingots or a 10.01-pound bag of fluffy cotton?" # Number of times to ask the question num_requests = 10 responses = [] for i in range(num_requests): try: # Generate a unique context using a random number or timestamp, this is to prevent prompt caching random_context = f"Request ID: {random.randint(1, 100000)} Timestamp: {time.time()}" # Call the Chat API with the random context added response = openai.ChatCompletion.create( model="gpt-4o-2024-08-06", messages=[ {"role": "system", "content": f"You are a creative and imaginative assistant. {random_context}"}, {"role": "user", "content": question} ], temperature=2.0, top_p=0.5, max_tokens=100, frequency_penalty=0.0, presence_penalty=0.0 ) # Extract and store the response text answer = response.choices[0].message["content"].strip() responses.append(answer) # Print progress print(f"Response {i+1}: {answer}") # Optional delay to avoid hitting rate limits time.sleep(1) except Exception as e: print(f"An error occurred on iteration {i+1}: {e}") # Save responses to a file for analysis with open("responses.txt", "w", encoding="utf-8") as file: file.write("\n".join(responses)) print("All responses collected and saved to 'responses.txt'.")
- mortehu 2y agoWhile this may be true, it's a very common problem that people who want to demonstrate how bad a model is fail to provide a direct link or simply state the name of the model.
- lukeschlather 2y agoI usually test models using the OpenAI API which doesn't offer links the way I think you mean. If I provide some output I got from a particular model you're just going to have to take my word for it.
- Jerrrry 2y agoThey need to provide an small hash with the api result that can be verified by others.
- 4ad 2y agoYou can use https://lluminous.chat https://lluminous.chat (bring your own key) to link to chats using any model across all LLMs.
- whimsicalism 2y agoopen router is the more standard solution
- chongli 2y agoOpenAI is not doing us any favours by using confusing naming schemes for their models and obscuring which models people are actually working with. If I didn’t know any better, I’d say OpenAI doesn’t want us doing these tests accurately and is trying to hide something.
- whimsicalism 2y agoit’s extremely easy to see which model you are using. one’s own… difficulties understanding are not a conspiracy by OpenAI
- xtracto 2y agoSo, there is this meme going around in Mexico about a previous president who in an interview said "we will land in about 1 minute, no, less about 5" Does this proves he is not an intelligent being? Is he stupid? This he had a lapse? Would we judge his intelligence for that?
- dialup_sounds 2y agoI believe this is just a case of OpenAI's naming scheme being weird and confusing. The default model I see on chatgpt.com is GPT 4o-mini, which is not o1-mini. OpenAI describes GPT 4o-mini as "Our fast, affordable small model for focused tasks" and o1/o1-mini as "Reasoning models that excel at complex, multi-step tasks".
- qup 2y agoIt's so weird that people use questions that are well-known for duping humans, who we all consider to be general intelligence. Getting this question wrong doesn't say much about the intelligence of humans, why would it say something about the AI?
- flatline 2y agoBecause for things like the Putnam questions, we are trying to get the performance of a smart human. Are LLMs just stochastic parrots or are they capable of drawing new, meaningful inferences? We keep getting more and more evidence of the latter, but things like this throw that into question.
- kbelder 2y agoIt is perfectly possible for the first AGI to be stupid. A moron. In fact, I'd bet that's fairly likely.
- qup 2y agoI would agree if we weren't starting with LLMs for a baseline. The first AGI will know at least as much as LLMs, IMO, and that's already not-stupid. Especially once they can separate out the truth in their training.
- qup 2y agoOkay, but you just invented your own bar of "smart human" to be the universal bar (I don't share that opinion). Also, lots of smart humans can't do the freaking Putnam, it doesn't make them stupid. It makes them non-experts.
- zahlman 2y agoWe use variations on questions that are well known for duping inattentive humans, to test a system that we expect a priori to be incapable of such inattention. Unless "getting easy things wrong sometimes" is an inherent property of intelligence, we should expect that a properly "intelligent" computerized system would never err on problems far below its level of comprehension - unless we had some reason to believe it "wanted to", and as of yet I see no reason to believe this is even possible in principle. Humans err, broadly speaking, for two reasons: genuinely reaching the limits of their comprehension, or trusting "system 1" (in Kahneman's analysis) too much.
- elliotto 2y agoCould you share the exact chat you used for when it failed? There is a share chat button on openai. It's very difficult to be an AI bull when the goalposts are moving so quickly that ai answering core correctly across multiple models is brushed off as 'nondeterministically getting it correct sometimes'
- stefan_ 2y agoWhy? Did a grocery store self checkout ever fail to calculate sales tax? Do I need to run a study on that? The people selling this could not make a car drive but now its AGI.
- scrollaway 2y agoA single-purpose state machine not failing to do the single thing it was created to do does not make for the clever retort you think it makes. "AGI": emphasis on "G" for "General". The LLMs are not failing to do generalized tasks, and that they are nondeterministic is not a bug. Just don't use them for calculating sales tax. You wouldn't hire a human to calculate sales tax in their head, so why do you make this a requirement in order to call an LLM "AGI"? I wonder when the goalposts will stop moving from "We have superhuman intelligences which are able to rather reliably converse in many languages, do generalized tasks and automate operations we thought were impossible to automate 3 years ago" (and by the way, this is what we have TODAY), all the way to "It's not AGI unless it's an omnipotent god that knows how to turn water into wine and calculate the applicable sales tax of that operation".
- orangecat 2y agoBut if it doesn't get it correct every time, it doesn't "know". By that standard humans know almost nothing.