15 ms·

The o1-preview model still hallucinates non-existing libraries and functions for me, and is quickly wrong about facts that aren't well-represented on the web. I

by layer8 2y ago

The o1-preview model still hallucinates non-existing libraries and functions for me, and is quickly wrong about facts that aren't well-represented on the web. It's the usual string of "You're absolutely correct, and I apologize for the oversight in my previous response. [Let me make another guess.]" While the reasoning may have been improved, this doesn't solve the problem of the model having no way to assess if what it conjures up from its weights is factual or not.

kranuck 2y ago
That problems feels somewhat fundamental to saying that these things have any ability to reason at all.
tptacek 2y ago
I've had the opposite experience with some coding samples. After reading Nick Carlini's post, I've gotten into the habit of powering through coding problems with GPT (where previously I'd just laugh and immediately give up) by just presenting it the errors in its code and asking it to fix them. o1 seems to be effectively screening for some of those errors (I assume it's just some, but I've noticed that the o1 things I've done haven't had obvious dumb errors like missing imports, and all my 4o attempts have).
- layer8 2y ago
  My experience is likely colored by the fact that I tend to turn to LLMs for problems I have trouble solving by myself. I typically don't use them for the low-hanging fruits. That's the frustrating thing. LLMs don't materially reduce the set of problems where I'm running against a wall or have trouble finding information.
  - mensetmanusman 2y ago
    LLMs are not for expanding the sphere of human knowledge, but for speeding up auto-correct of higher order processing to help you more quickly reach the shell of the sphere and make progress with your own mind :)
    - tsunamifury 2y ago
      Definitely. When we talk about being skilled in a T shape LLMs are all about spreading your top of T and not making the bottom go deeper.
      - ben_w 2y ago
        Indeed, not much more depth — though even Terence Tao reported useful results from an earlier version, so perhaps the breadth is a depth all of it's own: https://mathstodon.xyz/@tao/110601051375142142 https://mathstodon.xyz/@tao/110601051375142142 I think of it as making the top bar of the T thicker, but yes, you're right, it also spreads it much wider.
      - skydhash 2y ago
        I prefer reading some book. Maybe the LLM was trained on some piece of knowledge not available on the net, but I much prefer the reliability and consistency of a book.
  - tptacek 2y ago
    I use LLMs for three things: * To catch passive voice and nominalizations in my writing. * To convert Linux kernel subsystems into Python so I can quickly understand them (I'm a C programmer but everyone reads Python faster). * To write dumb programs using languages and libraries I haven't used much before; for instance, I'm an ActiveRecord person and needed to do some SQLAlchemy stuff today, and GPT 4o (and o1) kept me away from the SQLAlchemy documentation. OpenAI talks about o1 going head to head with PhDs. I could care less. But for the specific problem we're talking about on this subthread: o1 seems materially better.
    - throwup238 2y ago
      > * To convert Linux kernel subsystems into Python so I can quickly understand them (I'm a C programmer but everyone reads Python faster). Do you have an example chat of this output? Sounds interesting. Do you just dump the C source code into the prompt and ask it to convert to Python?
      - tptacek 2y ago
        No, ChatGPT is way cooler than that. It's already read every line of kernel code ever written. I start with a subsystem: the device mapper is a good recent example. I ask things like "explain the linux device mapper. if it was a class in an object-oriented language, what would its interface look like?" and "give me dm_target as a python class". I get stuff like: def linear_ctr(target, argc, argv): print("Constructor called with args:", argc, argv) # Initialize target-specific data here return 0 def linear_dtr(target): print("Destructor called") # Clean up target-specific data here def linear_map(target, bio): print("Mapping I/O request") # Perform mapping here return 0 linear_target = DmTarget(name="linear", version=(1, 0, 0), module="dm_mod") linear_target.set_ctr(linear_ctr) linear_target.set_dtr(linear_dtr) linear_target.set_map(linear_map) info = linear_target.get_info() print(info) (A bunch of stuff elided). I don't care at all about the correctness of this code, because I'm just using it as a roadmap for the real Linux kernel code. The example use case code is an example of something GPT 4o provides that I didn't even know I wanted.
  - Terr_ 2y ago
    LLMs: When the code can be made by an enthusiastic new intern with web-search and copy-paste skills, and no ability to improve under mentorship. :p Tangentially related, a comic on them: https://existentialcomics.com/comic/557 https://existentialcomics.com/comic/557
  - Al-Khwarizmi 2y ago
    It's funny because I'm very happy with the productivity boost from LLMs, but I use them in a way that is pretty much diametrically opposite to yours. I can't think of many situations where I would use them for a problem that I tried to solve and failed - not only because they would probably fail, but in many cases it would even be difficult to know that it failed. I use it for things that are not hard, can be solved by someone without a specialized degree that took the effort to learn some knowledge or skill, but would take too much work to do. And there are a lot of those, even in my highly specialized job.
  - troupo 2y ago
    > That's the frustrating thing. LLMs don't materially reduce the set of problems where I'm running against a wall or have trouble finding information. As you step outside regular Stack Overflow questions for top-3 languages, you run into limitations of these predictive models. There's no "reasoning" behind them. They are still, largely, bullshit machines.
    - KoolKat23 2y ago
      you're both on the wrong wavelength. No one has claimed it is better than an expert human yet. Be glad, for now your jobs are safe, why not use it as a tool to boost your productivity, yes, even though you'll get proportionally less use than others in other perhaps less "expert" jobs.
      - troupo 2y ago
        In order for it to boost productivity it needs to answer more than the regular questions for the top-3 languages on Stackoverflow, no? It often fails even for those questions. If I need to babysit it for every line of code, it's not a productivity boost.
        KoolKat23 2y ago
        Why does it need to answer more than that? You underestimate the opportunity that exists for automation out there. In my own case I've used it to make simple custom browser extensions transcribing PDFs, I don't have the time and wouldn't of made the effort to make the extension myself, the task would of continued to be done manually. It took two hours to make and it works, that's all I need in this case. Perfection is the enemy of good.
        troupo 2y ago
        > Perfection is the enemy of good. Where exactly did I write anything about perfection? For me "AIs" are incapable of producing working code: https://news.ycombinator.com/item?id=41534233 https://news.ycombinator.com/item?id=41534233
mrtesthah 2y ago
It begs the question of whether we can supply a function to be called (e.g., one that compiles and runs code) to evaluate intermediate CoT results
- colordrops 2y ago
  The answer is yes if you are willing to code it. OpenAI supports tool calls. Even if it didn't you could just make multiple calls to their API and submit the result of the code execution yourself.
  - cma 2y ago
    The intermediate CoT results aren't in the API.
    - colordrops 2y ago
      I may be mistaken but I don't believe the first version of the comment I replied to mentioned intermediate CoT results.
- ttul 2y ago
  It seems OpenAI has decided to keep the CoT results a secret. If they were to allow the model to call out to tools to help fill in the CoT steps, then this might reveal what the model is thinking - something they do not want the outside world to know about. I could imagine OpenAI might allow their own vetted tools to be used, but perhaps it will be a while (if ever) before developers are allowed to hook up their own tools. The risks here are substantial. A model fine-tuned to run chain-of-thought that can answer graduate level physics problems at an expert level can probably figure out how to scam your grandma out of her savings too.
  - 8n4vidtmkvmk 2y ago
    It's only a matter of time. When some other company releases the tool, they likely will too.
faangguyindia 2y ago
>The o1-preview model still hallucinates non-existing libraries and functions for me, and is quickly wrong about facts that aren't well-represented on the web. It's the usual string of "You're absolutely correct, and I apologize for the oversight in my previous response. [Let me make another guess.]" After that you switch to Claude Soñnet and after sometime it also gets stuck. Problem with LLM is that they are not aware of libraries. I've fed them library version, using requirements.txt, python version I am using etc... They still make mistakes and try to use methods which do not exist. Where to go from here? At this point I manually pull the library version I am using and go to its docs, I generate a page which uses the this library correctly (then I feed that example into LLM) Using this approach works. Now I just need to automate it so that I don't have to manually find the library, create specific example which uses the methods I need in my code! Directly feeding the docs isn't working well either.
- mediaman 2y ago
  One trick that people are using, when using Cursor and specifically Cursor's compose function, is to dump library docs into a text file in your repo, and then @ that doc file when you're asking it to do something involving that library. That seems to eliminate a lot of the issues, though it's not a seamless experience, and it adds another step of having to put the library docs in a text file. Alternatively, cursor can fetch a web page, so if there's a good page of docs you can bring that in by @ the web page. Eventually, I could imagine LLMs automatically creating library text doc files to include when the LLM is using them to avoid some of these problems. It could also solve some of the issues of their shaky understanding of newer frameworks like SvelteKit.
  - throwup238 2y ago
    Cursor also has the shadow workspace feature [1] that is supposed to send feedback from linting and language servers to the LLM. I'm not sure whether it's enabled in compose yet though. [1] https://www.cursor.com/blog/shadow-workspace https://www.cursor.com/blog/shadow-workspace
shmatt 2y ago
I honestly can’t believe this is the hyped up “strawberry” everyone was claiming is pretty much AGI. Senior employees leaving due to its powers being so extreme I’m in the “probabilistic token generators aren’t intelligence” camp so I don’t actually believe in AGI, but I’ll be honest the never ending rumors / chatter almost got to me Remember, this is the model some media outlet reported recently that is so powerful OAI is considering charging $2k/month for
- firejake308 2y ago
  I mean, considering how many tokens their example prompt consumed, I wouldn't be surprised if it costs ~$2k/month/user to run
  - deleted 2y ago
    [deleted]
- kqr 2y ago
  > probabilistic token generators aren’t intelligence Maybe this has been extensively discussed before, but since I've lived under a rock: which parts of intelligence do you think are not representable as conditional probability distributions?
  - deleted 2y ago
    [deleted]
  - lewhoo 2y ago
    > which parts of intelligence do you think are not representable as conditional probability distributions Maybe I'm wrong here but a lot of our brilliance comes from acting against the statistical consensus. What I mean is, Nicolaus Copernicus probably consumed a lot of knowledge on how the Earth is the center of the universe etc. and probably nothing contradicting that notion. Can a LLM do that ?
    - KoolKat23 2y ago
COAGULOPATH 2y ago
Yes, this only helps multi-step reasoning. The model still has problems with general knowledge and deep facts. There's no way you can "reason" a correct answer to "list the tracklisting of some obscure 1991 demo by a band not on Wikipedia." You either know or you don't. I usually test new models with questions like "what are the levels in [semi-famous PC game from the 90s]?" The release version of GPT-4 could get about 75% correct. o1-preview gets about half correct. o1-mini gets 0% correct. Fair enough. The GPT-4 line aren't meant to be search engines or encyclopedias. This is still a useful update though.
- mattmanser 2y ago
  It's actually much worse than that and you're inadvertently down playing how bad it is. It doesn't even know mildly obsecure facts that are on the internet. For example last night I was trying to do something with C# generics and it confidently told me I could use pattern matching on the type in a switch statwmnt, and threw out some convincing looking code. You can't, it's impossible. It wàa completely wrong. When I told that this, it told me I was right, and proceeded to give me code that was even more wrong. This is an obscure, but well documented, part of the spec. So it's not about facts that aren't on the internet, it's just bad at facts fullstop. What it's good at is facts the internet agrees on. Unless the internet is wrong. Which is not always a good thing with the way the language it uses to speak is so confident. If you want to fuck with AI models as a bunch of code questions on Reddit, GitHub and SO with example code saying 'can I do X'. The answer is no, but chatgpt/codepilot/etc. will start spewing out that nonsense as if it's fact. As for non-proframming, we're about to see the birth of a new SEO movement of tricking AI models to believe your 'facts'.
  - pcdoodle 2y ago
    >>>As for non-proframming, we're about to see the birth of a new SEO movement of tricking AI models to believe your 'facts'. This is kinda crazy to think about.
    - simonw 2y ago
      If you ask Google Gemini right now for the name of the whale in half moon bay harbor it will tell you it’s called Teresa T. That was thanks to my experiment in influencing AI search: https://simonwillison.net/2024/Sep/8/teresa-t-whale-pillar-point/ https://simonwillison.net/2024/Sep/8/teresa-t-whale-pillar-p...
motoboi 2y ago
Just pass a link to a GitHub issue and ask for a response or even a webpage to summarize and will see the beautiful hallucinations it will come up to as the model is not web browsing yet.
_fs 2y ago
o1-preview != o1. In public coding AI comparison tests, results showed 4o scoring around 35%, o1-preview scoring ~50% and o1 scoring ~85%. o1 is not yet released, but has been run through many comparison tests with public results posted.
- arthurcolle 2y ago
  gpt-4 base was never released and this will be the same thing
- barrkel 2y ago
  Don't forget about o1-mini. It seems better than o1-preview for problems that fit it (don't require so much real world knowledge).
- kristianp 2y ago
  Good reminder. Why did OpenAI talk about o1 and not release it? o1-preview must be a stripped down version: cheaper to run somehow?
feralderyl 2y ago
I think this model is a precursor model that is designed for agentic behavior. I expect very soon OpenAI to allow this model tool use that will allow it to verify its code creations and whatever else it claims through use of various tools like a search engine, a virtual machine instance with code execution capabilities, api calling and other advanced tool use.
burtonator 2y ago
I'm honestly confused as to why it is doing this and why it thinks I'm right when I tell it that it is incorrect. I've tried asking it factual information, and it asserts that it's incorrect but it will definitely hallucinate questions like the above. You'd think the reasoning would nail that and most of the chain-of-thought systems I've worked on would have fixed this by asking it if the resulting answer was correct.
MPSimmons 2y ago
The failure is in how you're using it. I don't mean this as a personal attack, but more to shed light on what's happening. A lot of people use LLMs as a search engine. It makes sense - it's basically a lossy compressed database of everything its ever read, and it generates output that is statistically likely - varying degrees of likeliness depending on the temperature, as well as how many times the particular weights your prompt ends up activating. The magic of LLMs, especially one like this that supposedly has advanced reasoning, isn't the existing knowledge in its weights. The magic is that _it knows english_. It knows english at or above a level equal to most fluent speakers, and it also can produce output that is not just a likely output, but is a logical output. It's not _just_ an output engine. It's an engine that outputs. Asking it about nuanced details in the corpus of data it has read won't give you good output unless it read a bunch of it. On the other hand, if you were to paste the entire documentation set to a tool it has never seen and ask it to use the tool in a way to accomplish your goals, THEN this model would be likely to produce useful output, despite the fact that it had never encountered the tool or its documentation before. Don't treat it as a database. Treat it as a naive but intelligent intern. Provide it data, give it a task, and let it surprise you with its output.
- pedrosorio 2y ago
  > It knows english at or above a level equal to most fluent speakers, and it also can produce output that is not just a likely output, but is a logical output This is not an apt description of the system that insists the doctor is the mother of the boy involved in a car accident when elementary understanding of English and very little logic show that answer to be obviously wrong. https://x.com/colin_fraser/status/1834336440819614036 https://x.com/colin_fraser/status/1834336440819614036
  - anon291 2y ago
    Keep in mind that the system always chooses randomly so there is always a possibility it commits to the wrong output. I don't know why openAi won't allow determinism but it doesn't, even with temperature set to zero
rcarr 2y ago
Has anyone tried asking it to generate the libraries/functions that it's hallucinating and seeing if it can do so correctly? And then seeing if it can continue solving the original problem with the new libraries? It'd be absolutely fascinating if it turns out it could do this.
- viraptor 2y ago
  Not for libraries, but functions will sometimes get created if you work with an agent coding loop. If the tests are in the verification step, the code will typically be correct.
  - 8n4vidtmkvmk 2y ago
    I sometimes give it snippets of code and omit helper functions if they seem obvious enough, and it adds its own implementation into the output.
kylebenzle 2y ago
> having no way to assess if what it conjures up from its weights is factual or not. This comment makes no sense in the context of what an LLM is. To even say such a thing demonstates a lack of understandting of the domain. What we are doing here is TEXT COMPLETION, no one EVER said anything about being accurate and "true". We are building models that can complete text, what did you think an LLM was, a "truth machine"?
- panja 2y ago
  I mean of course you're right, but then I question what's the usefulness?
jiggawatts 2y ago
> The o1-preview model still hallucinates non-existing libraries and functions for me Oooh... oohhh!! I just had a thought: By now we're all familiar with the strict JSON output mode capability of these LLMs. That's just a matter of filtering the token probability vector by the output grammar. Only valid tokens are allowed, which guarantees that the output matches the grammar. But... why just data grammars? Why not the equivalent of "tab-complete"? I wonder how hard it would be to hook up the Language Server Protocol (LSP) as seen in Visual Studio code to an AI and have it only emit syntactically valid code! No more hallucinated functions! I mean, sure, the semantics can still be incorrect, but not the syntax.
- loremaster 2y ago
  This would be a big undertaking to get working for just one language+package-manager combination, but would be beautiful if it worked.
- TeMPOraL 2y ago
  I still fail to see the overall problem. Hallucinating non-existing libraries is a good programming practice in many cases: you express your solution in terms of an imaginary API that is convenient for you, and then you replace your API with real functions, and/or implement it in terms of real functions.
fsndz 2y ago
My point of view: this is a real advancement. I've always believed that with the right data allowing the LLM to be trained to imitate reasoning, it's possible to improve its performance. However, this is still pattern matching, and I suspect that this approach may not be very effective for creating true generalization. As a result, once o1 becomes generally available, we will likely notice the persistent hallucinations and faulty reasoning, especially when the problem is sufficiently new or complex, beyond the "reasoning programs" or "reasoning patterns" the model learned during the reinforcement learning phase. https://www.lycee.ai/blog/openai-o1-release-agi-reasoning https://www.lycee.ai/blog/openai-o1-release-agi-reasoning
spaceman_2020 2y ago
I don’t really see this as a massive problem. Its code. If it doesn’t run, you ask it to reconsider, give some more info if necessary, and it usually gets it right. The system doesn’t become useless if it takes 2 tries instead of 1 to get it right Still saves an incredible amount of time vs doing it yourself
- troupo 2y ago
  I haven't found a single instance where it saved me any significant amount of time. In all cases I still had to rewrite the whole thing myself, or abandon endeavor. And a few times the amount of time I spent trying to coax a correct answer out of AI trumped any potential savings I could've had
- latexr 2y ago
  > Its code. If it doesn’t run, you ask it to reconsider It is perfectly possible to have code that runs without errors but gives a wrong answer. And you may not even realise it’s wrong until it bites you in production.
- benterix 2y ago
  While I agree, I saw it abused in this way a lot, in the sense that the code did what it was supposed to do in a given scenario but was obviously flawed in various was so it was just sitting there waiting for a disaster.
deleted 2y ago
[deleted]
timbaboon 2y ago
The best one I got recently was after I pointed out that the method didn’t exist, it proposed another method and said “use this method if it exists” :D
AbstractH24 2y ago
One of the biggest problems with this generation of AI is how people conflate the natural language abilities and the access to what it knows. Both abilities are powerful, but they are very different powers.
HarHarVeryFunny 2y ago
To the extent we've now got the output of the underlying model wrapped in an agent that can evaluate that output, I'd expect it to be able to detect it's own hallucinations some of the time and therefore provide an alternate answer. It's like when an LLM gives you a wrong answer and all it takes is "are you sure?" to get it to generate a different answer. Of course the underlying problem of the model not knowing what it knows or doesn't know persists, so giving it the ability to reflect on what it just blurted out isn't always going to help. It seems the next step is for them to integrate RAG and tool use into this agentic wrapper, which may help in some cases.
empath75 2y ago
You should not be asking it questions that require it to already know detailed information about apis and libraries. It is not good at that, and it will never be good at that. If you need it to write code that uses a particular library or api, include the relevant documentation and examples. It's your right to dismiss it, if you want, but if you want to get some value out of it, you should play to it's strengths and not look for things that it fails at as a gotcha.
markk 2y ago
Stupid question: Why can't models be trained in such a way to rate the authoritativeness of inputs? As a human, I contain a lot of bad information, but I'm aware of the source. I trust my physics textbook over something my nephew thinks.
xyproto 2y ago
Just ask it for things it has seen before on the internet and you're golden. Mixes of ideas, new ideas and precise and clear thinking; not so much.

by layer8 2y ago

kranuck 2y ago

That problems feels somewhat fundamental to saying that these things have any ability to reason at all.

tptacek 2y ago

I've had the opposite experience with some coding samples. After reading Nick Carlini's post, I've gotten into the habit of powering through coding problems with GPT (where previously I'd just laugh and immediately give up) by just presenting it the errors in its code and asking it to fix them. o1 seems to be effectively screening for some of those errors (I assume it's just some, but I've noticed that the o1 things I've done haven't had obvious dumb errors like missing imports, and all my 4o attempts have).

layer8 2y ago
My experience is likely colored by the fact that I tend to turn to LLMs for problems I have trouble solving by myself. I typically don't use them for the low-hanging fruits. That's the frustrating thing. LLMs don't materially reduce the set of problems where I'm running against a wall or have trouble finding information.
- mensetmanusman 2y ago
  LLMs are not for expanding the sphere of human knowledge, but for speeding up auto-correct of higher order processing to help you more quickly reach the shell of the sphere and make progress with your own mind :)
  - tsunamifury 2y ago
    Definitely. When we talk about being skilled in a T shape LLMs are all about spreading your top of T and not making the bottom go deeper.
    - ben_w 2y ago
      Indeed, not much more depth — though even Terence Tao reported useful results from an earlier version, so perhaps the breadth is a depth all of it's own: https://mathstodon.xyz/@tao/110601051375142142 https://mathstodon.xyz/@tao/110601051375142142 I think of it as making the top bar of the T thicker, but yes, you're right, it also spreads it much wider.
    - skydhash 2y ago
      I prefer reading some book. Maybe the LLM was trained on some piece of knowledge not available on the net, but I much prefer the reliability and consistency of a book.
- tptacek 2y ago
  I use LLMs for three things: * To catch passive voice and nominalizations in my writing. * To convert Linux kernel subsystems into Python so I can quickly understand them (I'm a C programmer but everyone reads Python faster). * To write dumb programs using languages and libraries I haven't used much before; for instance, I'm an ActiveRecord person and needed to do some SQLAlchemy stuff today, and GPT 4o (and o1) kept me away from the SQLAlchemy documentation. OpenAI talks about o1 going head to head with PhDs. I could care less. But for the specific problem we're talking about on this subthread: o1 seems materially better.
  - throwup238 2y ago
    > * To convert Linux kernel subsystems into Python so I can quickly understand them (I'm a C programmer but everyone reads Python faster). Do you have an example chat of this output? Sounds interesting. Do you just dump the C source code into the prompt and ask it to convert to Python?
    - tptacek 2y ago
      No, ChatGPT is way cooler than that. It's already read every line of kernel code ever written. I start with a subsystem: the device mapper is a good recent example. I ask things like "explain the linux device mapper. if it was a class in an object-oriented language, what would its interface look like?" and "give me dm_target as a python class". I get stuff like: def linear_ctr(target, argc, argv): print("Constructor called with args:", argc, argv) # Initialize target-specific data here return 0 def linear_dtr(target): print("Destructor called") # Clean up target-specific data here def linear_map(target, bio): print("Mapping I/O request") # Perform mapping here return 0 linear_target = DmTarget(name="linear", version=(1, 0, 0), module="dm_mod") linear_target.set_ctr(linear_ctr) linear_target.set_dtr(linear_dtr) linear_target.set_map(linear_map) info = linear_target.get_info() print(info) (A bunch of stuff elided). I don't care at all about the correctness of this code, because I'm just using it as a roadmap for the real Linux kernel code. The example use case code is an example of something GPT 4o provides that I didn't even know I wanted.
- Terr_ 2y ago
  LLMs: When the code can be made by an enthusiastic new intern with web-search and copy-paste skills, and no ability to improve under mentorship. :p Tangentially related, a comic on them: https://existentialcomics.com/comic/557 https://existentialcomics.com/comic/557
- Al-Khwarizmi 2y ago
  It's funny because I'm very happy with the productivity boost from LLMs, but I use them in a way that is pretty much diametrically opposite to yours. I can't think of many situations where I would use them for a problem that I tried to solve and failed - not only because they would probably fail, but in many cases it would even be difficult to know that it failed. I use it for things that are not hard, can be solved by someone without a specialized degree that took the effort to learn some knowledge or skill, but would take too much work to do. And there are a lot of those, even in my highly specialized job.
- troupo 2y ago
  > That's the frustrating thing. LLMs don't materially reduce the set of problems where I'm running against a wall or have trouble finding information. As you step outside regular Stack Overflow questions for top-3 languages, you run into limitations of these predictive models. There's no "reasoning" behind them. They are still, largely, bullshit machines.
  - KoolKat23 2y ago
    you're both on the wrong wavelength. No one has claimed it is better than an expert human yet. Be glad, for now your jobs are safe, why not use it as a tool to boost your productivity, yes, even though you'll get proportionally less use than others in other perhaps less "expert" jobs.
    - troupo 2y ago
      In order for it to boost productivity it needs to answer more than the regular questions for the top-3 languages on Stackoverflow, no? It often fails even for those questions. If I need to babysit it for every line of code, it's not a productivity boost.
      - KoolKat23 2y ago
        Why does it need to answer more than that? You underestimate the opportunity that exists for automation out there. In my own case I've used it to make simple custom browser extensions transcribing PDFs, I don't have the time and wouldn't of made the effort to make the extension myself, the task would of continued to be done manually. It took two hours to make and it works, that's all I need in this case. Perfection is the enemy of good.
        troupo 2y ago
        > Perfection is the enemy of good. Where exactly did I write anything about perfection? For me "AIs" are incapable of producing working code: https://news.ycombinator.com/item?id=41534233 https://news.ycombinator.com/item?id=41534233

mrtesthah 2y ago

It begs the question of whether we can supply a function to be called (e.g., one that compiles and runs code) to evaluate intermediate CoT results

colordrops 2y ago
The answer is yes if you are willing to code it. OpenAI supports tool calls. Even if it didn't you could just make multiple calls to their API and submit the result of the code execution yourself.
- cma 2y ago
  The intermediate CoT results aren't in the API.
  - colordrops 2y ago
    I may be mistaken but I don't believe the first version of the comment I replied to mentioned intermediate CoT results.
ttul 2y ago
It seems OpenAI has decided to keep the CoT results a secret. If they were to allow the model to call out to tools to help fill in the CoT steps, then this might reveal what the model is thinking - something they do not want the outside world to know about. I could imagine OpenAI might allow their own vetted tools to be used, but perhaps it will be a while (if ever) before developers are allowed to hook up their own tools. The risks here are substantial. A model fine-tuned to run chain-of-thought that can answer graduate level physics problems at an expert level can probably figure out how to scam your grandma out of her savings too.
- 8n4vidtmkvmk 2y ago
  It's only a matter of time. When some other company releases the tool, they likely will too.

faangguyindia 2y ago

>The o1-preview model still hallucinates non-existing libraries and functions for me, and is quickly wrong about facts that aren't well-represented on the web. It's the usual string of "You're absolutely correct, and I apologize for the oversight in my previous response. [Let me make another guess.]" After that you switch to Claude Soñnet and after sometime it also gets stuck. Problem with LLM is that they are not aware of libraries. I've fed them library version, using requirements.txt, python version I am using etc... They still make mistakes and try to use methods which do not exist. Where to go from here? At this point I manually pull the library version I am using and go to its docs, I generate a page which uses the this library correctly (then I feed that example into LLM) Using this approach works. Now I just need to automate it so that I don't have to manually find the library, create specific example which uses the methods I need in my code! Directly feeding the docs isn't working well either.

mediaman 2y ago
One trick that people are using, when using Cursor and specifically Cursor's compose function, is to dump library docs into a text file in your repo, and then @ that doc file when you're asking it to do something involving that library. That seems to eliminate a lot of the issues, though it's not a seamless experience, and it adds another step of having to put the library docs in a text file. Alternatively, cursor can fetch a web page, so if there's a good page of docs you can bring that in by @ the web page. Eventually, I could imagine LLMs automatically creating library text doc files to include when the LLM is using them to avoid some of these problems. It could also solve some of the issues of their shaky understanding of newer frameworks like SvelteKit.
- throwup238 2y ago
  Cursor also has the shadow workspace feature [1] that is supposed to send feedback from linting and language servers to the LLM. I'm not sure whether it's enabled in compose yet though. [1] https://www.cursor.com/blog/shadow-workspace https://www.cursor.com/blog/shadow-workspace

shmatt 2y ago

I honestly can’t believe this is the hyped up “strawberry” everyone was claiming is pretty much AGI. Senior employees leaving due to its powers being so extreme I’m in the “probabilistic token generators aren’t intelligence” camp so I don’t actually believe in AGI, but I’ll be honest the never ending rumors / chatter almost got to me Remember, this is the model some media outlet reported recently that is so powerful OAI is considering charging $2k/month for

firejake308 2y ago
I mean, considering how many tokens their example prompt consumed, I wouldn't be surprised if it costs ~$2k/month/user to run
- deleted 2y ago
  [deleted]
kqr 2y ago
> probabilistic token generators aren’t intelligence Maybe this has been extensively discussed before, but since I've lived under a rock: which parts of intelligence do you think are not representable as conditional probability distributions?
- deleted 2y ago
  [deleted]
- lewhoo 2y ago
  > which parts of intelligence do you think are not representable as conditional probability distributions Maybe I'm wrong here but a lot of our brilliance comes from acting against the statistical consensus. What I mean is, Nicolaus Copernicus probably consumed a lot of knowledge on how the Earth is the center of the universe etc. and probably nothing contradicting that notion. Can a LLM do that ?
  - KoolKat23 2y ago

COAGULOPATH 2y ago

Yes, this only helps multi-step reasoning. The model still has problems with general knowledge and deep facts. There's no way you can "reason" a correct answer to "list the tracklisting of some obscure 1991 demo by a band not on Wikipedia." You either know or you don't. I usually test new models with questions like "what are the levels in [semi-famous PC game from the 90s]?" The release version of GPT-4 could get about 75% correct. o1-preview gets about half correct. o1-mini gets 0% correct. Fair enough. The GPT-4 line aren't meant to be search engines or encyclopedias. This is still a useful update though.

mattmanser 2y ago
It's actually much worse than that and you're inadvertently down playing how bad it is. It doesn't even know mildly obsecure facts that are on the internet. For example last night I was trying to do something with C# generics and it confidently told me I could use pattern matching on the type in a switch statwmnt, and threw out some convincing looking code. You can't, it's impossible. It wàa completely wrong. When I told that this, it told me I was right, and proceeded to give me code that was even more wrong. This is an obscure, but well documented, part of the spec. So it's not about facts that aren't on the internet, it's just bad at facts fullstop. What it's good at is facts the internet agrees on. Unless the internet is wrong. Which is not always a good thing with the way the language it uses to speak is so confident. If you want to fuck with AI models as a bunch of code questions on Reddit, GitHub and SO with example code saying 'can I do X'. The answer is no, but chatgpt/codepilot/etc. will start spewing out that nonsense as if it's fact. As for non-proframming, we're about to see the birth of a new SEO movement of tricking AI models to believe your 'facts'.
- pcdoodle 2y ago
  >>>As for non-proframming, we're about to see the birth of a new SEO movement of tricking AI models to believe your 'facts'. This is kinda crazy to think about.
  - simonw 2y ago
    If you ask Google Gemini right now for the name of the whale in half moon bay harbor it will tell you it’s called Teresa T. That was thanks to my experiment in influencing AI search: https://simonwillison.net/2024/Sep/8/teresa-t-whale-pillar-point/ https://simonwillison.net/2024/Sep/8/teresa-t-whale-pillar-p...

motoboi 2y ago

Just pass a link to a GitHub issue and ask for a response or even a webpage to summarize and will see the beautiful hallucinations it will come up to as the model is not web browsing yet.

_fs 2y ago

o1-preview != o1. In public coding AI comparison tests, results showed 4o scoring around 35%, o1-preview scoring ~50% and o1 scoring ~85%. o1 is not yet released, but has been run through many comparison tests with public results posted.

arthurcolle 2y ago
gpt-4 base was never released and this will be the same thing
barrkel 2y ago
Don't forget about o1-mini. It seems better than o1-preview for problems that fit it (don't require so much real world knowledge).
kristianp 2y ago
Good reminder. Why did OpenAI talk about o1 and not release it? o1-preview must be a stripped down version: cheaper to run somehow?

feralderyl 2y ago

I think this model is a precursor model that is designed for agentic behavior. I expect very soon OpenAI to allow this model tool use that will allow it to verify its code creations and whatever else it claims through use of various tools like a search engine, a virtual machine instance with code execution capabilities, api calling and other advanced tool use.

burtonator 2y ago

I'm honestly confused as to why it is doing this and why it thinks I'm right when I tell it that it is incorrect. I've tried asking it factual information, and it asserts that it's incorrect but it will definitely hallucinate questions like the above. You'd think the reasoning would nail that and most of the chain-of-thought systems I've worked on would have fixed this by asking it if the resulting answer was correct.

MPSimmons 2y ago

The failure is in how you're using it. I don't mean this as a personal attack, but more to shed light on what's happening. A lot of people use LLMs as a search engine. It makes sense - it's basically a lossy compressed database of everything its ever read, and it generates output that is statistically likely - varying degrees of likeliness depending on the temperature, as well as how many times the particular weights your prompt ends up activating. The magic of LLMs, especially one like this that supposedly has advanced reasoning, isn't the existing knowledge in its weights. The magic is that _it knows english_. It knows english at or above a level equal to most fluent speakers, and it also can produce output that is not just a likely output, but is a logical output. It's not _just_ an output engine. It's an engine that outputs. Asking it about nuanced details in the corpus of data it has read won't give you good output unless it read a bunch of it. On the other hand, if you were to paste the entire documentation set to a tool it has never seen and ask it to use the tool in a way to accomplish your goals, THEN this model would be likely to produce useful output, despite the fact that it had never encountered the tool or its documentation before. Don't treat it as a database. Treat it as a naive but intelligent intern. Provide it data, give it a task, and let it surprise you with its output.

pedrosorio 2y ago
> It knows english at or above a level equal to most fluent speakers, and it also can produce output that is not just a likely output, but is a logical output This is not an apt description of the system that insists the doctor is the mother of the boy involved in a car accident when elementary understanding of English and very little logic show that answer to be obviously wrong. https://x.com/colin_fraser/status/1834336440819614036 https://x.com/colin_fraser/status/1834336440819614036
- anon291 2y ago
  Keep in mind that the system always chooses randomly so there is always a possibility it commits to the wrong output. I don't know why openAi won't allow determinism but it doesn't, even with temperature set to zero

rcarr 2y ago

Has anyone tried asking it to generate the libraries/functions that it's hallucinating and seeing if it can do so correctly? And then seeing if it can continue solving the original problem with the new libraries? It'd be absolutely fascinating if it turns out it could do this.

viraptor 2y ago
Not for libraries, but functions will sometimes get created if you work with an agent coding loop. If the tests are in the verification step, the code will typically be correct.
- 8n4vidtmkvmk 2y ago
  I sometimes give it snippets of code and omit helper functions if they seem obvious enough, and it adds its own implementation into the output.

kylebenzle 2y ago

> having no way to assess if what it conjures up from its weights is factual or not. This comment makes no sense in the context of what an LLM is. To even say such a thing demonstates a lack of understandting of the domain. What we are doing here is TEXT COMPLETION, no one EVER said anything about being accurate and "true". We are building models that can complete text, what did you think an LLM was, a "truth machine"?

panja 2y ago
I mean of course you're right, but then I question what's the usefulness?

jiggawatts 2y ago

> The o1-preview model still hallucinates non-existing libraries and functions for me Oooh... oohhh!! I just had a thought: By now we're all familiar with the strict JSON output mode capability of these LLMs. That's just a matter of filtering the token probability vector by the output grammar. Only valid tokens are allowed, which guarantees that the output matches the grammar. But... why just data grammars? Why not the equivalent of "tab-complete"? I wonder how hard it would be to hook up the Language Server Protocol (LSP) as seen in Visual Studio code to an AI and have it only emit syntactically valid code! No more hallucinated functions! I mean, sure, the semantics can still be incorrect, but not the syntax.

loremaster 2y ago
This would be a big undertaking to get working for just one language+package-manager combination, but would be beautiful if it worked.
TeMPOraL 2y ago
I still fail to see the overall problem. Hallucinating non-existing libraries is a good programming practice in many cases: you express your solution in terms of an imaginary API that is convenient for you, and then you replace your API with real functions, and/or implement it in terms of real functions.

fsndz 2y ago

My point of view: this is a real advancement. I've always believed that with the right data allowing the LLM to be trained to imitate reasoning, it's possible to improve its performance. However, this is still pattern matching, and I suspect that this approach may not be very effective for creating true generalization. As a result, once o1 becomes generally available, we will likely notice the persistent hallucinations and faulty reasoning, especially when the problem is sufficiently new or complex, beyond the "reasoning programs" or "reasoning patterns" the model learned during the reinforcement learning phase. https://www.lycee.ai/blog/openai-o1-release-agi-reasoning https://www.lycee.ai/blog/openai-o1-release-agi-reasoning

spaceman_2020 2y ago

I don’t really see this as a massive problem. Its code. If it doesn’t run, you ask it to reconsider, give some more info if necessary, and it usually gets it right. The system doesn’t become useless if it takes 2 tries instead of 1 to get it right Still saves an incredible amount of time vs doing it yourself

troupo 2y ago
I haven't found a single instance where it saved me any significant amount of time. In all cases I still had to rewrite the whole thing myself, or abandon endeavor. And a few times the amount of time I spent trying to coax a correct answer out of AI trumped any potential savings I could've had
latexr 2y ago
> Its code. If it doesn’t run, you ask it to reconsider It is perfectly possible to have code that runs without errors but gives a wrong answer. And you may not even realise it’s wrong until it bites you in production.
benterix 2y ago
While I agree, I saw it abused in this way a lot, in the sense that the code did what it was supposed to do in a given scenario but was obviously flawed in various was so it was just sitting there waiting for a disaster.

deleted 2y ago

[deleted]

timbaboon 2y ago

The best one I got recently was after I pointed out that the method didn’t exist, it proposed another method and said “use this method if it exists” :D

AbstractH24 2y ago

One of the biggest problems with this generation of AI is how people conflate the natural language abilities and the access to what it knows. Both abilities are powerful, but they are very different powers.

HarHarVeryFunny 2y ago

To the extent we've now got the output of the underlying model wrapped in an agent that can evaluate that output, I'd expect it to be able to detect it's own hallucinations some of the time and therefore provide an alternate answer. It's like when an LLM gives you a wrong answer and all it takes is "are you sure?" to get it to generate a different answer. Of course the underlying problem of the model not knowing what it knows or doesn't know persists, so giving it the ability to reflect on what it just blurted out isn't always going to help. It seems the next step is for them to integrate RAG and tool use into this agentic wrapper, which may help in some cases.

empath75 2y ago

You should not be asking it questions that require it to already know detailed information about apis and libraries. It is not good at that, and it will never be good at that. If you need it to write code that uses a particular library or api, include the relevant documentation and examples. It's your right to dismiss it, if you want, but if you want to get some value out of it, you should play to it's strengths and not look for things that it fails at as a gotcha.

markk 2y ago

Stupid question: Why can't models be trained in such a way to rate the authoritativeness of inputs? As a human, I contain a lot of bad information, but I'm aware of the source. I trust my physics textbook over something my nephew thinks.

xyproto 2y ago

Just ask it for things it has seen before on the internet and you're golden. Mixes of ideas, new ideas and precise and clear thinking; not so much.