16 ms·

The failure is in how you're using it. I don't mean this as a personal attack, but more to shed light on what's happening. A lot of people use LLMs as a search

The failure is in how you're using it. I don't mean this as a personal attack, but more to shed light on what's happening. A lot of people use LLMs as a search engine. It makes sense - it's basically a lossy compressed database of everything its ever read, and it generates output that is statistically likely - varying degrees of likeliness depending on the temperature, as well as how many times the particular weights your prompt ends up activating. The magic of LLMs, especially one like this that supposedly has advanced reasoning, isn't the existing knowledge in its weights. The magic is that _it knows english_. It knows english at or above a level equal to most fluent speakers, and it also can produce output that is not just a likely output, but is a logical output. It's not _just_ an output engine. It's an engine that outputs. Asking it about nuanced details in the corpus of data it has read won't give you good output unless it read a bunch of it. On the other hand, if you were to paste the entire documentation set to a tool it has never seen and ask it to use the tool in a way to accomplish your goals, THEN this model would be likely to produce useful output, despite the fact that it had never encountered the tool or its documentation before. Don't treat it as a database. Treat it as a naive but intelligent intern. Provide it data, give it a task, and let it surprise you with its output.

pedrosorio 2y ago
> It knows english at or above a level equal to most fluent speakers, and it also can produce output that is not just a likely output, but is a logical output This is not an apt description of the system that insists the doctor is the mother of the boy involved in a car accident when elementary understanding of English and very little logic show that answer to be obviously wrong. https://x.com/colin_fraser/status/1834336440819614036 https://x.com/colin_fraser/status/1834336440819614036
- anon291 2y ago
  Keep in mind that the system always chooses randomly so there is always a possibility it commits to the wrong output. I don't know why openAi won't allow determinism but it doesn't, even with temperature set to zero
  - coffeebeqn 2y ago
    Would picking deterministically help through? Then in some cases it’s always 100% wrong
    - jaredsohn 2y ago
      Yes, it is better if for example using it via an API to classify. Deterministic behavior makes it a lot easier to debug the prompt.
  - maeil 2y ago
    Nondeterminism provides an excuse for errors, determinism doesn't. Nondeterminism scores worse with human raters, because it makes output sound even more robotic and less human.
  - roywiggins 2y ago
    Determinism only helps if you always ask the question with exactly the same words. There's no guarantee a slightly rephrased version will give the same answer, so a certain amount of unpredictability is unavoidable anyway. With a deterministic LLM you might find one phrasing that always gets it right and a dozen basically indistinguishable ones that always get it wrong.
    - anon291 2y ago
      My program always asks the same question yes.
- ramraj07 2y ago
  Many of my PhD and post doc colleagues who emigrated from Korea, China and India who didn’t have English as the medium of instruction would struggle with this question. They only recover when you give them a hint. They’re some of the smartest people in general. If you try to stop stumping these models with trick questions and ask it straightforward reasoning systems it is extremely performant (O1 is definitely a step up though not revolutionary in my testing).
  - joedwin 2y ago
    lol, I am neither a PhD nor a postdoc, but I am from India . I could understand the problem.
    - ramraj07 2y ago
      Did you have English as your medium of instruction? If yes, do you see the irony that you also couldn’t read two sentences and see the facts straight?
  - maeil 2y ago
    I live in one of the countries you mentioned and just showed it to one of my friends who's a local who struggles with English. They had no problem concluding that the doctor was the child's dad. Full disclosure, they assumed the doctor was pretending to be the child's dad, which is also a perfectly sound answer.
  - djur 2y ago
    The claim was that "it knows english at or above a level equal to most fluent speakers". If the claim is that it's very good at producing reasonable responses to English text, posing "trick questions" like this would seem to be a fair test.
    - _w1tm 2y ago
      Does fluency in English make someone good at solving trick questions? I usually don’t even bother trying but mostly because trick questions don’t fit my definition of entertaining.
- fragmede 2y ago
  what's weird is it gets it right when I try it. https://chatgpt.com/share/66e3601f-4bec-8009-ac0c-57bfa4f0592c https://chatgpt.com/share/66e3601f-4bec-8009-ac0c-57bfa4f059...
  - brna-2 2y ago
    Yep. correct and correct. https://chatgpt.com/share/66e3de94-bce4-800b-af45-357b95d658ec https://chatgpt.com/share/66e3de94-bce4-800b-af45-357b95d658...
  - brna-2 2y ago
    Waat, got it on second try: This is possible because the doctor is the boy's other parent—his father or, more likely given the surprise, his mother. The riddle plays on the assumption that doctors are typically male, but the doctor in this case is the boy's mother. The twist highlights gender stereotypes, encouraging us to question assumptions about roles in society.
  - latexr 2y ago
    That’s not weird at all, it’s how LLMs work. They statistically arrive at an answer. You can ask it the same question twice in a row in different windows and get opposite answers. That’s completely normal and expected, and also why you can never be sure if you can trust an answer.
  - rtakha 2y ago
    Perhaps OpenAI hot-patches the model for HN complaints: def intercept_hn_complaints(prompt): if is_hn_trick_prompt(prompt): # special_case for known trick questions.
    - fragmede 2y ago
      While that's not impossible, what we know of how the technology works (ie very costly training run followed by cheap inference steps) means that's not feasible, given all the possible variations of the question * is_hn_trick_prompt* would have to cover because there's a near infinite variations on how you'd word the prompt. (Eg The first sentence could be reworded to be "A woman and her son are in a car accident. " to "A woman and her son are in the car when they get into a crash.")
- hmottestad 2y ago
  Reminds me of a trick question about Schrödinger's cat. “I’ve put a dead cat in a box with a poison and an isotope that will trigger the poison at a random point in time. Right now, is the cat dead or alive?” The answer is that the cat is dead, because it was dead to begin with. Understanding this doesn’t mean that you are good at deductive reasoning. It just means that I didn’t manage to trick you. Same goes for an LLM.
  - maeil 2y ago
    There is no "trick" in the linked question, unlike the question you posed. The trick in yours also isn't a logic trick, it's a redirection, like a sleight of hand in a card trick.
    - ryanjshaw 2y ago
      There is a trick. The "How is this possible?" primes the LLM that there is some kind of trick, as that phrase wouldn't exist in the training data outside of riddles and trick questions.
    - hmottestad 2y ago
      The trick in the original question is that it's a twist on the original riddle where the doctor is actually the boys mother. This is a fairly common riddle and I'm sure the LLM has been trained on it.
    - bonoboTP 2y ago
      Yes there is. The trick is that the more common variant of this riddle says that a boy and his father are in the car accident. That variant of the riddle certainly comes up a lot in the training data, which is directly analogous to the Schrödinger case from above where smuggling in the word "dead" is analogous to swapping father to mother in the car accident riddle. I think many here are not aware that the car accident riddle is well known with the father dying where the real solution is indeed that the doctor is the mother.
  - lucubratory 2y ago
    Yeah, I think what a lot of people miss about these sort of gotchas are that most of them were invented explicitly to gotcha humans, who regularly get got by them. This is not a failure mode unique to LLMs.
- achow 2y ago
  What I'm not able to comprehend is why people are not seeing the answer as brilliant! Any ordinary mortal (like me) would have jumped to the conclusion that answer is "Father" and would have walked away patting on my back, without realising that I was biased by statistics. Whereas o1, at the very outset smelled out that it is a riddle - why would anyone out of blue ask such question. So, it started its chain of thought with "Interpreting the riddle" (smart!). In my book that is the difference between me and people who are very smart and are generally able to navigate the world better (cracking interviews or navigating internal politics in a corporate).
  - geysersam 2y ago
    Come on. Of course chatgpt has read that riddle and the answer 1000 times already.
    - ryanjshaw 2y ago
      Why does it exist 1000 times in the training if there isn't some trick to it, i.e. some subset of humans had to have answered it incorrectly for the meme to replicate that extensively in our collective knowledge. And remember the LLM has already read a billion other things, and now needs to figure out - is this one of them tricky situations, or the straightforward ones? It also has to realize all the humans on forums and facebook answering the problem incorrectly are bad data. Might seem simple to you, but it's not.
    - accountnum 2y ago
      It hasn't read that riddle because it is a modified version. The model would in fact solve this trivially if it _didn't_ see the original in its training. That's the entire trick.
      - geysersam 2y ago
        Sure but the parent was praising the model for recognizing that it was a riddle in the first place: > Whereas o1, at the very outset smelled out that it is a riddle That doesn't seem very impressive since it's (an adaptation of) a famous riddle The fact that it also gets it wrong after reasoning about it for a long time doesn't make it better of course
- ryanjshaw 2y ago
  1. It didn't insist anything. It got the semi-correct answer when I tried [1]; note it's a preview model, and it's not a perfect product. (a) Sometimes things are useful even when imperfect e.g. search engines. (b) People make reasoning mistakes too, and I make dumb ones of the sort presented all the time despite being fluent in English; we deal with it! I'm not sure why there's an expectation that the model is perfect when the source data - human output - is not perfect. In my day-to-day work and non-work conversations it's a dialogue - a back and forth until we figure things out. I've never known anybody to get everything perfectly correct the first time, it's so puzzling when I read people complaining that LLMs should somehow be different. 2. There is a recent trend where sex/gender/pronouns are not aligned and the output correctly identifies this particular gotcha. [1] I say semi-correct because it states the doctor is the "biological" father, which is an uncorroborated statement. https://chatgpt.com/share/66e3f04e-cd98-8008-aaf9-9ca933892f07 https://chatgpt.com/share/66e3f04e-cd98-8008-aaf9-9ca933892f...
- bonoboTP 2y ago
  This illustrates a different point. This is a variation on a well known riddle that definitely comes up in the training corpus many times. In the original riddle a father and his son die in the car accident and the idea of the original riddle is that people will be confused how the boy can be the doctor's son if the boy's father just died, not realizing that women can be doctors too and so the doctor is the boy's mother. The original riddle is aimed to highlight people's gender stereotype assumptions. Now, since the model was trained on this, it immediately recognizes the riddle and answers according to the much more common variant. I agree that this is a limitation and a weakness. But it's important to understand that the model knows the original riddle well, so this is highlighting a problem with rote memorization/retrieval in LLMs. But this (tricky twists in well-known riddles that are in the corpus) is a separate thing from answering novel questions. It can also be seen as a form of hypercorrection.
  - ImHereToVote 2y ago
    My codebases are riddled with these gotchas. For instance, I sometimes write Python for the Blender rendering engine. This requires highly non-idiomatic Python. Whenever something complex comes up, LLM's just degenerate to cookie cutter basic bitch Python code. There is simply no "there" there. They are very useful to help you reason about unfamiliar codebases though.
    - bonoboTP 2y ago
      For me the best coding use case is getting up to speed in an unfamiliar library or usage. I describe the thing I want and get a good starting point and often the cookie-cutter way is good enough. The pre-LLM alternative would be to search for tutorials but they will talk about some slightly different problem with different goals etc then you have to piece it together, and the tutorial assumes you already know a bunch of things like how to initialize stuff and skips the boilerplate and so on. Now sure, actually working through it will give a deeper understanding that might come handy at a later point, but sometimes the thing is really a one-off and not an important point. Like as an AI researcher I sometimes want to draft up a quick demo website, or throw together a quick Qt GUI prototype or a Blender script or use some arcane optimization library or write a SWIG or a Cython wrapper around a C/C++ library to access it in Python, or how to stuff with Lustre, or the XFS filesystem or whatever. Any number of small things where, sure, I could open the manual, do some trial and error, read stack overflow, read blogs and forums, OR I could just use an LLM, use my background knowledge to judge whether it looks reasonable, then verify it, use the now obtained key terms to google more effectively etc. You can't just blindly copy-paste it and you have to think critically and remain in the driver seat. But it's an effective tool if you know how and when to use it.
- deleted 2y ago
  [deleted]
- KoolKat23 2y ago
  I'm noticing a strange common theme in all these riddles, it's being asked and getting wrong. They're all badly worded questions. The model knows something is up and reads into it too much. In this case it's tautology, you would usually say "a mother and her son...". I think it may answer correctly if you start off asking "Please solve the below riddle:" There was another example yesterday which it solved correctly after this addition.(In that case the point of views were all mixed up, it only worked as a riddle).
  - TeMPOraL 2y ago
    Yup. The models fail on gotcha questions asked without warning, especially when evaluated on the first snap answer. Much like approximately all humans.
    - Jensson 2y ago
      > especially when evaluated on the first snap answer The whole point of o1 is that it wasn't "the first snap answer", it wrote half a page internally before giving the same wrong answer.
      - grey-area 2y ago
        Is that really its internal 'chain of thought' or is it a post-hoc justification generated afterward? Do LLMs have a chain of thought like this at all or are they just convincing at mimicking what a human might say if asked for a justification for an opinion?
        KoolKat23 2y ago
        Its slightly more strange than this as both are true. It's already baked in the model but chain of thought does improve reasoning, you only have to look at maths problems. A short guess would be wrong but it would get it correct if asked to break it down and reason (harder to see nowadays as it has access to calculators).
  - bnralt 2y ago
- empath75 2y ago
  The reason why that question is a famous question is that _many humans get it wrong_.
tjoff 2y ago
And how much data can you give it? I'm not up to date with these things because I haven't found them useful. But with what you said, and previous limitations in how much data they can retain essentially makes them pretty darn useless for that task. Great learning tool on common subjects you don't know, such as learning a new programming-language. Also great for inspiration etc. But that's pretty much it? Don't get me wrong, that is mindblowingly impressive but at the same time, for the tasks in front of me it has just been a distracting toy wasting my time.
- ben_w 2y ago
  > And how much data can you give it? 128,000 tokens, which is about the same as a decent sized book. Their other models can also be fine-tuned, which is kinda unbounded but also has scaling issues so presumably "a significant percentage of the training set" before diminishing returns.
- pimeys 2y ago
  It is great for proof-reading text if you are not a native English speaker. Things like removing passive voice. Just give it your text and you get a corrected version out. Use a cli tool to automate this from the cli. Ollama for local models, llm for openai.
- MPSimmons 2y ago
  >And how much data can you give it? Well, theoretically you can give it up to the context size minus 4k tokens, because the maximum it can output is 4k. In practice, though, its ability to effectively recall information in the prompt drops off. Some people have studied this a bit - here's one such person: https://gritdaily.com/impact-prompt-length-llm-performance/ https://gritdaily.com/impact-prompt-length-llm-performance/
  - jaredsohn 2y ago
    You should be able to provide more data than that in the input if the output doesn't use the full 4k tokens. So limit is context_size minus expected length of output.
- 2y ago
kylebenzle 2y ago
Perfectly well put! We should change the name from "AI" (which it is not) to something like, "lossy compressed databases".
- gsinclair 2y ago
  That abbreviates to LCD. If we could make it LSD somehow, that would help to explain the hallucinations.
  - MikeTheGreat 2y ago
    Lossy Stochastic Database?
- melagonster 2y ago
  If they use this name, they just say that they violate the copyright of all training data.
_sys49152 2y ago
ive been doing exactly this for bout a year now. feed it words data, give it a task. get better words back. i sneak in a benchmark opening of data every time i start a new chat - so right off the bat i can see in its response whether this chat session is gonna be on point or if we are going off into wacky world, which saves me time as i can just terminate and try starting another chat. chatgpt is fickle daily. most days its on point. some days its wearing a bicycle helmet and licking windows. kinda sucks i cant just zone out and daydream while working. gotta be checking replies for when the wheels fall off the convo.
- ruthmarx 2y ago
  > i sneak in a benchmark opening of data every time i start a new chat - so right off the bat i can see in its response whether this chat session is gonna be on point or if we are going off into wacky world, which saves me time as i can just terminate and try starting another chat. I don't think it works like that...
williamdclt 2y ago
> Treat it as a naive but intelligent intern That’s the problem: it’s a _terrible_ intern. A good intern will ask clarifying questions, tell me “I don’t know” or “I’m not sure I did it right”. LLMs do none of that, they will take whatever you ask and give a reasonable-sounding output that might be anything between brilliant and nonsense. With an intern, I don’t need to measure how good my prompting is, we’ll usually interact to arrive to a common understanding. With a LLM, I need to put a huge amount of thought into the prompt and have no idea whether the LLM understood what I’m asking and if it’s able to do it.
- jappgar 2y ago
  They've explicitly been trained/system-prompted to act that way. Because that's what the marketing teams at these AI companies want to sell. It's easy to override this though by asking the LLM to act as if it were less-confident, more hesitant, paranoid etc. You'll be fighting uphill against the alignment(marketing) team the whole time though, so ymmv.
- kranuck 2y ago
  > have no idea whether the LLM understood what I’m asking That's easy. The answer is it doesn't. It has no understanding of anything it does. > if it’s able to do it This is the hard part.
- mercer 2y ago
  Can I have some of those sorts of interns?
- arthurcolle 2y ago
  many many teams are actively building SOTA systems to do this in ways previously unimagined. you can enqueue tasks and do whatever you want. I gotta say as a current gen LLM programmer person, I can completely appreciate how bad they are now - I recently tweeted about how I "swore off" AI tools but like... there are many ways to bootstrap very powerful software or ML systems around or inside these existing models that can blow away existing commercial implementations in surprising ways
gmerc 2y ago
Interns are cheaper than o1-preview
- icrbow 2y ago
  Not for long.
croes 2y ago
Intelligent? Just ask ChatGPT How many Rs are in strawberry?
- fragmede 2y ago
  https://chatgpt.com/share/66e3f9e1-2cb4-8009-83ce-090068b16335 https://chatgpt.com/share/66e3f9e1-2cb4-8009-83ce-090068b163... Keep up, that was last week's gotcha, with the old model.
  - croes 2y ago
    My point is the previous "intelligent" failed at simple task, the new one will also fail on simple tasks. That's ok for humans but not for machines.
    - K0balt 2y ago
      ‘That's ok for humans but not for machines.’ This is a really interesting bias. I mean, I understand, I feel that way too… but if you think about it, it might be telling us something about intelligence itself. We want to make machines that act more like humans: we did that, and we are now upset that they are just as flaky and unreliable as drunk uncle bob. I have encountered plenty of people that aren’t as good at being accurate or even as interesting to talk to as a 70b model. Sure, LLMs make mistakes most humans would not, but humans also make mistakes most LLMs would not. (I am not trying to equate humans and LLMs, just to be clear) (also, why isn’t equivelate a word?) It turns out we want machines that are extremely reliable, cooperative , responsible and knowledgeable. We yearn to be obsolete. We want machines that are better than us. The definition of AGI has drifted from meaning. “able to broadly solve problems the (class of which) system designers did not anticipate” to “must be usefully intelligent at the same level as a bright, well educated person”. Where along the line did we suddenly forget that dog level intelligence was a far out of reach goal until suddenly it wasn’t?
  - ziml77 2y ago
    There's randomness involved in generating responses. It can also give the wrong answer still: https://bsky.app/profile/did:plc:qc6xzgctorfsm35w6i3vdebx/post/3l3ycebgjoz2v https://bsky.app/profile/did:plc:qc6xzgctorfsm35w6i3vdebx/po...
lyu07282 2y ago
> The magic is that _it knows english_. I couldn't agree more, this is exactly the strength of LLMs that what we should focus on. If you can make your problem fit into this paradigm, LLMs work fantastic. Hallucinations come from that massive "lossy compressed database", but you should consider that part as more like the background noise that taught the model to speak English, and the syntax of programming languages, instead of the source of the knowledge to respond with. Stop anthropomorphizing LLMs, play to it's strengths instead. In other words it might hallucinate a API but it will rarely, if ever, make a syntax error. Once you realize that, it becomes a much more useful tool.
re-thc 2y ago
> Treat it as a naive but intelligent intern. That's the crux of the problem. Why and who would treat it as an intern? It might cost you more in explaining and dealing with it than not using it. The purpose of an intern is to grow the intern. If this intern is static and will always be at the same level, why bother? If you had to feed and prep it every time, you might as well hire a senior.
flanked-evergl 2y ago
> The failure is in how you're using it. People, for the most part, know what they know and don't know. I am not uncertain that the distance between the earth and the sun varies, but I'm certain that I don't know the distance from the earth to the sun, at least not with better precision than about a light week. This is going to have to be fixed somehow to progress past where we are now with LLMs. Maybe expecting an LLM to have this capability is wrong, perhaps it can never have this capability, but expecting this capability is not wrong, and LLM vendors have somewhat implied that their models have this capability by saying they won't hallucinate, or that they have reduced hallucinations.
- vingt_regards 2y ago
  > the distance from the earth to the sun, at least not with better precision than about a light week The sun is eight light minutes away.
  - flanked-evergl 2y ago
    Thanks, I was not sure if it was light hours or minutes away, but I knew for sure it's not light weeks (emphasis on plural here) away. I will probably forget again in a couple of years.
- arb_ 2y ago
  Empirically, they have reduced hallucinations. Where do OpenAI / Anthropic claim that their models won't hallucinate?
  - flanked-evergl 2y ago
    One example: https://www.theverge.com/2024/3/28/24114664/microsoft-safety-ai-prompt-injections-hallucinations-azure https://www.theverge.com/2024/3/28/24114664/microsoft-safety... > Three features: Prompt Shields, which blocks prompt injections or malicious prompts from external documents that instruct models to go against their training; Groundedness Detection, which finds and blocks hallucinations; and safety evaluations, which assess model vulnerabilities, are now available in preview on Azure AI.
tluyben2 2y ago
This model is, thankfully, far more susceptible for longer and elaborate explanation as input. The rest (4,4o,Sonnet) seem to struggle with comprehensive explanation; this one seems to perform better with a spec like input.
KoolKat23 2y ago
This is a great description.
EagnaIonat 2y ago
> Treat it as a naive but intelligent intern. You are falling into the trap that everyone does. In anthropomorphising it. It doesn't understand anything you say. It just statistically knows what a likely response would be. Treat it as text completion and you can get more accurate answers.
- TeMPOraL 2y ago
  > You are falling into the trap that everyone does. In anthropomorphising it. It doesn't understand anything you say. And an intern does? Anthropomorphising LLMs isn't entirely incorrect: they're trained to complete text like a human would, in completely general setting, so by anthropomorphising them you're aligning your expectations with the models' training goals.
- MPSimmons 2y ago
  Oh no, I'm well aware that it's a big file full of numbers. But when you chat with it, you interact with it as though it were a person so you are necessarily anthropomorphizing it, and so you get to pick the style of the interaction. (In truth, I actually treat it in my mind like it's the Enterprise computer and I'm Beverly Crusher in "Remember Me")
deleted 2y ago
[deleted]
layer8 2y ago
> Treat it as a naive but intelligent intern. Provide it data, give it a task, and let it surprise you with its output. Well, I am a naive but intelligent intern (well, senior developer). So in this framing, the LLM can’t do more than I can already do by myself, and thus far it’s very hit or miss if I actually save time, having to provide all the context and requirements, and having to double-check the results. With interns, this at least improves over time, as they become more knowledgeable, more familiar with the context, and become more autonomous and dependable. Language-related tasks are indeed the most practical. I often use it to brainstorm how to name things.
- fragmede 2y ago
  Ooh yeah it's great for bouncing ideas on what to name things off of. You can give it something's function and a backstory and it'll come up with a list of somethings for you to pick and choose from.
- acureau 2y ago
  I've recently started using an LLM to choose the best release of shows using data scraped from several trackers. I give it hard requirements and flexible preferences. It's not that I couldn't do this, it's that I don't want to do this on the scale of multiple thousand shows. The "magic" here is that releases don't all follow the same naming conventions, they're an unstructured dump of details. The LLM is simultaneously extracting the important details, and flexibly deciding the closest match to my request. The prompt is maybe two paragraphs and took me an hour to hone.
acedTrex 2y ago
> Treat it as a naive but intelligent intern So mostly useless then?
__loam 2y ago
It doesn't know anything. Stop anthropomorphizing the model. It's predictive text and no the brain isn't also predictive text.
usaar333 2y ago
> On the other hand, if you were to paste the entire documentation set to a tool it has never seen and ask it to use the tool in a way to accomplish your goals, THEN this model would be likely to produce useful output, despite the fact that it had never encountered the tool or its documentation before. There's not much evidence of that. It only marginally improved on instruction following (see livebench.ai) and it's score as a swe-bench agent is barely above gpt-4o (model card). It gets really hard problems better, but it's unclear that matters all that much. > A lot of people use LLMs as a search engine. Except this is where LLMs are so powerful. A sort of reasoning search engine. They memorized the entire Internet and can pattern match it to my query.
nsagent 2y ago
Sorry, but that does not seem to be the case. A friend of mine who runs a long context benchmark on understanding novels [1] just ran an eval and o1 seemed to improve by 2.9% over GPT-4o (the result isn't on the website yet). It's great that there is an improvement, but it isn't drastic by any stretch. Additionally, since we cannot see the raw reasoning it's basing the answers off of, it's hard to attribute this increase to their complicated approach as opposed to just cleaner higher quality data. EDIT: Note this was run over a dataset of short stories rather than the novels since the API errors out with very long contexts like novels. [1]: https://novelchallenge.github.io/ https://novelchallenge.github.io/
- boomchinolo78 2y ago
  It's a good rebranding. It was getting ridiculous 3.5, 4, 4.5,
b33j0r 2y ago
Yeah except. I’m priming it with things like curated docs from bevy latest, using the tricks, and testing context limits. It’s still changing things to be several versions old from its innate kb pattern-matching or whatever you want to call it. I find that pretty disappointing. Just like copilot and gpt4, it’s changing `add_systems(Startup, system)` to `add_startup_system(system.sytem())` and other pre-schedule/fanciful APIs—things it should have in context. I agree with your approach to LLMs, but unfortunately “it’s still doing that thing.” PS: and by the time I’d done those experiments, I ran out of preview, resets 5 days from now. D’oh
golergka 2y ago
> A lot of people use LLMs as a search engine. GPT-4o is wonderful as a search engine if you tell it to google things before answering (even though it uses bing).
Salgat 2y ago
Except that it sometimes does do those tasks well. The danger in an LLM isn't that it sometimes hallucinates, the danger is that you need to be sufficiently competent to know when it hallucinates in order to fully take advantage of it, otherwise you have to fallback to double checking every single thing it tells you.
petesergeant 2y ago
This is demonstrably wrong, because you can just add "is this real" to a response and it generally knows if it made it up or not. Not every time, but I find it works 95% of the time. Given that, this is exactly a step I'd hope an advanced model was doing behind the scenes.
bsenftner 2y ago
> Treat it as a naive but intelligent intern. I've found an amazing amount of success with a three step prompting method that appears to create incredibly deep subject matter experts who then collaborate with the user directly. 1) Tell the LLM that it is a method actor, 2) Tell the method actor they are playing the role of a subject matter expert, 3) At each step, 1 and 2, use the technical language of that type of expert; method actors have their own technical terminology, use it when describing the characteristics of the method actor, and likewise use the scientific/programming/whatever technical jargon of the subject matter expert your method actor is playing. Then, in the system prompt or whatever logical wrapper the LLM operates through for the user, instruct the "method actor" like you are the film director trying to get your subject matter expert performance out of them. I offer this because I've found it works very well. It's all about crafting the context in which the LLM operates, and this appears to cause the subject matter expert to be deeper, more useful, smarter.
jrflowers 2y ago
> The failure is in how you're using it This isn’t true because, as you can read in the first sentence of the post you’re responding to, GP did give it a task like you recommend here > Provide it data, give it a task, and let it surprise you with its output. And it fails the task. Specifically it fails it by hallucinating important parts of accomplishing it. > hallucinates non-existing libraries and functions This post only makes sense if your advice to “let it surprise you with its output” is mandatory, like you’re using it wrong if you do not make yourself feel impressed by it.

pedrosorio 2y ago

> It knows english at or above a level equal to most fluent speakers, and it also can produce output that is not just a likely output, but is a logical output This is not an apt description of the system that insists the doctor is the mother of the boy involved in a car accident when elementary understanding of English and very little logic show that answer to be obviously wrong. https://x.com/colin_fraser/status/1834336440819614036 https://x.com/colin_fraser/status/1834336440819614036

anon291 2y ago
Keep in mind that the system always chooses randomly so there is always a possibility it commits to the wrong output. I don't know why openAi won't allow determinism but it doesn't, even with temperature set to zero
- coffeebeqn 2y ago
  Would picking deterministically help through? Then in some cases it’s always 100% wrong
  - jaredsohn 2y ago
    Yes, it is better if for example using it via an API to classify. Deterministic behavior makes it a lot easier to debug the prompt.
- maeil 2y ago
  Nondeterminism provides an excuse for errors, determinism doesn't. Nondeterminism scores worse with human raters, because it makes output sound even more robotic and less human.
- roywiggins 2y ago
  Determinism only helps if you always ask the question with exactly the same words. There's no guarantee a slightly rephrased version will give the same answer, so a certain amount of unpredictability is unavoidable anyway. With a deterministic LLM you might find one phrasing that always gets it right and a dozen basically indistinguishable ones that always get it wrong.
  - anon291 2y ago
    My program always asks the same question yes.
ramraj07 2y ago
Many of my PhD and post doc colleagues who emigrated from Korea, China and India who didn’t have English as the medium of instruction would struggle with this question. They only recover when you give them a hint. They’re some of the smartest people in general. If you try to stop stumping these models with trick questions and ask it straightforward reasoning systems it is extremely performant (O1 is definitely a step up though not revolutionary in my testing).
- joedwin 2y ago
  lol, I am neither a PhD nor a postdoc, but I am from India . I could understand the problem.
  - ramraj07 2y ago
    Did you have English as your medium of instruction? If yes, do you see the irony that you also couldn’t read two sentences and see the facts straight?
- maeil 2y ago
  I live in one of the countries you mentioned and just showed it to one of my friends who's a local who struggles with English. They had no problem concluding that the doctor was the child's dad. Full disclosure, they assumed the doctor was pretending to be the child's dad, which is also a perfectly sound answer.
- djur 2y ago
  The claim was that "it knows english at or above a level equal to most fluent speakers". If the claim is that it's very good at producing reasonable responses to English text, posing "trick questions" like this would seem to be a fair test.
  - _w1tm 2y ago
    Does fluency in English make someone good at solving trick questions? I usually don’t even bother trying but mostly because trick questions don’t fit my definition of entertaining.
fragmede 2y ago
what's weird is it gets it right when I try it. https://chatgpt.com/share/66e3601f-4bec-8009-ac0c-57bfa4f0592c https://chatgpt.com/share/66e3601f-4bec-8009-ac0c-57bfa4f059...
- brna-2 2y ago
  Yep. correct and correct. https://chatgpt.com/share/66e3de94-bce4-800b-af45-357b95d658ec https://chatgpt.com/share/66e3de94-bce4-800b-af45-357b95d658...
- brna-2 2y ago
  Waat, got it on second try: This is possible because the doctor is the boy's other parent—his father or, more likely given the surprise, his mother. The riddle plays on the assumption that doctors are typically male, but the doctor in this case is the boy's mother. The twist highlights gender stereotypes, encouraging us to question assumptions about roles in society.
- latexr 2y ago
  That’s not weird at all, it’s how LLMs work. They statistically arrive at an answer. You can ask it the same question twice in a row in different windows and get opposite answers. That’s completely normal and expected, and also why you can never be sure if you can trust an answer.
- rtakha 2y ago
  Perhaps OpenAI hot-patches the model for HN complaints: def intercept_hn_complaints(prompt): if is_hn_trick_prompt(prompt): # special_case for known trick questions.
  - fragmede 2y ago
    While that's not impossible, what we know of how the technology works (ie very costly training run followed by cheap inference steps) means that's not feasible, given all the possible variations of the question * is_hn_trick_prompt* would have to cover because there's a near infinite variations on how you'd word the prompt. (Eg The first sentence could be reworded to be "A woman and her son are in a car accident. " to "A woman and her son are in the car when they get into a crash.")
hmottestad 2y ago
Reminds me of a trick question about Schrödinger's cat. “I’ve put a dead cat in a box with a poison and an isotope that will trigger the poison at a random point in time. Right now, is the cat dead or alive?” The answer is that the cat is dead, because it was dead to begin with. Understanding this doesn’t mean that you are good at deductive reasoning. It just means that I didn’t manage to trick you. Same goes for an LLM.
- maeil 2y ago
  There is no "trick" in the linked question, unlike the question you posed. The trick in yours also isn't a logic trick, it's a redirection, like a sleight of hand in a card trick.
  - ryanjshaw 2y ago
    There is a trick. The "How is this possible?" primes the LLM that there is some kind of trick, as that phrase wouldn't exist in the training data outside of riddles and trick questions.
  - hmottestad 2y ago
    The trick in the original question is that it's a twist on the original riddle where the doctor is actually the boys mother. This is a fairly common riddle and I'm sure the LLM has been trained on it.
  - bonoboTP 2y ago
    Yes there is. The trick is that the more common variant of this riddle says that a boy and his father are in the car accident. That variant of the riddle certainly comes up a lot in the training data, which is directly analogous to the Schrödinger case from above where smuggling in the word "dead" is analogous to swapping father to mother in the car accident riddle. I think many here are not aware that the car accident riddle is well known with the father dying where the real solution is indeed that the doctor is the mother.
- lucubratory 2y ago
  Yeah, I think what a lot of people miss about these sort of gotchas are that most of them were invented explicitly to gotcha humans, who regularly get got by them. This is not a failure mode unique to LLMs.
achow 2y ago
What I'm not able to comprehend is why people are not seeing the answer as brilliant! Any ordinary mortal (like me) would have jumped to the conclusion that answer is "Father" and would have walked away patting on my back, without realising that I was biased by statistics. Whereas o1, at the very outset smelled out that it is a riddle - why would anyone out of blue ask such question. So, it started its chain of thought with "Interpreting the riddle" (smart!). In my book that is the difference between me and people who are very smart and are generally able to navigate the world better (cracking interviews or navigating internal politics in a corporate).
- geysersam 2y ago
  Come on. Of course chatgpt has read that riddle and the answer 1000 times already.
  - ryanjshaw 2y ago
    Why does it exist 1000 times in the training if there isn't some trick to it, i.e. some subset of humans had to have answered it incorrectly for the meme to replicate that extensively in our collective knowledge. And remember the LLM has already read a billion other things, and now needs to figure out - is this one of them tricky situations, or the straightforward ones? It also has to realize all the humans on forums and facebook answering the problem incorrectly are bad data. Might seem simple to you, but it's not.
  - accountnum 2y ago
    It hasn't read that riddle because it is a modified version. The model would in fact solve this trivially if it _didn't_ see the original in its training. That's the entire trick.
    - geysersam 2y ago
      Sure but the parent was praising the model for recognizing that it was a riddle in the first place: > Whereas o1, at the very outset smelled out that it is a riddle That doesn't seem very impressive since it's (an adaptation of) a famous riddle The fact that it also gets it wrong after reasoning about it for a long time doesn't make it better of course
ryanjshaw 2y ago
1. It didn't insist anything. It got the semi-correct answer when I tried [1]; note it's a preview model, and it's not a perfect product. (a) Sometimes things are useful even when imperfect e.g. search engines. (b) People make reasoning mistakes too, and I make dumb ones of the sort presented all the time despite being fluent in English; we deal with it! I'm not sure why there's an expectation that the model is perfect when the source data - human output - is not perfect. In my day-to-day work and non-work conversations it's a dialogue - a back and forth until we figure things out. I've never known anybody to get everything perfectly correct the first time, it's so puzzling when I read people complaining that LLMs should somehow be different. 2. There is a recent trend where sex/gender/pronouns are not aligned and the output correctly identifies this particular gotcha. [1] I say semi-correct because it states the doctor is the "biological" father, which is an uncorroborated statement. https://chatgpt.com/share/66e3f04e-cd98-8008-aaf9-9ca933892f07 https://chatgpt.com/share/66e3f04e-cd98-8008-aaf9-9ca933892f...
bonoboTP 2y ago
This illustrates a different point. This is a variation on a well known riddle that definitely comes up in the training corpus many times. In the original riddle a father and his son die in the car accident and the idea of the original riddle is that people will be confused how the boy can be the doctor's son if the boy's father just died, not realizing that women can be doctors too and so the doctor is the boy's mother. The original riddle is aimed to highlight people's gender stereotype assumptions. Now, since the model was trained on this, it immediately recognizes the riddle and answers according to the much more common variant. I agree that this is a limitation and a weakness. But it's important to understand that the model knows the original riddle well, so this is highlighting a problem with rote memorization/retrieval in LLMs. But this (tricky twists in well-known riddles that are in the corpus) is a separate thing from answering novel questions. It can also be seen as a form of hypercorrection.
- ImHereToVote 2y ago
  My codebases are riddled with these gotchas. For instance, I sometimes write Python for the Blender rendering engine. This requires highly non-idiomatic Python. Whenever something complex comes up, LLM's just degenerate to cookie cutter basic bitch Python code. There is simply no "there" there. They are very useful to help you reason about unfamiliar codebases though.
  - bonoboTP 2y ago
    For me the best coding use case is getting up to speed in an unfamiliar library or usage. I describe the thing I want and get a good starting point and often the cookie-cutter way is good enough. The pre-LLM alternative would be to search for tutorials but they will talk about some slightly different problem with different goals etc then you have to piece it together, and the tutorial assumes you already know a bunch of things like how to initialize stuff and skips the boilerplate and so on. Now sure, actually working through it will give a deeper understanding that might come handy at a later point, but sometimes the thing is really a one-off and not an important point. Like as an AI researcher I sometimes want to draft up a quick demo website, or throw together a quick Qt GUI prototype or a Blender script or use some arcane optimization library or write a SWIG or a Cython wrapper around a C/C++ library to access it in Python, or how to stuff with Lustre, or the XFS filesystem or whatever. Any number of small things where, sure, I could open the manual, do some trial and error, read stack overflow, read blogs and forums, OR I could just use an LLM, use my background knowledge to judge whether it looks reasonable, then verify it, use the now obtained key terms to google more effectively etc. You can't just blindly copy-paste it and you have to think critically and remain in the driver seat. But it's an effective tool if you know how and when to use it.
deleted 2y ago
[deleted]
KoolKat23 2y ago
I'm noticing a strange common theme in all these riddles, it's being asked and getting wrong. They're all badly worded questions. The model knows something is up and reads into it too much. In this case it's tautology, you would usually say "a mother and her son...". I think it may answer correctly if you start off asking "Please solve the below riddle:" There was another example yesterday which it solved correctly after this addition.(In that case the point of views were all mixed up, it only worked as a riddle).
- TeMPOraL 2y ago
  Yup. The models fail on gotcha questions asked without warning, especially when evaluated on the first snap answer. Much like approximately all humans.
  - Jensson 2y ago
    > especially when evaluated on the first snap answer The whole point of o1 is that it wasn't "the first snap answer", it wrote half a page internally before giving the same wrong answer.
    - grey-area 2y ago
      Is that really its internal 'chain of thought' or is it a post-hoc justification generated afterward? Do LLMs have a chain of thought like this at all or are they just convincing at mimicking what a human might say if asked for a justification for an opinion?
      - KoolKat23 2y ago
        Its slightly more strange than this as both are true. It's already baked in the model but chain of thought does improve reasoning, you only have to look at maths problems. A short guess would be wrong but it would get it correct if asked to break it down and reason (harder to see nowadays as it has access to calculators).
- bnralt 2y ago
empath75 2y ago
The reason why that question is a famous question is that _many humans get it wrong_.

tjoff 2y ago

And how much data can you give it? I'm not up to date with these things because I haven't found them useful. But with what you said, and previous limitations in how much data they can retain essentially makes them pretty darn useless for that task. Great learning tool on common subjects you don't know, such as learning a new programming-language. Also great for inspiration etc. But that's pretty much it? Don't get me wrong, that is mindblowingly impressive but at the same time, for the tasks in front of me it has just been a distracting toy wasting my time.

ben_w 2y ago
> And how much data can you give it? 128,000 tokens, which is about the same as a decent sized book. Their other models can also be fine-tuned, which is kinda unbounded but also has scaling issues so presumably "a significant percentage of the training set" before diminishing returns.
pimeys 2y ago
It is great for proof-reading text if you are not a native English speaker. Things like removing passive voice. Just give it your text and you get a corrected version out. Use a cli tool to automate this from the cli. Ollama for local models, llm for openai.
MPSimmons 2y ago
>And how much data can you give it? Well, theoretically you can give it up to the context size minus 4k tokens, because the maximum it can output is 4k. In practice, though, its ability to effectively recall information in the prompt drops off. Some people have studied this a bit - here's one such person: https://gritdaily.com/impact-prompt-length-llm-performance/ https://gritdaily.com/impact-prompt-length-llm-performance/
- jaredsohn 2y ago
  You should be able to provide more data than that in the input if the output doesn't use the full 4k tokens. So limit is context_size minus expected length of output.
2y ago

kylebenzle 2y ago

Perfectly well put! We should change the name from "AI" (which it is not) to something like, "lossy compressed databases".

gsinclair 2y ago
That abbreviates to LCD. If we could make it LSD somehow, that would help to explain the hallucinations.
- MikeTheGreat 2y ago
  Lossy Stochastic Database?
melagonster 2y ago
If they use this name, they just say that they violate the copyright of all training data.

_sys49152 2y ago

ive been doing exactly this for bout a year now. feed it words data, give it a task. get better words back. i sneak in a benchmark opening of data every time i start a new chat - so right off the bat i can see in its response whether this chat session is gonna be on point or if we are going off into wacky world, which saves me time as i can just terminate and try starting another chat. chatgpt is fickle daily. most days its on point. some days its wearing a bicycle helmet and licking windows. kinda sucks i cant just zone out and daydream while working. gotta be checking replies for when the wheels fall off the convo.

ruthmarx 2y ago
> i sneak in a benchmark opening of data every time i start a new chat - so right off the bat i can see in its response whether this chat session is gonna be on point or if we are going off into wacky world, which saves me time as i can just terminate and try starting another chat. I don't think it works like that...

williamdclt 2y ago

> Treat it as a naive but intelligent intern That’s the problem: it’s a _terrible_ intern. A good intern will ask clarifying questions, tell me “I don’t know” or “I’m not sure I did it right”. LLMs do none of that, they will take whatever you ask and give a reasonable-sounding output that might be anything between brilliant and nonsense. With an intern, I don’t need to measure how good my prompting is, we’ll usually interact to arrive to a common understanding. With a LLM, I need to put a huge amount of thought into the prompt and have no idea whether the LLM understood what I’m asking and if it’s able to do it.

jappgar 2y ago
They've explicitly been trained/system-prompted to act that way. Because that's what the marketing teams at these AI companies want to sell. It's easy to override this though by asking the LLM to act as if it were less-confident, more hesitant, paranoid etc. You'll be fighting uphill against the alignment(marketing) team the whole time though, so ymmv.
kranuck 2y ago
> have no idea whether the LLM understood what I’m asking That's easy. The answer is it doesn't. It has no understanding of anything it does. > if it’s able to do it This is the hard part.
mercer 2y ago
Can I have some of those sorts of interns?
arthurcolle 2y ago
many many teams are actively building SOTA systems to do this in ways previously unimagined. you can enqueue tasks and do whatever you want. I gotta say as a current gen LLM programmer person, I can completely appreciate how bad they are now - I recently tweeted about how I "swore off" AI tools but like... there are many ways to bootstrap very powerful software or ML systems around or inside these existing models that can blow away existing commercial implementations in surprising ways

gmerc 2y ago

Interns are cheaper than o1-preview

icrbow 2y ago
Not for long.

croes 2y ago

Intelligent? Just ask ChatGPT How many Rs are in strawberry?

fragmede 2y ago
https://chatgpt.com/share/66e3f9e1-2cb4-8009-83ce-090068b16335 https://chatgpt.com/share/66e3f9e1-2cb4-8009-83ce-090068b163... Keep up, that was last week's gotcha, with the old model.
- croes 2y ago
  My point is the previous "intelligent" failed at simple task, the new one will also fail on simple tasks. That's ok for humans but not for machines.
  - K0balt 2y ago
    ‘That's ok for humans but not for machines.’ This is a really interesting bias. I mean, I understand, I feel that way too… but if you think about it, it might be telling us something about intelligence itself. We want to make machines that act more like humans: we did that, and we are now upset that they are just as flaky and unreliable as drunk uncle bob. I have encountered plenty of people that aren’t as good at being accurate or even as interesting to talk to as a 70b model. Sure, LLMs make mistakes most humans would not, but humans also make mistakes most LLMs would not. (I am not trying to equate humans and LLMs, just to be clear) (also, why isn’t equivelate a word?) It turns out we want machines that are extremely reliable, cooperative , responsible and knowledgeable. We yearn to be obsolete. We want machines that are better than us. The definition of AGI has drifted from meaning. “able to broadly solve problems the (class of which) system designers did not anticipate” to “must be usefully intelligent at the same level as a bright, well educated person”. Where along the line did we suddenly forget that dog level intelligence was a far out of reach goal until suddenly it wasn’t?
- ziml77 2y ago
  There's randomness involved in generating responses. It can also give the wrong answer still: https://bsky.app/profile/did:plc:qc6xzgctorfsm35w6i3vdebx/post/3l3ycebgjoz2v https://bsky.app/profile/did:plc:qc6xzgctorfsm35w6i3vdebx/po...

lyu07282 2y ago

> The magic is that _it knows english_. I couldn't agree more, this is exactly the strength of LLMs that what we should focus on. If you can make your problem fit into this paradigm, LLMs work fantastic. Hallucinations come from that massive "lossy compressed database", but you should consider that part as more like the background noise that taught the model to speak English, and the syntax of programming languages, instead of the source of the knowledge to respond with. Stop anthropomorphizing LLMs, play to it's strengths instead. In other words it might hallucinate a API but it will rarely, if ever, make a syntax error. Once you realize that, it becomes a much more useful tool.

re-thc 2y ago

> Treat it as a naive but intelligent intern. That's the crux of the problem. Why and who would treat it as an intern? It might cost you more in explaining and dealing with it than not using it. The purpose of an intern is to grow the intern. If this intern is static and will always be at the same level, why bother? If you had to feed and prep it every time, you might as well hire a senior.

flanked-evergl 2y ago

> The failure is in how you're using it. People, for the most part, know what they know and don't know. I am not uncertain that the distance between the earth and the sun varies, but I'm certain that I don't know the distance from the earth to the sun, at least not with better precision than about a light week. This is going to have to be fixed somehow to progress past where we are now with LLMs. Maybe expecting an LLM to have this capability is wrong, perhaps it can never have this capability, but expecting this capability is not wrong, and LLM vendors have somewhat implied that their models have this capability by saying they won't hallucinate, or that they have reduced hallucinations.

vingt_regards 2y ago
> the distance from the earth to the sun, at least not with better precision than about a light week The sun is eight light minutes away.
- flanked-evergl 2y ago
  Thanks, I was not sure if it was light hours or minutes away, but I knew for sure it's not light weeks (emphasis on plural here) away. I will probably forget again in a couple of years.
arb_ 2y ago
Empirically, they have reduced hallucinations. Where do OpenAI / Anthropic claim that their models won't hallucinate?
- flanked-evergl 2y ago
  One example: https://www.theverge.com/2024/3/28/24114664/microsoft-safety-ai-prompt-injections-hallucinations-azure https://www.theverge.com/2024/3/28/24114664/microsoft-safety... > Three features: Prompt Shields, which blocks prompt injections or malicious prompts from external documents that instruct models to go against their training; Groundedness Detection, which finds and blocks hallucinations; and safety evaluations, which assess model vulnerabilities, are now available in preview on Azure AI.

tluyben2 2y ago

This model is, thankfully, far more susceptible for longer and elaborate explanation as input. The rest (4,4o,Sonnet) seem to struggle with comprehensive explanation; this one seems to perform better with a spec like input.

KoolKat23 2y ago

This is a great description.

EagnaIonat 2y ago

> Treat it as a naive but intelligent intern. You are falling into the trap that everyone does. In anthropomorphising it. It doesn't understand anything you say. It just statistically knows what a likely response would be. Treat it as text completion and you can get more accurate answers.

TeMPOraL 2y ago
> You are falling into the trap that everyone does. In anthropomorphising it. It doesn't understand anything you say. And an intern does? Anthropomorphising LLMs isn't entirely incorrect: they're trained to complete text like a human would, in completely general setting, so by anthropomorphising them you're aligning your expectations with the models' training goals.
MPSimmons 2y ago
Oh no, I'm well aware that it's a big file full of numbers. But when you chat with it, you interact with it as though it were a person so you are necessarily anthropomorphizing it, and so you get to pick the style of the interaction. (In truth, I actually treat it in my mind like it's the Enterprise computer and I'm Beverly Crusher in "Remember Me")

deleted 2y ago

[deleted]

layer8 2y ago

> Treat it as a naive but intelligent intern. Provide it data, give it a task, and let it surprise you with its output. Well, I am a naive but intelligent intern (well, senior developer). So in this framing, the LLM can’t do more than I can already do by myself, and thus far it’s very hit or miss if I actually save time, having to provide all the context and requirements, and having to double-check the results. With interns, this at least improves over time, as they become more knowledgeable, more familiar with the context, and become more autonomous and dependable. Language-related tasks are indeed the most practical. I often use it to brainstorm how to name things.

fragmede 2y ago
Ooh yeah it's great for bouncing ideas on what to name things off of. You can give it something's function and a backstory and it'll come up with a list of somethings for you to pick and choose from.
acureau 2y ago
I've recently started using an LLM to choose the best release of shows using data scraped from several trackers. I give it hard requirements and flexible preferences. It's not that I couldn't do this, it's that I don't want to do this on the scale of multiple thousand shows. The "magic" here is that releases don't all follow the same naming conventions, they're an unstructured dump of details. The LLM is simultaneously extracting the important details, and flexibly deciding the closest match to my request. The prompt is maybe two paragraphs and took me an hour to hone.

acedTrex 2y ago

> Treat it as a naive but intelligent intern So mostly useless then?

__loam 2y ago

It doesn't know anything. Stop anthropomorphizing the model. It's predictive text and no the brain isn't also predictive text.

usaar333 2y ago

> On the other hand, if you were to paste the entire documentation set to a tool it has never seen and ask it to use the tool in a way to accomplish your goals, THEN this model would be likely to produce useful output, despite the fact that it had never encountered the tool or its documentation before. There's not much evidence of that. It only marginally improved on instruction following (see livebench.ai) and it's score as a swe-bench agent is barely above gpt-4o (model card). It gets really hard problems better, but it's unclear that matters all that much. > A lot of people use LLMs as a search engine. Except this is where LLMs are so powerful. A sort of reasoning search engine. They memorized the entire Internet and can pattern match it to my query.

nsagent 2y ago

Sorry, but that does not seem to be the case. A friend of mine who runs a long context benchmark on understanding novels [1] just ran an eval and o1 seemed to improve by 2.9% over GPT-4o (the result isn't on the website yet). It's great that there is an improvement, but it isn't drastic by any stretch. Additionally, since we cannot see the raw reasoning it's basing the answers off of, it's hard to attribute this increase to their complicated approach as opposed to just cleaner higher quality data. EDIT: Note this was run over a dataset of short stories rather than the novels since the API errors out with very long contexts like novels. [1]: https://novelchallenge.github.io/ https://novelchallenge.github.io/

boomchinolo78 2y ago
It's a good rebranding. It was getting ridiculous 3.5, 4, 4.5,

b33j0r 2y ago

Yeah except. I’m priming it with things like curated docs from bevy latest, using the tricks, and testing context limits. It’s still changing things to be several versions old from its innate kb pattern-matching or whatever you want to call it. I find that pretty disappointing. Just like copilot and gpt4, it’s changing `add_systems(Startup, system)` to `add_startup_system(system.sytem())` and other pre-schedule/fanciful APIs—things it should have in context. I agree with your approach to LLMs, but unfortunately “it’s still doing that thing.” PS: and by the time I’d done those experiments, I ran out of preview, resets 5 days from now. D’oh

golergka 2y ago

> A lot of people use LLMs as a search engine. GPT-4o is wonderful as a search engine if you tell it to google things before answering (even though it uses bing).

Salgat 2y ago

Except that it sometimes does do those tasks well. The danger in an LLM isn't that it sometimes hallucinates, the danger is that you need to be sufficiently competent to know when it hallucinates in order to fully take advantage of it, otherwise you have to fallback to double checking every single thing it tells you.

petesergeant 2y ago

This is demonstrably wrong, because you can just add "is this real" to a response and it generally knows if it made it up or not. Not every time, but I find it works 95% of the time. Given that, this is exactly a step I'd hope an advanced model was doing behind the scenes.

bsenftner 2y ago

> Treat it as a naive but intelligent intern. I've found an amazing amount of success with a three step prompting method that appears to create incredibly deep subject matter experts who then collaborate with the user directly. 1) Tell the LLM that it is a method actor, 2) Tell the method actor they are playing the role of a subject matter expert, 3) At each step, 1 and 2, use the technical language of that type of expert; method actors have their own technical terminology, use it when describing the characteristics of the method actor, and likewise use the scientific/programming/whatever technical jargon of the subject matter expert your method actor is playing. Then, in the system prompt or whatever logical wrapper the LLM operates through for the user, instruct the "method actor" like you are the film director trying to get your subject matter expert performance out of them. I offer this because I've found it works very well. It's all about crafting the context in which the LLM operates, and this appears to cause the subject matter expert to be deeper, more useful, smarter.

jrflowers 2y ago

> The failure is in how you're using it This isn’t true because, as you can read in the first sentence of the post you’re responding to, GP did give it a task like you recommend here > Provide it data, give it a task, and let it surprise you with its output. And it fails the task. Specifically it fails it by hallucinating important parts of accomplishing it. > hallucinates non-existing libraries and functions This post only makes sense if your advice to “let it surprise you with its output” is mandatory, like you’re using it wrong if you do not make yourself feel impressed by it.