5 ms·
I haven’t tried o3, but one issue I struggle with in large context analysis tasks is the LLMs are never thorough. In a task like this thread summarization, I ty
by threecheese 2y ago
I haven’t tried o3, but one issue I struggle with in large context analysis tasks is the LLMs are never thorough. In a task like this thread summarization, I typically need to break the document down and loop through chunks to ensure it actually “reads” everything. I might have had to recurse into individual conversations with some small max-depth and leaf count and run inference on each, and then have some aggregation at the end, otherwise it would miss a lot (or appear to, based on the output).
Is this a case of PEBKAC?
- syntaxing 2y agoDepending on what you’re trying to do, it’s worth trying the 1M context Qwen Models. They only released 7 and 14B so it’s “intelligence” is limited but should be more than capable for coherent summary.
- andrewci 2y agoAre there any tools you use to do this chunking? Or is this a custom workflow? I've noticed the same thing both on copy/paste text and uploaded documents when using the LLM chat tools.
- sdesol 2y ago> I haven’t tried o3, but one issue I struggle with in large context analysis tasks is the LLMs are never thorough. o3 does look very promising with regards to large context analysis. I used the same raw data and ran the same prompt as Simon for GPT-4o, GPT-4o mini and DeepSeek R1 and compared their output. You can find the analysis below: https://beta.gitsense.com/?chat=46493969-17b2-4806-a99c-5d93012b3580 https://beta.gitsense.com/?chat=46493969-17b2-4806-a99c-5d93... The o3-min model was quite thorough. With reasoning models, it looks like dealing with long context might have gotten a lot better. Edit: I was curious if I could get R1 to be more thorough and got the following interesting tidbits. - Depth Variance: R1 analysis provides more technical infrastructure insights, while o3-mini focuses on developer experience - Geopolitical Focus: Only R1 analysis addresses China-West tensions explicitly - Philosophical Scope: R1 contains broader industry meta-commentary absent in o3-mini - Contrarian Views: o3-mini dedicates specific section to minority opinions - Temporal Aspects: R1 emphasizes future-looking questions, o3-mini focuses on current implementation You can find the full analysis at https://beta.gitsense.com/?chat=95741f4f-b11f-4f0b-8239-83c7e558ac2a&model=Fake+LLM+Simulator+1 https://beta.gitsense.com/?chat=95741f4f-b11f-4f0b-8239-83c7...
- mvkel 2y agoo1-pro is incredibly good at this. You'll be amazed
- andrewci 2y ago[dead]
- scarface_74 2y agoTry Google’s NotebookLM
- ben_w 2y agoI put one of my own blog posts through NotebookLM soon after it became available, it hallucinated content I didn't write and missed out things I had written. Nice TTS, but otherwise I found it unimpressive.
- scarface_74 2y agoI’m not talking about the TTS and podcast creation. I’m talking about just asking questions where it gives you the answer with citations.
- ben_w 2y agoGiven what it got wrong was in the LLM part, that is a distinction without a difference.
- scarface_74 2y agoThe podcast creation does a lot of small talk. Just like people do. Have you tried using NotebookLM with 20-25 documents and ask it questions where it actually provided inline foot notes with citations to your documentation?
- inciampati 2y ago> Is this a case of PEBKAC? It's 2025 and every useful conversation with an LLM ends with context exhaustion. There are those who argue this is a feature and not a bug. Or that the context lengths we have are enough. I think they lack imagination. True general intelligence lies on the other side of infinite context length. Memory makes computation universal, remember? http://thinks.lol/2025/01/memory-makes-computation-universal/ http://thinks.lol/2025/01/memory-makes-computation-universal...