7 ms·
I used o3-mini to summarize this thread so far. Here's the result: https://gist.github.com/simonw/09e5922be0cbb85894cf05e6d75ae050 https://gist.github.com/simon
by simonw 2y ago
I used o3-mini to summarize this thread so far. Here's the result: https://gist.github.com/simonw/09e5922be0cbb85894cf05e6d75ae050 https://gist.github.com/simonw/09e5922be0cbb85894cf05e6d75ae...
For 18,936 input, 2,905 output it cost 3.3612 cents.
Here's the script I used to do it: https://til.simonwillison.net/llms/claude-hacker-news-themes#user-content-adding-a--m-model-option https://til.simonwillison.net/llms/claude-hacker-news-themes...
- deleted 2y ago[deleted]
- layman51 2y agoI noticed that it thought that GoatInGrey wrote “openai is no longer relevant.” However, they were just quoting a different user (buyucu) who was the person who first wrote that.
- simonw 2y agoGood catch. That's likely an artifact of the way I flatten the nested JSON from the comments API. I originally did that to save on tokens but modern models have much larger input windows so I may not need to do that any more.
- acka 2y agoThis is why I keep advocating that JSON should die, or at least no longer be used with LLMs. LLMs (and human brains) are simply not wired up for counting nested curly brackets across long spans of quoted text joined by colons and commas, and it is far too easy for humans to make mistakes when chunking JSON. IMO, (Strict)YAML is a very good alternative, it has even been suggested to me by multiple LLMs when I asked them what they thought the best format for presenting conversations to an LLM would be. It is very easy to chunk simple YAML and present it to an LLM directly off the wire: you only need to remember to repeat the indentation and names of all higher level keys (properties) pertaining to the current chunk at the top of the chunk, then start a text block containing the remaining text in the chunk, and the LLM will happily take it from there: topic: subtopic: text: | Subtopic text for this chunk. If you want to make sure that the LLM understands that it is dealing with chunks of a larger body of text, you can start and end the text blocks of the chunks with an ellipsis ('...').
- inciampati 2y agoLLMs (transformers) literally cannot balance parentheses. That's outside of their complexity class (TC0). You'd want a real UTM to count parentheses!
- m0rde 2y agoWould you elaborate on why counting braces is different from counting spaces to determine hierarchy? Or is it more about the repetition of higher levels keys in chunks (which could be done in JSON)?
- acka 2y agoRepetition of topics and subtopics is by far the most important part, reinforcing attention on the topic at hand even if the text in the chunk appears unrelated to the topic when viewed in isolation. Keeping the indentation is also important because it is an implicit and repeated indication of the nesting level of the content that follows. LLMs have trouble with balancing nested parentheses (as the sibling comment to yours explains). Dealing with text where indentation matters is easier for LLMs, and because they have been exposed to large amounts of it (such as Python code and lists of bullet points) during training, they have learned to handle this quite well.
- threecheese 2y agoI haven’t tried o3, but one issue I struggle with in large context analysis tasks is the LLMs are never thorough. In a task like this thread summarization, I typically need to break the document down and loop through chunks to ensure it actually “reads” everything. I might have had to recurse into individual conversations with some small max-depth and leaf count and run inference on each, and then have some aggregation at the end, otherwise it would miss a lot (or appear to, based on the output). Is this a case of PEBKAC?
- syntaxing 2y agoDepending on what you’re trying to do, it’s worth trying the 1M context Qwen Models. They only released 7 and 14B so it’s “intelligence” is limited but should be more than capable for coherent summary.
- andrewci 2y agoAre there any tools you use to do this chunking? Or is this a custom workflow? I've noticed the same thing both on copy/paste text and uploaded documents when using the LLM chat tools.
- sdesol 2y ago> I haven’t tried o3, but one issue I struggle with in large context analysis tasks is the LLMs are never thorough. o3 does look very promising with regards to large context analysis. I used the same raw data and ran the same prompt as Simon for GPT-4o, GPT-4o mini and DeepSeek R1 and compared their output. You can find the analysis below: https://beta.gitsense.com/?chat=46493969-17b2-4806-a99c-5d93012b3580 https://beta.gitsense.com/?chat=46493969-17b2-4806-a99c-5d93... The o3-min model was quite thorough. With reasoning models, it looks like dealing with long context might have gotten a lot better. Edit: I was curious if I could get R1 to be more thorough and got the following interesting tidbits. - Depth Variance: R1 analysis provides more technical infrastructure insights, while o3-mini focuses on developer experience - Geopolitical Focus: Only R1 analysis addresses China-West tensions explicitly - Philosophical Scope: R1 contains broader industry meta-commentary absent in o3-mini - Contrarian Views: o3-mini dedicates specific section to minority opinions - Temporal Aspects: R1 emphasizes future-looking questions, o3-mini focuses on current implementation You can find the full analysis at https://beta.gitsense.com/?chat=95741f4f-b11f-4f0b-8239-83c7e558ac2a&model=Fake+LLM+Simulator+1 https://beta.gitsense.com/?chat=95741f4f-b11f-4f0b-8239-83c7...
- breakingcups 2y agoIt's definitely making some errors
- kandesbunzler 2y agoLike?
- aprilthird2021 2y agoEven though it was told that it MUST quote users directly, it still outputs: > It’s already a game changer for many people. But to have so many names like o1, o3-mini, GPT-4o, & GPT-4o-mini suggests there may be too much focus on internal tech details rather than clear communication." (paraphrase based on multiple similar sentiments) It also hallucinates quotes. For example: > "I’m pretty sure 'o3-mini' works better for that purpose than 'GPT 4.1.3'." – TeMPOraL But that comment is not in the user TeMPOraL's comment history. Sentiment analysis is also faulty. For example: > "I’d bet most users just 50/50 it, which actually makes it more remarkable that there was a 56% selection rate." – jackbrookes – This quip injects humor into an otherwise technical discussion about evaluation metrics. It's not a quip though. That comment was meant in earnest
- romanhn 2y agoThat's funny, the quote exists, but it got the user wrong.
- breakingcups 2y agoIn addition to that, it has a section dedicated all to Model Naming and Branding Confusion, but then it puts the following comment in the Performancce and Benchmarking section, even though the value of the comment is ostensibly more to do with the naming being a hindrance rather than make a valuable remark on the benchmarking, which is more of a casualty to the naming confusion: "The model naming all around is so confusing. Very difficult to tell what breakthrough innovations occurred." – patrickhogan1"
- Eduard 2y ago3.3612 cents (I guess USD cents) is expensive!
- BoorishBears 2y agoSame immediate thought: the free option I provide on my production site is a model that runs on 2xA40. That's 96GB of VRAM for 78 cents an hour serving at least 4 or 5 concurrent requests at any given time. O3 Mini is probably not a very large model and OpenAI has layers upon layers of efficiencies, so they must be making an absolute killing charging 3.3 cents for a few seconds of compute
- largbae 2y agoThanks for sharing this! And no apparent self-awareness! OpenAI has come a long way from the Sydney days: https://answers.microsoft.com/en-us/bing/forum/all/this-ai-chatbot-sidney-is-misbehaving/e3d6a29f-06c9-441c-bc7d-51a68e856761 https://answers.microsoft.com/en-us/bing/forum/all/this-ai-c...
- Valakas_ 2y agoFor those that like simpler ways (although dependent on Google) NotebookLM does all that in 2 clicks. And you can ask it questions about it, references are provided.
- tkgally 2y agoBorrowing most of Simon’s prompt, I tried the following with o3-mini-high in the chat interface with Search turned on: “Summarize the themes of the opinions expressed in discussions on Hacker News on January 31 and February 1, 2025, about OpenAI’s release od [sic] ChatGPT o3-mini. For each theme, output a header. Include direct "quotations" (with author attribution) where appropriate. You MUST quote directly from users when crediting them, with double quotes. Fix HTML entities. Go long. Include a section of quotes that illustrate opinions uncommon in the rest of the piece” The result is here: https://chatgpt.com/share/679d790d-df6c-8011-ad78-3695c2e254fd https://chatgpt.com/share/679d790d-df6c-8011-ad78-3695c2e254... Most of the cited quotations seem to be accurate, but at least one (by uncomplexity_) does not appear in the named commenter’s comment history. I haven’t attempted to judge how accurate the summary is. Since the discussions here are continuing at this moment, this summary will be gradually falling out of date in any case.
- thousand_nights 2y agogood morning!
- qwertox 2y agoHow is the `system` prompt "injected"? Afaik o1 onwards no longer have a SYSTEM role. Is it just a normal part of the prompt?
- naiv 2y agoIt's called 'developer message' now https://x.com/OpenAIDevs/status/1885407759887155301 https://x.com/OpenAIDevs/status/1885407759887155301
- tempaccount420 2y agoI'm guessing it's less powerful than a system prompt? And there is still a system prompt, but only OpenAI can use it...
- s_dev 2y agoCurrently on the internet people skip the article and go straight to the comments. Soon people will skip the comments and go striaght to an AI summary reading neither the original article nor the comments.
- fsmv 2y agoI won't. I'm looking for real life human voices.
- darthrupert 2y agoThat would be an actual improvement. Reading the comments section usually just leads to personal energy waste.
- zapkyeskrill 2y agoYes, but who will comment then and on what?
- anonzzzies 2y agoOn both HN & Reddit, I find the comments more informative and less frustrating than reading the article usually. But I guess YMMV.
- brabel 2y agoYou need to read the article first to know that. But most people won't.
- Jensson 2y agoYou can read the article after the comments.
- anonzzzies 2y agoIf the comments say it's worth it I read it, but often (especially on HN) the top comment starts with a summary of errors/inconsistencies in the article and then I don't really feel the need.
- wkat4242 2y agoWhy use a reasoning model for a summarisation task? Serious question, would it benefit? I don't have much experience with reasoning models yet. That's why.
- BoorishBears 2y agoIn theory you could get a better summary if it manages to reason out underlying motivations of a summary for example, or proposes a summary and then "considers" it, realizing it missed something. You can illicit that with any model by prompting underlying reasons or using chain-of-thought, but a reasoning model could do it without prompting
- profsummergig 2y agoSomething I've not understood about their pricing system. I have the $20 plan. How does this "3.3612 cents" apply to my situation?
- elliotec 2y agoThe API has a different pricing structure than using the app. This cost seems high to me compared to previous models I’ve used large data sets with.
- georgeck 2y agoI have been trying to approach the problem in a similar way, and in my observation, it is also important to capture the discussion hierarchy in the context that we share with the LLM. The solution that I have adopted is as follows. Each comment is represented in the following notation: [discussion_hierarchy] Author Name: <comment> To this end, I format the output from Algolia as follows: [1] author1: First reply to the post [1.1] author2: First reply to [1] [1.1.1] author3: Second-level reply to [1.1] [1.2] author4: Second reply to [1] After this, I provide a system prompt as follows: You are an AI assistant specialized in summarizing Hacker News discussions. Your task is to provide concise, meaningful summaries that capture the essence of the thread without losing important details. Follow these guidelines: 1. Identify and highlight the main topics and key arguments. 2. Capture diverse viewpoints and notable opinions. 3. Analyze the hierarchical structure of the conversation, paying close attention to the path numbers (e.g., [1], [1.1], [1.1.1]) to track reply relationships. 4. Note where significant conversation shifts occur. 5. Include brief, relevant quotes to support main points. 6. Maintain a neutral, objective tone. 7. Aim for a summary length of 150-300 words, adjusting based on thread complexity. Input Format: The conversation will be provided as text with path-based identifiers showing the hierarchical structure of the comments: [path_id] Author: Comment This list is sorted based on relevance and engagement, with the most active and engaging branches at the top. Example: [1] author1: First reply to the post [1.1] author2: First reply to [1] [1.1.1] author3: Second-level reply to [1.1] [1.2] author4: Second reply to [1] Your output should be well-structured, informative, and easily digestible for someone who hasn't read the original thread. Use markdown formatting for clarity and readability. The benefit is that, I can parse the output from the LLM and create links back to the original comment thread. You can read about my approach in more detail here: https://gist.github.com/simonw/09e5922be0cbb85894cf05e6d75ae050?permalink_comment_id=5418383#gistcomment-5418383 https://gist.github.com/simonw/09e5922be0cbb85894cf05e6d75ae...
- falseAss 2y agoI just installed and tried. Pretty neat stuff! Would be great if the addon allows user to override the sys prompt (it might need minor tweak when changing different server backend)?
- digdugdirk 2y agoOut of curiosity - do you have an estimate on how much you've spent on various LLM API services through all of your experimentation? And in your experience, what service do you feel hits a good sweet spot for performance/price if summarizing long text excerpts is the main use case? Inference time isn't an issue, this will be an ongoing background task.