4 ms·
I got gpt-4.5-preview to summarize this discussion thread so far (at 324 comments): hn-summary.sh 43197872 -m gpt-4.5-preview Using this script: https://til
by simonw 2y ago
I got gpt-4.5-preview to summarize this discussion thread so far (at 324 comments):
hn-summary.sh 43197872 -m gpt-4.5-preview
Using this script: https://til.simonwillison.net/llms/claude-hacker-news-themes#user-content-adding-a--m-model-option https://til.simonwillison.net/llms/claude-hacker-news-themes...
Here's the result: https://gist.github.com/simonw/5e9f5e94ac8840f698c280293d39965e https://gist.github.com/simonw/5e9f5e94ac8840f698c280293d399...
It took 25797 input tokens and 1225 input tokens, for a total cost (calculated using https://tools.simonwillison.net/llm-prices https://tools.simonwillison.net/llm-prices ) of $2.11! It took 154 seconds to generate.
- djhworld 2y agointeresting summary but it's hard to gauge whether this is better/worse than just piping the contents into a much cheaper model.
- azinman2 2y agoIt’d be great if someone would do that with the same data and prompt to other models. I did like the formatting and attributions but didn’t necessarily want attributions like that for every section. I’m also not sure if it’s fully matching what I’m seeing in the thread but maybe the data I’m seeing is just newer.
- simonw 2y agoGood call. Here's the same exact prompt run against: GPT-4o: https://gist.github.com/simonw/592d651ec61daec66435a6f718c0618b https://gist.github.com/simonw/592d651ec61daec66435a6f718c06... GPT-4o Mini: https://gist.github.com/simonw/cc760217623769f0d7e4687332bce409 https://gist.github.com/simonw/cc760217623769f0d7e4687332bce... Claude 3.7 Sonnet: https://gist.github.com/simonw/6f11e1974e4d613258b3237380e0ecb3 https://gist.github.com/simonw/6f11e1974e4d613258b3237380e0e... Claude 3.5 Haiku: https://gist.github.com/simonw/c178f02c97961e225eb615d4b9a1dea3 https://gist.github.com/simonw/c178f02c97961e225eb615d4b9a1d... Gemini 2.0 Flash: https://gist.github.com/simonw/0c6f071d9ad1cea493de4e5e7a0986bb https://gist.github.com/simonw/0c6f071d9ad1cea493de4e5e7a098... Gemini 2.0 Flash Lite: https://gist.github.com/simonw/8a71396a4a219d8281e294b61a9d6dd5 https://gist.github.com/simonw/8a71396a4a219d8281e294b61a9d6... Gemini 2.0 Pro (gemini-2.0-pro-exp-02-05): https://gist.github.com/simonw/112e3f4660a1a410151e86ec677e34ab https://gist.github.com/simonw/112e3f4660a1a410151e86ec677e3...
- iamjs 2y agoAt a glance, none of these appear to be meaningfully worse than GPT-4.5
- mastercheif 2y agoSeeing the other models, I actually come away impressed with how well GPT-4.5 is organizing the information and how well it reads. I find it a lot easier to quickly parse. It's more human-like.
- jwr 2y agoI actually think the Claude 3.7 Sonnet summary is better.
- hexa00 2y agoyeah I liked it too, especially for 10x less the price lol
- unoti 2y agoI noticed 4o mini didn't follow the directions to quote users. My favourite part of the 4.5 summary was how it quoted Antirez. 4o mini brought out the same quote, but failed to attribute it as instructed.
- Agentlien 2y agoIt's fascinating, but while this does mean it strays from the given example, I actually feel the result is a better summary. The 4.5 version is so long you might just read the whole thread yourself.
- NitpickLawyer 2y agoInteresting, thanks for doing this. I'd say that (at a glance) for now it's still worth to use more passes with smaller models than one pass with 4.5 Now, if you'd want to generate training data, I could see wanting to have the best answers possible, where even slight nuances would matter. 4.5 seems to adhere to instructions much better than the others. You might get the same result w/ generating n samples and "reflect" on them with a mixture of models, but then again you might not. Going through thousands of generations manually is also costly.
- joe_the_user 2y agoThe headline and section: "Dystopian and Social Concerns about AI Features" are interesting. It's roughly true... but somehow that broad statement seems minimize the point discussed. I'd headline that thread as "Concerns about output tone". There were comments about dystopian implications of tone, marketing implications of tone and implementation issues of tone. Of course, that I can comment about the fine-points of an AI summary shows it's made progress. But there's a lot riding on how much progress these things can make and what sort. So it's still worth looking at.
- alew1 2y agoDidn't seem to realize that "Still more coherent than the OpenAI lineup" wouldn't make sense out of context. (The actual comment quoted there is responding to someone who says they'd name their models Foo, Bar, Baz.)
- colordrops 2y agoMaybe it's just confirmation bias but the language in your result output seems higher quality that previous models. Seems more natural and eloquent.
- 3vidence 2y agoI don't know why but something about this section made me chuckle """ These perspectives highlight that there remains nuance—even appreciation—of explorative model advancement not solely focused on immediate commercial viability """ Feels like the model is seeking validation
- stevage 2y agoHuh. Disregarding the 4.5-specific bit here, a browser extension or possibly website that did this in general could be really useful. Maybe even something that just noticed whenever you visited a site that had had significant HN discussion in the past, then let you trigger a summary.
- wordpad25 2y agothere are literally hundreds of extensions and sites that do this the problem is that they are competing each other into the ground hence they go unmaintained very quickly getrecall.ai has been the most mature so far
- stevage 2y agoThanks, it's amazing how much stuff is out there I don't know about.
- bredren 2y agoHundreds that specifically focus on noticing a page you’re currently viewing has been not only posted to but undergone significant discussion on HN, and then providing a summary of those conversations? Or that just provide summaries in general?
- ruibiks 2y agoHey, check this one out with all the different flavors that existed out there. I think I made something better. https://cofyt.app https://cofyt.app As far as I am aware, feel free to test it head-to-head. This is better than gecall, and you can chat with a transcript for detailed answers to your prompts
- wordpad25 2y agoI tried it out, looks nice and clean. But as I mentioned, my main concern is what will happen in 6 months when you fail to get traction and abandon it. Because that's what happened to previous 5 products I tried which were all "good enough" . Getrecall seems to have a big enough user base that will actually stick around.
- dahsameer 2y ago$2.11! At this point, I'm more concerned about AI price than egg price.
- vivzkestrel 2y agoyou know what? it would be damn nice to do this to literally every post in HN and give people a summary so that they dont have to read 500 comments
- munksbeer 2y agoAs expected, comments on LLM threads are overwhelmingly negative. Personally, I still feel excited to see boundaries being pushed, however incremental our anecdotal opinions make them seem.
- sundarurfriend 2y agoI disagree with most of the knee-jerk negativity in LLM threads, but in this case it mostly seems warranted. There are no "boundaries being pushed" here, this is just a desperate release from a company that finds itself losing more and more mindshare to other models and companies.
- egobrain27 2y agoSeems to have trouble recognizing sarcasm: "For example, there are now a bunch of vendors that sell 'respond to RFP' AI products... paying 30x for marginally better performance makes perfect sense." — hn_throwaway_99 (an uncommon opinion supporting possible niche high-cost uses).
- mlyle 2y ago? You think hn_throwaway_99's comment is sarcastic? It makes perfect sense to me read "straight." That is, sales orgs save a bunch of money using AI to respond to RFPs; they would still save a bunch of money using a more expensive AI, and any marginal improvement in sales closed would pay for it. It maybe excessively summarized his comment which confused you-- but this is the kind of mistake human curators of quotes make, too.
- sebzim4500 2y agoI don't think they are being sarcastic. Maybe you are the bot /s