7 ms·
The programming task they gave o3-mini high (creating Python server that allows chatting with OpenAI API and run some code in terminal) didn't seem very hard? S
by croemer 2y ago
The programming task they gave o3-mini high (creating Python server that allows chatting with OpenAI API and run some code in terminal) didn't seem very hard? Strange choice of example for something that's claimed to be a big step forwards.
YT timestamped link: https://www.youtube.com/watch?v=SKBG1sqdyIU&t=768s https://www.youtube.com/watch?v=SKBG1sqdyIU&t=768s (thanks for the fixed link @photonboom)
Updated: I gave the task to Claude 3.5 Sonnet and it worked first shot: https://claude.site/artifacts/36cecd49-0e0b-4a8c-befa-faa5aaa102e6 https://claude.site/artifacts/36cecd49-0e0b-4a8c-befa-faa5aa...
- bearjaws 2y agoIt's good that it works since if you ask GPT-4o to use the openai sdk it will often produce invalid and out of date code.
- luke-stanley 2y agoBut they did use a prompt that included a full example of how to call their latest model and API!
- m3kw9 2y agoI would say they didn’t need to demo anything, because if you are gonna use the output code live on a demo it may make compile errors and then look stupid trying to fix it live
- croemer 2y agoIf it was a safe bet problem, then they should have said that. To me it looks like they faked excitement for something not exciting which lowers credibility of the whole presentation.
- sunaookami 2y agoThey actually did that the last time when they showed the apps integration. First try in Xcode didn't work.
- m3kw9 2y agoYeah I think that time it was ok because they were demoing the app function, but for this they are demoing the model smarts
- csomar 2y agoModels are predictable at 0 temperatures. They might have tested the output beforehand.
- fzzzy 2y agoModels in practice haven't been deterministic at 0 temperature, although nobody knows exactly why. Either hardware or software bugs.
- Jensson 2y agoWe know exactly why, it is because floating point operations aren't associative but the GPU scheduler assumes they are, and the scheduler isn't deterministic. Running the model strictly hurts performance so they don't do that.
- fzzzy 2y agoCool, thanks a lot for the explanation. Makes sense.
- photonboom 2y agoheres the right timestamp: https://www.youtube.com/watch?v=SKBG1sqdyIU&t=768s https://www.youtube.com/watch?v=SKBG1sqdyIU&t=768s
- phil917 2y agoYeah I agree that wasn't particularly mind blowing to me and seems fairly in line with what existing SOTA models can do. Especially since they did it in steps. Maybe I'm missing something.
- MyFirstSass 2y agoWhat? Is this what this is? Either this is a complete joke or we're missing something. I've been doing similar stuff in Claude for months and it's not that impressive when you see how limited they really are when going non boilerplate.
- HeatrayEnjoyer 2y agoSonnet isn't a "mini" sized model. Try it with Haiku.
- croemer 2y agoHow mini is o3-mini compared to Sonnet and why does it matter whether it's mini or not? Isn't the point of the demo to show what's now possible that wasn't before? 4o is cheaper than o1 mini so mini doesn't mean much for costs.
- deleted 2y ago[deleted]
- zelphirkalt 2y agoLooks like quite shoddy code though. Like, the procedure for running a shell command is pure side-effect procedural code, neither returning the exit code of the command nor its output. Like the incomplete stackoverflow answer it probably was trained from. It might do one job at a time, but once this stuff gets integrated into one coherent thing, one needs to rewrite lots of it, to actually be composable. Though, of course one can argue, that lots of human written code is not much different from this.
- croemer 2y agoWhich code is shoddy? The Claude or o3-mini one? If you mean Claude, then have you checked the o3-mini one is better?
- zelphirkalt 2y agoYoutube is currently blocking my VPN, can't watch it.