8 ms·
My work gives us Copilot access since We're a Microsoft Shop™ It's too bad Copilot is by far the dumbest competitor in the space My favorite interaction so fa
by hbn 1y ago
My work gives us Copilot access since We're a Microsoft Shop™
It's too bad Copilot is by far the dumbest competitor in the space
My favorite interaction so far was when I prompted it with:
ffmpeg command to convert movie.mov into a reasonably sized mp4
Sure, it's not the most direction instructions, but I tend to give it just enough to get the job done, assuming the LLM knows what its purpose is as an LLM, and it always works with the other chatbots.
Copilot's response:
I implemented and executed the Python code above to convert movie.mov to a reasonably sized movie.mp4 using ffmpeg.
However, the Python code failed since it was not able to find and access movie.mov file.
Do you want me to try again or is there anything else that I can help you with?
Note that I didn't cut anything out. It didn't actually provide me any "Python code above"
- whirlwin 1y agoSo much for the quality testing done by Microsoft...
- vpShane 1y agoIt's an AI race and they're all vibing off of what the other ones are doing and implementing accordingly. Meanwhile, I've been enjoying Google's AI studio. I can definitely see though the AI companies are trying, they're trying to give devs, and teams the tools they need to operate more efficiently and faster. However, piping code requests, and hey->do this, AI gives you something; the privacy aspect of it. On the in-terms, these ARE LLMs, devs/infra people do look at how their tools are being used. You can pull a lot of info with and about an organizatin and what their internals are up to just from how they're using the AI's information. So while we're looking at Microsoft's quality of testing, what does that really mean in terms of how they're viewing the tool usage.
- robots0only 1y agoohh wow, that's bad, just tried this with Gemini 2.5 Flash/Pro (and worked perfectly) -- I assume all frontier models should get this right (even simpler models should).
- quantadev 1y agoI'd be willing to bet a more clear prompt would've given a good answer. People generally tend to overlook the fact that AIs aren't like "google". They're not really doing pure "word search" similar to Google. They expect a sensible sentence structure in order to work their best.
- roywiggins 1y agoMaybe, but this sort of prompt structure doesn't bamboozle the better models at all. If anything they are quite good at guessing at what you mean even when your sentence structure is crap. People routinely use them to clean up their borderline-unreadable prose.
- quantadev 1y agoI wish I had a nickle for every time I've seen someone get a garbage response from a garbage prompt and then blame the LLM.
- macNchz 1y agoI'm all about clear prompting, but even using the verbatim prompt from the OP "ffmpeg command to convert movie.mov into a reasonably sized mp4", the smallest current models from Google and OpenAI (gemini-2.5-flash-lite and gpt-4.1-nano) both produced me a working output with explanations for what each CLI arg does. Hell, the Q4 quantized Mistral Small 3.1 model that runs on my 16GB desktop GPU did perfectly as well. All three tests resulted in a command using x264 with crf 23 that worked without edits and took a random .mov I had from 75mb to 51mb, and included explanations of how to adjust the compression to make it smaller.
- quantadev 1y agoThere's as much variability in LLM AI as there is in human intelligence. What I'm saying is that I bet if that guy wrote a better prompt his "failing LLM" is much more likely to stop failing, unless it's just completely incompetent. What I always find hilarious too is when the AI Skeptics try to parlay these kinds of "failures" into evidence LLMs cannot reason. If course they can reason.
- hu3 1y agoWhat model? It got right first try here with your exact prompt using the free GPT-4.1 model. https://i.imgur.com/toLzwCk.png https://i.imgur.com/toLzwCk.png ffmpeg -i movie.mov -c:v libx264 -preset medium -crf 23 -c:a aac -b:a 128k movie.mp4 BUT, I have this in my copilot-instructions.md file: # Always follow these steps when responding to any request 1. Please do a round of thinking in <thinking></thinking> tags 2. Then a round of self-critique in <critique></critique> tags 3. Then a final round of <thinking>, before responding. 4. If you need more information, ask for it.
- easton 1y agoThat's GitHub Copilot, not "microsoft copilot", the bot built into the Microsoft 365 landing site. it doesn't give you model options, for example.
- clvx 1y agoWhoever decided to use the same brand for different experiences should be demoted at minimum. What a way to destroy trust.
- airstrike 1y agoI suggest you never visit https://www.office.com/ https://www.office.com/
- DangitBobby 1y agoGood lord https://news.ycombinator.com/item?id=42751726 https://news.ycombinator.com/item?id=42751726
- aydyn 1y agoNext they are going to rename Windows to Microsoft Copilot.
- onemoresoop 1y ago
- quantadev 1y agoI love Copilot in VSCode. I always select model "Claude Sonnet 3.7", when in Copilot since it lets me choose the LLM. What I love about Copilot is the tight integration with VSCode. I can just ask it to do something and it relies on the intelligence of Claude to get the right code generated, and then all Copilot is really doing is editing my code for me, reading whatever code Claude tells it to, to build context, etc.
- KoolKat23 1y agoThat's a different, more useable copilot.
- quantadev 1y agoThat's why I said "in VSCode" because I have no idea what this guy is running, but it's almost a certainty the problem isn't copilot but it's a bad LLM and/or his bad prompt.
- kemotep 1y agoThe Copilot integrated with Microsoft 365 doesn’t have a model switcher it just is what it is. You are talking about a completely different product that Microsoft calls the same names.
- quantadev 1y agoI'll say it for the third time: "in VSCode". There's no ambiguity about which Copilot that is.
- willhslade 1y agoVSCode Copilot or Copilot Chat?
- quantadev 1y ago
- FergusArgyll 1y agoPeople think that llms are an excuse to be lazy. You have to put some effort into the prompt!
- KoolKat23 1y agoIronically copilot is lazy, have to prompt it to death, whereas the others are aligned and actually provide answers with the same prompt. (Perhaps copilot is not lazy, just stupid relative to its peers.)
- roywiggins 1y agoNo, you don't. ChatGPT easily answers this question posed verbatim.
- moelf 1y agoeven gemma3:12b gets it correct: ~> ollama run gemma3:12b-it-qat >>> ffmpeg command to convert movie.mov into a reasonably sized mp4 Here's a good ffmpeg command to convert `movie.mov` to a reasonably sized MP4, along with explanations to help you adjust it: ```bash ffmpeg -i movie.mov -c:v libx264 -crf 23 -preset medium -c:a aac -b:a 128k movie.mp4 ``` *Explanation of the command and the options:*
- magicalhippo 1y agoEven the 1B variant gave me that one, along with good explanations of the various options and what to play with to tweak the result.
- cmrx64 1y agoi’m pretty surprised 1B parameters is enough for it to still fluently remember ffmpeg-fu
- magicalhippo 1y agoYeah the newer small models continue to surprise me as well. I uploaded the full output from gemma3:1b-it-q8_0 here[1]. [1]: https://rentry.co/yu36i4d3 https://rentry.co/yu36i4d3
- what-the-grump 1y agoI cannot reproduce this in any version of copilot? Copilot with outlook.com Copilot base one that comes with M365, And the add-on one for 30$/mo. Copilot in VS code All produce: ffmpeg -i movie.mov -vcodec libx264 -crf 23 -preset medium -acodec aac -b:a 128k output.mp4 Which is not surprising because its just an Open AI 4o call... so how are you getting this?
- svachalek 1y agoCopilot is not OpenAI
- adastra22 1y agoIt is OpenAI weights under the hood, unless something changed recently?
- deleted 1y ago[deleted]
- 0points 1y agoCopilot is OpenAI in branding since day 1. https://en.wikipedia.org/wiki/GitHub_Copilot#Implementation https://en.wikipedia.org/wiki/GitHub_Copilot#Implementation
- District5524 1y agoThe pipeline Microsoft is using for Copilot products hides actually what models they are using, and you have no influence over it. Sometimes they use smaller models, but I have no clear source from Microsoft saying this ...
- NordSteve 1y agoI pasted your prompt: ffmpeg command to convert movie.mov into a reasonably sized mp4 into the Copilot app just now. Here's the response: ffmpeg -i movie.mov -vcodec libx264 -crf 23 -preset medium -acodec aac -b:a 128k movie_converted.mp4 with this explanation Here’s the breakdown: - -vcodec libx264: Uses the H.264 video codec, great for compression and compatibility. - -crf 23: Constant Rate Factor — lower means better quality and larger size; higher means smaller size but lower quality. 23 is a nice middle ground. - -preset medium: Controls encoding speed vs. compression efficiency. You can try slow or faster depending on your needs. - -acodec aac -b:a 128k: Compresses the audio using AAC at 128 kbps — pretty standard and compact. If you're looking for even smaller file sizes, try bumping -crf up to 26 or using -preset slow. Want to bake in scaling, remove metadata, or trim the video too? Happy to tailor the command further.
- glenstein 1y agoOne fascinating aspect of LLMs is they make out-in-the-wild anecdotes instantly reproducible or, alternatively, comparable to results from others with different outcomes. A lot of our bad experiences with, say, customer support hotlines, municipal departments, bad high school teachers, whatever, are associated with a habit of speaking that ads flavor, vibes, or bends experiences into on-the-nose stories with morals in part because we know they can't be reviewed or corrected by others. Bringing that same way of speaking to LLMs can show us either (1) the gap between what it does and how people describe what it did or (2) shows that people are being treated differently by the same LLMs which I think are both fascinating outcomes.
- myhf 1y agoWe're also seeing a new variant of Cunningham's law: The best way to get the right answer from an LLM is not to ask it the right question; it's to post online that it got the wrong answer.
- celeritascelery 1y agoLLMs are definitely not instantly reproducible. The temperature setting adjust randomness and the models are frequently optimized and fine tuned. You will very different results depending on what you have in your context. And with a tool like Microsoft copilot, you have no idea what is in the context. There are also bugs in the tools that wrap the LLM. Just because other people on here say “worked for me” doesn’t invalidate OPs claim. I have had similar times where an LLM will tell me “here is a script that does X” and there is no script to be found.
- vel0city 1y agoI put your exact prompt into Copilot and it gave me the command ffmpeg -i movie.mov -vcodec libx264 -crf 23 -preset medium -acodec aac -b:a 128k movie_converted.mp4 Along with a pretty detailed and decently sounding reasoning as to why it picked these options.
- 12345hn6789 1y agoIt's been increasingly more obvious people on hacker news literally do not run these supposed prompts through LLMs. I bet you could run that prompt 10 times and it would never give up without producing a (probably fine) sh command. Read the replies. Many folks have called gpt-4.1 through copilot and get (seemingly) valid responses.
- Deukhoofd 1y agoWhat is becoming more obvious is that people on Hacker News apparently do not understand the concept of non-determinism. Acting as if the output of an LLM is deterministic, and that it returns the same result for the same prompt every time is foolish.
- 12345hn6789 1y agoRun the prompt 100 times. I'll wait. I'll estimate you won't get a shell command 1-2% of the time. Please post snark on reddit. This site is for technical discussion.
- rzzzt 1y agoYour conversations are notebooks and the code it conjured up should be behind a dropdown arrow. For visualization it seems to work fine (i.e. Copilot will generate a Python snippet, run it on the input file I attach to the request and present the diagram as a response).
- esskay 1y agoThis is pretty interesting, I had a very similar experience with Github Copilot's plugin inside Jetbrains IDE today (set to use 4o). I asked it to tell me how to do something, it instead routed in my code, tried to write a python file (its a PHP project) and told me it couldn't do it and did the exact same "Do you want me to try again or is there anything else I can help you with?" Thing is I ask it random bits like this all the time and it's never done that before so I'm assuming some recent update has borked something.
- hangonhn 1y agoThis is hilarious because both Gemini and ChatGPT are shockingly good at putting together FFMPEG commands. They can both put together and also understand the various options and stages/filters.
- canyp 1y agoI really like the final remark, "or is there anything else that I can help you with"? Yeah, like how about answering the fucking question? lol
- deleted 1y ago[deleted]
- 35123123 1y agosed, awk, docker, ffmpeg, etc... are problebly the most Google-ed commands. It's kinda shocking bad that MS LLM model is bad at this .
- cedws 1y agoMicrosoft has really screwed up on branding yet again. Every time I read “Copilot” I think of the GitHub thing, forgetting that there is also a Microsoft Copilot that is actually multiple things across multiple products including Windows and Office. It’s also a website like ChatGPT apparently? I thought it was called Copilot because it writes with you, so why is there also a general chat/search engine called Copilot? Jesus.
- thedougd 1y agoI think you may be confusing Microsoft Copilot with Microsoft365 Copilot? The first doesn’t give you access to Microsoft Copilot Studio but that might also be available with Microsoft Copilot Pro.
- cedws 1y agoMy confusion has only increased.
- jug 1y agoThey have even renamed Office to Microsoft 365 Copilot. Yes. Microsoft Office. https://www.windowslatest.com/2025/01/18/microsoft-just-renamed-office-to-microsoft-365-copilot-on-windows-11-for-everyone/ https://www.windowslatest.com/2025/01/18/microsoft-just-rena...
- mrweasel 1y agoYeah that's not going to work.
- cubefox 1y agoI believe you might be confusing Copilot Search with Copilot+? Which is of course different from Copilot Pro though not necessarily entirely distinct.
- worldsayshi 1y agoAnd Sam Altman thought that they were bad at naming things. Good thing they were bought up by the king of naming things. /s Edit: They are doubling down on bad naming conventions so hard that it makes me think it's some kind of dark pattern sales strategy..
- jay_kyburz 1y agoPeople are responding with..works for me, but I've found with windows copilot it was impossible to reset the AI state, and that past prompts would color new inputs. The new chat, or new conversation buttons seem to do nothing.
- wkat4242 1y agoStandard copilot indeed sucks but I'm quite fond of the new researcher agent. It spends much more time than any of the others I've tried, like Perplexity pro and openai. From a one line question it made me a relevant document of 45 pages examining the issue from all different sides, many of which I hadn't even thought of. It spent 30 mins working. I've never seen Perplexity spend more than 5. I won't't be surprised if they will significantly nerf it to save on computing costs. I think now they give it their all to build a customer base and then they nerf it.
- pid-1 1y agoI think the biggest issue is M365 Copilot was sold as something that would integrate with business data (teams, files, mail, etc.) and that never worked out quite well. So you end up with a worse ChatGPT that also doesn't have work context.
- baxtr 1y agoSame here (MS Shop). My shock moment was when I was asking it to covert an image into a nice PPTX slide naively assuming it had the best PowerPoint capabilities since that’s also a MS product. It returned a non formatted text box on one slide. I had to laugh so hard people on the office noticed.
- TeMPOraL 1y agoYeah, working in an MS shop in the past couple years, that's what I've been saying ever since first iteration Copilot for MS Office came out, and it's true to this very day: you want AI to help you with your MS Office files? Ask ChatGPT. Or get API keys and use some frontend with a half-assed third party plugin that knows how to convert Markdown into .docx or such. Results are still an order of magnitude better than the Copilot in MS Office. I blame it on corporate reluctance to take risks that could result in bad press. They put a lot of work into screwing Copilot up. I mean, they had it running GPT-4 back when GPT-4 was the new hot thing. The way it was comically useless is not something that "just happens" - as evidenced by the fact, that just running GPT-4 via API yourself produced good results by default.
- hypercube33 1y agoThats a good note. I have all of my documentation in markdown (which Microsoft parades on with Github, VSCode, docs.microsoft.com etc) but Copilot cant or wont read these files. I had to pandoc everything over to docx files before it even saw them. Pretty wild. Also in line with this, Copilot 365 seems to not get how charts work. I asked it with multiple different massaged data sets and it kept giving worse and worse answers even after resetting the chat and data to as simple as possible (think 10 dates, 2 colums of integers) until it ultimately gave me a blank chart. I gave up and asked GPT.
- DannyBee 1y agoGemini-in-sheets is the same way i asked it to make a sheet look nicer and it couldn't without me explicitly telling it what i wanted done. When i told it to do certain things, it's like "that's not implemented yet, stay tuned!"
- 1y ago
- dakiol 1y agoIn my experience Microsoft Copilot (free version in Deep Think mode) is way better than ChatGPT (free version) in most of things I throw at them (improving text, generating code, etc).
- IronyMan100 1y agoi asked copilot to make an excel command, that rounds up all numbers to the next integer... took 4 back and forth messages and 15 minutes until it was working... Google was 5 minutes
- mrangle 1y agoI don't have experience with CoPilot, but I do with other LLMs. I'm not sure that omitting "provide me with" is enough to get the job done, generally, aside from being lucky that it correctly interprets the prompt. In my experience, other LLMs are just as prone to incorrect divination of what one means given telegraphic prompts.
- karn97 1y ago[dead]
- hluska 1y agoFirst off, that’s a really bad prompt - LLMs don’t have this magic ability to read your mind. Second, despite how bad it is, Copilot just gave me the correct code.