4 ms·

The other day I've clicked on one of Outlook calendar's copilot prefilled questions: "who are the main attendees of this meeting". It started a long winding spe

The other day I've clicked on one of Outlook calendar's copilot prefilled questions: "who are the main attendees of this meeting". It started a long winding speech that went nowhere, so I typed in "but WHO are the attendees" and finally it admitted "I don't know, I can't see that".

ZeroConcerns 11mo ago
Absolutely! There are so many scenarios where they could actually add some value, and they're fulfilling, like, exactly none of those? Even in Visual Studio Enterprise, their flagship developer product, the GPT integration mostly just destroys code regardless of model output. I truly cannot fathom how any of that made it past even a cursory review. Or how that situation would last for over 6 months, but, yet, here we are. And, again, it's fine with me: I'll just use Claude Code, but if I were a Microsoft VP-or-above, the lack of execution would sort-of, well concern me? But maybe I'm just focused on the wrong things. I mean, Cloudflare brought down, like, half the Internet twice in the past two weeks, and they're still a tech darling, so possibly incompetence is the new hotness now?
- Sanzig 11mo ago
  I have Copilot at work, it feels so useless sometimes. As an example, I had a report which I needed to make some batch edits to. I figured why not let the robot take a crack at it, so I clicked the Copilot button and spent a couple minutes describing what I needed changed. Copilot tells me it can't edit my current document, but it can create a new one. I figured okay, Microsoft doesn't want to set it loose on the original, guess it makes sense that it requires a copy. So I said yes. Nope. Instead of creating a copy of my document and editing it, it created an entirely new document which excised basically everything in the original report and replaced it with a very short summary - I'm talking 5000 words down to 500. All my tables and figures were gone, as was the standard report template my employer uses. What utter garbage. Office productivity is a major use case for LLMs, and here the largest vendor of productivity software on the planet is happy to fuck it up.
  - KurSix 11mo ago
    The chatbot doesn't see the document structure; it sees tokens. When it generates a response, it's not editing; it's rewriting from scratch. Without a mechanism capable of precise diff/patch operations (like Cursor attempts with code), an LLM will always be a destructive tool for complex documents. Microsoft marketed this as an editor, but in reality, it's a summary generator
- Angostura 11mo ago
  I’ve found it fairly useful in Excel. The suggestions to clean up data are pretty good and it’s spat out some quite gnarly formula on request
- danudey 11mo ago
  Imagine a circumstance where Windows Search was as good as Apple's Spotlight, and could integrate with cloud services to index documents, browser bookmarks, web history, maybe podcasts, etc.) Hey Copilot, where is that document I was reading about the new network diagramming software Jacob is testing out? Or Hey Copilot, my disk is getting pretty full. What software is taking up a lot of space that I haven't used for a while? Or are there any files I can move to cloud storage to free up space? But no, instead it's just 'we're going to take screenshots of all your windows, OCR it, and index it, so that when someone infects your machine they can see your credit card numbers and pornography habits.'
- rickydroll 11mo ago
  > I truly cannot fathom how any of that made it past even a cursory review. Maybe it's a fifth column group working to destroy Microsoft.
- KellyCriterion 11mo ago
  LOL Since DECADES Outlook is _not_ capable of putting the first(!) just(!) useful datetime-stamp found in an email which says "calendar" or something in the text when pressing CTRL+R: This function creates a calendar entry from the current email with current receiver list, but default-date/time is alwsys set to the current even if recipients last ansewer was simply "lets do: 11am" - why does even this simple shit not work today? You dont need AI for this simple idea even, a robus / probability parser will do fine in 99% cases
  - jll29 11mo ago
    Agree 100%. Don't train a 100-billion parameter neural transformer if you can solve it with a bash one-liner. Use your human intelligence and save our planet!
  - mlrtime 11mo ago
    I would just be happy to never have a ICS file every saved to my drive again. Whoever created the concept of a calendar invite saving a ICS file has created me hours of frustration.
- TeMPOraL 11mo ago
  It's so broken. I have a suspicion the AI teams are in some kind of an internal standoff with privacy/compliance teams, and Copilot simply never gets access to any tools or data. Because if not that, then I don't know what. I can half-ass a better product with ChatGPT API and a PowerShell script, and I could've since GPT-4 was released; in fact the product can actually write itself, it's that simple to do a better job.
  - ethbr1 11mo ago
    Being Microsoft, I'd guess it's less that and more Product Team A doesn't want to spend time building enabling features for Product Team B to look good. The org chart joke, etc: https://www.reddit.com/r/ProgrammerHumor/comments/6jw33z/internal_structure_of_tech_companies/#lightbox https://www.reddit.com/r/ProgrammerHumor/comments/6jw33z/int...
  - fragmede 11mo ago
    > the product can actually write itself, Used to be, that was just something people said. Turns out, we live in a future where that's literally true!
- KurSix 11mo ago
  Compare this with Cursor or Supermaven. There AI acts as an extension of the cursor, understanding the code's AST. VS Copilot often feels like smart copy-paste that doesn't understand where it is Microsoft, owning VS Code and LSP, should have done this better than anyone, but they lost to startups on UX
HarHarVeryFunny 11mo ago
Sounds like Siri - unable to control much of anything on the iPhone outside of reading/sending text messages and setting alarms.
- donkey_brains 11mo ago
  And that’s ok. Those are core features of the phone that absolutely must work reliably and consistently. Far better to do a few important things really well than a hundred things execrably.
  - throw310822 11mo ago
    However the other day I asked the Gemini assistant on my phone to check the birthdays in my calendar, get all their dates, then make a graph of how many fall in each period with a 15-day moving average. It did everything as instructed including writing a python script to generate the graph, then discussed the results with me :)
  - HarHarVeryFunny 11mo ago
    I would expect Siri to be able to do anything on the iPhone that I can - change settings, report stats, kill/launch apps, etc. It would be nice if it could control 3rd party apps too, like GMail, but being able to control the stuff that Apple themselves have built doesn't seem a lot to ask.
    - array_key_first 11mo ago
      Apple is somewhat privacy focused, allegedly. I would imagine that letting an unpredictable LLM read whatever system data it wants is not something they want to partake in.
      - zaphirplane 11mo ago
        We all know they Apple lack the capability, not a choice.
      - 11mo ago
burningChrome 11mo ago
I've fooled around with some vibe coding on several LLM's like Claude, Gemini, and ChatGPT with some pretty decent results. Since I have a full Copilot license at my corporate day gig, I figured I would try using Copilot for a basic static site. Nothing too hard, and something that's been handled easily with the other LLM's. The prompt was pretty basic just to get something to start working with. "Build a four page template. With a home or index page, two pages of content and a contact page with a responsive slide out menu from the left hand side of the page." It ran and put everything in a folder. I open the home page and everything was broken. I opened the files in VS Code and saw this: <ul class="drawer__list"> <li>index.htmlHome</a></li> <li>services.htmlServices</a></li> <li><a class="nav-linkeduling</a></li> <li>contact.htmlContact</a></li> </ul> And then this: <head> <meta charset="utf-8" /> <meta name="viewport" content="width=device-width, initial-scale=1" /> <title>Home · Acme Web</title> <meta name="description" content="Accessible, responsive starter template with a slide-out menu."/> <linkts/css/styles.css /assets/css/styles.css </head> I mean, if you can't even this right, I don't have much hope it can do anything more complicated. To say this was pretty sad is an understatement and clarified how far Microsoft is behind other LLM's.
- imiric 11mo ago
  Not a problem. We can call it HTML6 and make browsers interpret it correctly.
- raxxorraxor 11mo ago
  I guess this is code people working at MS would commit to a useless file in order to simulate work done with AI support. There are cheap 1b open coding models that do a far better job.
estetlinus 11mo ago
Kudos to you. I never use new buttons out of fear of something irreversible happening, like sending a random email or deleting something. I still feel uncomfortable with the Gmail UX, I would _never_ use a ”hello iz magic ai”-button.
outside2344 11mo ago
I asked Microsoft 365 Copilot to create a new word document for me (since they have hidden the link on office.com) and... it refused to do that. Edit: Just tried again. It refused to do it. I mean WTF.
artrockalter 11mo ago
It's so easy to ship completely broken AI features because you can't really unit test them and unit tests have been the main standard for whether code is working for a long time now. The most successful AI companies (OpenAI, Anthropic, Cursor) are all dogfooding their products as far as I can tell, and I don't really see any other reliable way to make sure the AI feature you ship actually works.
- arresin 11mo ago
  Microsoft: What? You want us to eat this slop? Are you crazy?!
  - danudey 11mo ago
    50% of our code is being written by AI! Or at least, autocompleted by AI. And then our developers have to fix 50% of THAT code so that it does what they actually wanted to do in the first place. But boy, it sure produces a lot of words!
- sk7 11mo ago
  Tests are called "evals" (evaluations) in the AI product development world. Basically you let humans review LLM output or feed it to another LLM with instructions how to evaluate it. https://www.lennysnewsletter.com/p/beyond-vibe-checks-a-pms-complete https://www.lennysnewsletter.com/p/beyond-vibe-checks-a-pms-...
  - azemetre 11mo ago
    Interesting, never really thought of it outside of this comment chain but I'm guessing approaches like this hurt the typical automated testing devs would do but seeing how this is MSFT (who already stopped having dedicated testing roles for a good while now, rip SDET roles) I can only imagine the quality culture is even worse for "AI" teams.
    - ethbr1 11mo ago
      Yes. Because why would there ever be a problem with a devqaops team objectively assessing their own work's effectiveness?
flkiwi 11mo ago
Me: Can you access my inbox and Teams messages? Copilot: Yep! Me: Please find any items in my inbox or sent items indicating (a) that I have agreed to take on a task or (b) identifying me as the person responsible for a task, removing duplicates and any items that I have unambiguously replied to via email or Teams. Time window is preceding 7 days. Copilot: Prints a list with, at best, 5% accuracy I know some folks have the peculiar idea that search is dead in favor of AI, but if AI can't accurately find information, it is useless. As near as I can tell, Copilot finds 3-4 items (but rarely the SAME 3-4 items across runs) and calls it a day. It just seems like nobody is actually testing any of this stuff. Microsoft is actively destroying its credibility because it's offering a tool with a party trick but is utterly unreliable. I will, therefore, not rely on it.
- PLenz 11mo ago
  It's a generalization problem. We can train LLMs that 'know' a lot of stuff in the global sense but the tasks that are interesting to people require the LLM to know a lot about you and your world in a very specific sense. The technical problem is that it's all corner cases and that's impossible to scale right now. No amount of context window is going to get you there either.
  - JamesSwift 11mo ago
    I respectfully disagree. The scenario posted by the GP is very clearly an "easy" task of basically sentiment analysis. The only thing it requires is a way to query your message history, then it just goes through them 1-by-1. I would expect any of the current SOTA agentic models to handle this with ease.

ZeroConcerns 11mo ago

Absolutely! There are so many scenarios where they could actually add some value, and they're fulfilling, like, exactly none of those? Even in Visual Studio Enterprise, their flagship developer product, the GPT integration mostly just destroys code regardless of model output. I truly cannot fathom how any of that made it past even a cursory review. Or how that situation would last for over 6 months, but, yet, here we are. And, again, it's fine with me: I'll just use Claude Code, but if I were a Microsoft VP-or-above, the lack of execution would sort-of, well concern me? But maybe I'm just focused on the wrong things. I mean, Cloudflare brought down, like, half the Internet twice in the past two weeks, and they're still a tech darling, so possibly incompetence is the new hotness now?

Sanzig 11mo ago
I have Copilot at work, it feels so useless sometimes. As an example, I had a report which I needed to make some batch edits to. I figured why not let the robot take a crack at it, so I clicked the Copilot button and spent a couple minutes describing what I needed changed. Copilot tells me it can't edit my current document, but it can create a new one. I figured okay, Microsoft doesn't want to set it loose on the original, guess it makes sense that it requires a copy. So I said yes. Nope. Instead of creating a copy of my document and editing it, it created an entirely new document which excised basically everything in the original report and replaced it with a very short summary - I'm talking 5000 words down to 500. All my tables and figures were gone, as was the standard report template my employer uses. What utter garbage. Office productivity is a major use case for LLMs, and here the largest vendor of productivity software on the planet is happy to fuck it up.
- KurSix 11mo ago
  The chatbot doesn't see the document structure; it sees tokens. When it generates a response, it's not editing; it's rewriting from scratch. Without a mechanism capable of precise diff/patch operations (like Cursor attempts with code), an LLM will always be a destructive tool for complex documents. Microsoft marketed this as an editor, but in reality, it's a summary generator
Angostura 11mo ago
I’ve found it fairly useful in Excel. The suggestions to clean up data are pretty good and it’s spat out some quite gnarly formula on request
danudey 11mo ago
Imagine a circumstance where Windows Search was as good as Apple's Spotlight, and could integrate with cloud services to index documents, browser bookmarks, web history, maybe podcasts, etc.) Hey Copilot, where is that document I was reading about the new network diagramming software Jacob is testing out? Or Hey Copilot, my disk is getting pretty full. What software is taking up a lot of space that I haven't used for a while? Or are there any files I can move to cloud storage to free up space? But no, instead it's just 'we're going to take screenshots of all your windows, OCR it, and index it, so that when someone infects your machine they can see your credit card numbers and pornography habits.'
rickydroll 11mo ago
> I truly cannot fathom how any of that made it past even a cursory review. Maybe it's a fifth column group working to destroy Microsoft.
KellyCriterion 11mo ago
LOL Since DECADES Outlook is _not_ capable of putting the first(!) just(!) useful datetime-stamp found in an email which says "calendar" or something in the text when pressing CTRL+R: This function creates a calendar entry from the current email with current receiver list, but default-date/time is alwsys set to the current even if recipients last ansewer was simply "lets do: 11am" - why does even this simple shit not work today? You dont need AI for this simple idea even, a robus / probability parser will do fine in 99% cases
- jll29 11mo ago
  Agree 100%. Don't train a 100-billion parameter neural transformer if you can solve it with a bash one-liner. Use your human intelligence and save our planet!
- mlrtime 11mo ago
  I would just be happy to never have a ICS file every saved to my drive again. Whoever created the concept of a calendar invite saving a ICS file has created me hours of frustration.
TeMPOraL 11mo ago
It's so broken. I have a suspicion the AI teams are in some kind of an internal standoff with privacy/compliance teams, and Copilot simply never gets access to any tools or data. Because if not that, then I don't know what. I can half-ass a better product with ChatGPT API and a PowerShell script, and I could've since GPT-4 was released; in fact the product can actually write itself, it's that simple to do a better job.
- ethbr1 11mo ago
  Being Microsoft, I'd guess it's less that and more Product Team A doesn't want to spend time building enabling features for Product Team B to look good. The org chart joke, etc: https://www.reddit.com/r/ProgrammerHumor/comments/6jw33z/internal_structure_of_tech_companies/#lightbox https://www.reddit.com/r/ProgrammerHumor/comments/6jw33z/int...
- fragmede 11mo ago
  > the product can actually write itself, Used to be, that was just something people said. Turns out, we live in a future where that's literally true!
KurSix 11mo ago
Compare this with Cursor or Supermaven. There AI acts as an extension of the cursor, understanding the code's AST. VS Copilot often feels like smart copy-paste that doesn't understand where it is Microsoft, owning VS Code and LSP, should have done this better than anyone, but they lost to startups on UX

HarHarVeryFunny 11mo ago

Sounds like Siri - unable to control much of anything on the iPhone outside of reading/sending text messages and setting alarms.

donkey_brains 11mo ago
And that’s ok. Those are core features of the phone that absolutely must work reliably and consistently. Far better to do a few important things really well than a hundred things execrably.
- throw310822 11mo ago
  However the other day I asked the Gemini assistant on my phone to check the birthdays in my calendar, get all their dates, then make a graph of how many fall in each period with a 15-day moving average. It did everything as instructed including writing a python script to generate the graph, then discussed the results with me :)
- HarHarVeryFunny 11mo ago
  I would expect Siri to be able to do anything on the iPhone that I can - change settings, report stats, kill/launch apps, etc. It would be nice if it could control 3rd party apps too, like GMail, but being able to control the stuff that Apple themselves have built doesn't seem a lot to ask.
  - array_key_first 11mo ago
    Apple is somewhat privacy focused, allegedly. I would imagine that letting an unpredictable LLM read whatever system data it wants is not something they want to partake in.
    - zaphirplane 11mo ago
      We all know they Apple lack the capability, not a choice.
    - 11mo ago

burningChrome 11mo ago

I've fooled around with some vibe coding on several LLM's like Claude, Gemini, and ChatGPT with some pretty decent results. Since I have a full Copilot license at my corporate day gig, I figured I would try using Copilot for a basic static site. Nothing too hard, and something that's been handled easily with the other LLM's. The prompt was pretty basic just to get something to start working with. "Build a four page template. With a home or index page, two pages of content and a contact page with a responsive slide out menu from the left hand side of the page." It ran and put everything in a folder. I open the home page and everything was broken. I opened the files in VS Code and saw this: <ul class="drawer__list"> <li>index.htmlHome</a></li> <li>services.htmlServices</a></li> <li><a class="nav-linkeduling</a></li> <li>contact.htmlContact</a></li> </ul> And then this: <head> <meta charset="utf-8" /> <meta name="viewport" content="width=device-width, initial-scale=1" /> <title>Home · Acme Web</title> <meta name="description" content="Accessible, responsive starter template with a slide-out menu."/> <linkts/css/styles.css /assets/css/styles.css </head> I mean, if you can't even this right, I don't have much hope it can do anything more complicated. To say this was pretty sad is an understatement and clarified how far Microsoft is behind other LLM's.

imiric 11mo ago
Not a problem. We can call it HTML6 and make browsers interpret it correctly.
raxxorraxor 11mo ago
I guess this is code people working at MS would commit to a useless file in order to simulate work done with AI support. There are cheap 1b open coding models that do a far better job.

estetlinus 11mo ago

Kudos to you. I never use new buttons out of fear of something irreversible happening, like sending a random email or deleting something. I still feel uncomfortable with the Gmail UX, I would _never_ use a ”hello iz magic ai”-button.

outside2344 11mo ago

I asked Microsoft 365 Copilot to create a new word document for me (since they have hidden the link on office.com) and... it refused to do that. Edit: Just tried again. It refused to do it. I mean WTF.

artrockalter 11mo ago

It's so easy to ship completely broken AI features because you can't really unit test them and unit tests have been the main standard for whether code is working for a long time now. The most successful AI companies (OpenAI, Anthropic, Cursor) are all dogfooding their products as far as I can tell, and I don't really see any other reliable way to make sure the AI feature you ship actually works.

arresin 11mo ago
Microsoft: What? You want us to eat this slop? Are you crazy?!
- danudey 11mo ago
  50% of our code is being written by AI! Or at least, autocompleted by AI. And then our developers have to fix 50% of THAT code so that it does what they actually wanted to do in the first place. But boy, it sure produces a lot of words!
sk7 11mo ago
Tests are called "evals" (evaluations) in the AI product development world. Basically you let humans review LLM output or feed it to another LLM with instructions how to evaluate it. https://www.lennysnewsletter.com/p/beyond-vibe-checks-a-pms-complete https://www.lennysnewsletter.com/p/beyond-vibe-checks-a-pms-...
- azemetre 11mo ago
  Interesting, never really thought of it outside of this comment chain but I'm guessing approaches like this hurt the typical automated testing devs would do but seeing how this is MSFT (who already stopped having dedicated testing roles for a good while now, rip SDET roles) I can only imagine the quality culture is even worse for "AI" teams.
  - ethbr1 11mo ago
    Yes. Because why would there ever be a problem with a devqaops team objectively assessing their own work's effectiveness?

flkiwi 11mo ago

Me: Can you access my inbox and Teams messages? Copilot: Yep! Me: Please find any items in my inbox or sent items indicating (a) that I have agreed to take on a task or (b) identifying me as the person responsible for a task, removing duplicates and any items that I have unambiguously replied to via email or Teams. Time window is preceding 7 days. Copilot: Prints a list with, at best, 5% accuracy I know some folks have the peculiar idea that search is dead in favor of AI, but if AI can't accurately find information, it is useless. As near as I can tell, Copilot finds 3-4 items (but rarely the SAME 3-4 items across runs) and calls it a day. It just seems like nobody is actually testing any of this stuff. Microsoft is actively destroying its credibility because it's offering a tool with a party trick but is utterly unreliable. I will, therefore, not rely on it.

PLenz 11mo ago
It's a generalization problem. We can train LLMs that 'know' a lot of stuff in the global sense but the tasks that are interesting to people require the LLM to know a lot about you and your world in a very specific sense. The technical problem is that it's all corner cases and that's impossible to scale right now. No amount of context window is going to get you there either.
- JamesSwift 11mo ago
  I respectfully disagree. The scenario posted by the GP is very clearly an "easy" task of basically sentiment analysis. The only thing it requires is a way to query your message history, then it just goes through them 1-by-1. I would expect any of the current SOTA agentic models to handle this with ease.