5 ms·

I recently saw a (pre-recorded) demo of M365 copilot. Here's some stuff it covered that the article doesn't attempt: * Asking Power BI to analyze some employee

I recently saw a (pre-recorded) demo of M365 copilot. Here's some stuff it covered that the article doesn't attempt: * Asking Power BI to analyze some employee data and generate a Power BI report page * Asking Power BI to re-theme the page to match a different Power BI report * Asking Power BI to refine some of the page content to answer a particular question about retention * Asking Word to generate draft minutes from a Teams meeting transcript * Asking Power Apps to create an app and iteratively build out features * Asking Power Automate to map out an automation flow This article seems to be focused on a very narrow understanding of Copilot's capabilities. I'm waiting to see if the above use cases are fact or fiction.

rf15 3y ago
People will get fired due to hallucinated bad reports, won't they.
- ryanjshaw 3y ago
  Microsoft specifically point out it's named "copilot" because it's there to assist you, not replace you. You still need to check the work and decide whether to put your name behind it or not.
  - ourmandave 3y ago
    This is emphasized in every "What is Copilot?" article or video I've seen. The not so fine print is Copilot may return garbage and it's up to you to confirm the results.
- latency-guy2 3y ago
  Well, I wouldn't really expect it. Its customer data, and it can technically be anything. Imagine making a system that is expected to be reliably excellent at generating things within that domain. It is already a very hard problem that I can confidently say is unsolved, and probably not even as good as a human can sort things. Even in a good dataset that is unstructured and unlabelled you will face issues in just a single domain (e.g. vision tasks trying to classify a variety of vehicles), imagine that being extended to a wide variety of subjects that can mix and match. I imagine at most, it'll be a reputation hit.
- Delphiza 3y ago
  Machine-generated or people-generated hallucinations? You would be surprised at how bad most existing reports in business are - and people don't get fired for those.
  - thuuuomas 3y ago
    This tedious, snarky response is neither clever nor insightful. Like, we get it - you resent your coworkers & consider them beneath you. Hopefully, your unique, elite output keeps you employable for a few quarters longer than them.
    - ehutch79 3y ago
      No, it’s a legit gripe. I’m routinely asked to make reports, but things stall when I ask, “so what do you actually need to see on the report, and where should the data come from”. A lot of people out there are kind of cargo cutting their way through things.
      - oblio 3y ago
        > cargo cutting Cargo culting. Though cargo cutting would be interesting to watch :-)
        ehutch79 3y ago
        ooops. Missed that typo.
    - andybak 3y ago
      I didn't read the comment as especially snarky. Your response was both snarky and personal which I think comes across much worse.
    - hdhuwgdue2 3y ago
      I don't think parent was trying to be clever. In my neck of the woods various reports have to be massaged hard to account for some of stupid data decisions in the old times. You would be amazed how many issued we identified so far and our mgmt uncharacteristically wants us to bring up issues. Some companies kill their messengers. So yeah.. I absolutely buy that a portion of business reports are of low quality.
- eviks 3y ago
  People are already killed and jailed due to garbage machine-generated data, so why would firing be off the dump?
latency-guy2 3y ago
You may have seen a video that was more of a "vision" rather than actual current capability, e.g. I know that PowerBI Copilot does not support re-theme. I also know that Power BI Copilot does not look at your actual data within your dataset, but does make use of tables, relationships, measures, and column names
- ryanjshaw 3y ago
  It's possible, that's why I want to see the real thing or a review that tests these claims.
nicklecompte 3y ago
The article is based on the author's own use case and understanding of Copilot's marketing. I don't think he used Copilot inappropriately, he just had an uncommon need that the tool badly failed to meet. I will add that Robichaux is far more diligent and skeptical of Copilot than most people who use it, including Microsoft. MSFT's public Bing demo had a ton of awful mistakes, which is shocking: I assumed MSFT would have faked the demo! The fact that they didn't suggests they have a magical and undeserved belief in OpenAI's technology. I suspect the demo you watched also had a ton of mistakes that went unnoticed - if Power BI makes a pretty graph with a plausible trendline, how many people are really going to go back and check the underlying code, making sure it's doing the right thing to the right data sources? If it generates draft minutes from a meeting transcript, but screws up 5% of the items, those errors likely won't be noticed in a quick spot-check, and will end up misleading people who didn't attend the meeting. Robichaux's post brings up a ton of issues that affect any use case - the tool just is not reliable.
doubloon 3y ago
Its a fact. You can open Power Automate Cloud which is basically Scratch for Grown Ups and type in “every day at 3 pm please check if i have an email from Jane Doe and if not then save a file to my cloud drive called “noemail.txt” and it basically does a halfway decent job , not chatgpt level but its getting there
dfxm12 3y ago
The article is about Microsoft 365 copilot. Copilot for power automate or power bi are separate products. This is probably poor marketing by MS. Additionally, they've rebranded their chatbots as copilots, too. If you're in IT, and your users are asking for "copilot", good luck. In my experience with copilot for power automate, the demos you see are mostly smoke and mirrors. I tried to recreate the demo as exactly as I could, I couldn't. Maybe they can help absolute beginners, but if you know the first thing about power automate, you're better off with a web search than asking the copilot for help. This is still v1 though (if it's not still technically a "preview"), but it really should be marketed so hard because it is currently not delivering.
probich0 3y ago
I didn't test any of these use cases directly. In the sandbox I have access to, I don't have any Power BI data (or Teams transcripts for that matter), and I'm not willing to move any of my real production data into it. Out of those use cases, though, four of them are just using Copilot as a natural-language front end (e.g. re-theming in Power BI). I would expect that to work well and, frankly, don't think it's that interesting as a use case. Sort of like Copilot for Windows… why do I want a natural-language system to tell me where setting X lives?

rf15 3y ago

People will get fired due to hallucinated bad reports, won't they.

ryanjshaw 3y ago
Microsoft specifically point out it's named "copilot" because it's there to assist you, not replace you. You still need to check the work and decide whether to put your name behind it or not.
- ourmandave 3y ago
  This is emphasized in every "What is Copilot?" article or video I've seen. The not so fine print is Copilot may return garbage and it's up to you to confirm the results.
latency-guy2 3y ago
Well, I wouldn't really expect it. Its customer data, and it can technically be anything. Imagine making a system that is expected to be reliably excellent at generating things within that domain. It is already a very hard problem that I can confidently say is unsolved, and probably not even as good as a human can sort things. Even in a good dataset that is unstructured and unlabelled you will face issues in just a single domain (e.g. vision tasks trying to classify a variety of vehicles), imagine that being extended to a wide variety of subjects that can mix and match. I imagine at most, it'll be a reputation hit.
Delphiza 3y ago
Machine-generated or people-generated hallucinations? You would be surprised at how bad most existing reports in business are - and people don't get fired for those.
- thuuuomas 3y ago
  This tedious, snarky response is neither clever nor insightful. Like, we get it - you resent your coworkers & consider them beneath you. Hopefully, your unique, elite output keeps you employable for a few quarters longer than them.
  - ehutch79 3y ago
    No, it’s a legit gripe. I’m routinely asked to make reports, but things stall when I ask, “so what do you actually need to see on the report, and where should the data come from”. A lot of people out there are kind of cargo cutting their way through things.
    - oblio 3y ago
      > cargo cutting Cargo culting. Though cargo cutting would be interesting to watch :-)
      - ehutch79 3y ago
        ooops. Missed that typo.
  - andybak 3y ago
    I didn't read the comment as especially snarky. Your response was both snarky and personal which I think comes across much worse.
  - hdhuwgdue2 3y ago
    I don't think parent was trying to be clever. In my neck of the woods various reports have to be massaged hard to account for some of stupid data decisions in the old times. You would be amazed how many issued we identified so far and our mgmt uncharacteristically wants us to bring up issues. Some companies kill their messengers. So yeah.. I absolutely buy that a portion of business reports are of low quality.
eviks 3y ago
People are already killed and jailed due to garbage machine-generated data, so why would firing be off the dump?

latency-guy2 3y ago

You may have seen a video that was more of a "vision" rather than actual current capability, e.g. I know that PowerBI Copilot does not support re-theme. I also know that Power BI Copilot does not look at your actual data within your dataset, but does make use of tables, relationships, measures, and column names

ryanjshaw 3y ago
It's possible, that's why I want to see the real thing or a review that tests these claims.

nicklecompte 3y ago

The article is based on the author's own use case and understanding of Copilot's marketing. I don't think he used Copilot inappropriately, he just had an uncommon need that the tool badly failed to meet. I will add that Robichaux is far more diligent and skeptical of Copilot than most people who use it, including Microsoft. MSFT's public Bing demo had a ton of awful mistakes, which is shocking: I assumed MSFT would have faked the demo! The fact that they didn't suggests they have a magical and undeserved belief in OpenAI's technology. I suspect the demo you watched also had a ton of mistakes that went unnoticed - if Power BI makes a pretty graph with a plausible trendline, how many people are really going to go back and check the underlying code, making sure it's doing the right thing to the right data sources? If it generates draft minutes from a meeting transcript, but screws up 5% of the items, those errors likely won't be noticed in a quick spot-check, and will end up misleading people who didn't attend the meeting. Robichaux's post brings up a ton of issues that affect any use case - the tool just is not reliable.

doubloon 3y ago

Its a fact. You can open Power Automate Cloud which is basically Scratch for Grown Ups and type in “every day at 3 pm please check if i have an email from Jane Doe and if not then save a file to my cloud drive called “noemail.txt” and it basically does a halfway decent job , not chatgpt level but its getting there

dfxm12 3y ago

The article is about Microsoft 365 copilot. Copilot for power automate or power bi are separate products. This is probably poor marketing by MS. Additionally, they've rebranded their chatbots as copilots, too. If you're in IT, and your users are asking for "copilot", good luck. In my experience with copilot for power automate, the demos you see are mostly smoke and mirrors. I tried to recreate the demo as exactly as I could, I couldn't. Maybe they can help absolute beginners, but if you know the first thing about power automate, you're better off with a web search than asking the copilot for help. This is still v1 though (if it's not still technically a "preview"), but it really should be marketed so hard because it is currently not delivering.

probich0 3y ago

I didn't test any of these use cases directly. In the sandbox I have access to, I don't have any Power BI data (or Teams transcripts for that matter), and I'm not willing to move any of my real production data into it. Out of those use cases, though, four of them are just using Copilot as a natural-language front end (e.g. re-theming in Power BI). I would expect that to work well and, frankly, don't think it's that interesting as a use case. Sort of like Copilot for Windows… why do I want a natural-language system to tell me where setting X lives?