13 ms·
First look at Microsoft 365 Copilot
- wenc 3y agoRAG does not guarantee elimination of hallucination it seems. If the foundation model's training outweighs what it is in the external source, it will still hallucinate. Interestingly ChatGPT-4 gets it right. I tried your first question with ChatGPT-4: User: what’s the single-engine service ceiling of a Baron 55 ChatGPT: The single-engine service ceiling of a Beechcraft Baron 55 is approximately 7,000 feet. This is the maximum altitude at which the aircraft can maintain a specified rate of climb, usually 100 feet per minute, with one engine inoperative.
- brandall10 3y agoA common approach with RAG is to tell it to only used the supplied data. I have yet to see that not work using 3.5, though it doesn’t mean it won’t hallucinate if the injected text does not have the answer to the question.
- Neil44 3y agoWe randomly have it enabled on our 365 tenant (we're an ms partner) and I tried it the other day for help with a Power Automate script. It had a good stab at the problem but didn't quite get there. Maybe with more experience at both prompting and power automate I would have had better luck with it.
- _pdp_ 3y agoThere is a confusion between hallucination and lack of information. RAGs work by finding the top N answers given some query, and then, based on this information, the underlying model tries to make up some text. This is unreasonable, and it will only work out of the box for the most straightforward use cases - i.e., chatting with a single document, but there are limitations. So, I am not surprised that Copilot cannot answer information that is readily available in the right place. It is also unlikely that one can find that document, among many others, on the first try unless one is quoting a specific phrase. The fundamental architecture needs to change. Copilot needs to act more like an agent - i.e., perform multi-step research to find this information and do it fast.
- zacmps 3y agoIt's possible to build RAG pipelines that support answering complex questions over multiple documents, and I don't think I would say the whole architecture needs to change. Doing it fast is another story, LLMs are pretty high latency at the moment.
- _pdp_ 3y agoIf you have any examples I can go through it will be greatly appreciated. My main concern is that all RAGs just look for the top N best matching results. That does not mean that the information is in there.
- lmeyerov 3y agoI'm not sure why all RAG implementations would do that? Ex: Louie.AI will use a combination of tools to answer a question (database queries, RAG vector index lookups, Python, interactive charts, ...), and will often do multiple attempts on the same tool until it decides it has exhausted the immediate use of that tool. Ex: Louie is even learning from usage, so as an analyst, if Louie stopped digging early and you decided to manually look further, Louie learns from this: It'll know in future sessions that it may be worth looking further in that kind of scenario. None of that, in isolation, is unique to Louie.AI: It's just part of what it means to do a 'full' agent implementation vs a langchain/llamaindex/openai wrapper. There's an interesting question around knowledge graph style questions here -- should an agent do iterative top-n vector similarity searches, or a single wider search over a knowledge graph, or maybe the documents should be combined into a knowledge graph and that's what's embedded. We're exploring a lot here in our bigger customer projects, and I can't say there's a clear universal answer...
- davedx 3y agoYup, it depends on the type of data being searched for and how the information is represented in the corpus. It also depends on how information is distributed: IME often longer legal documents have a lot of cross referencing, so you need multiple clauses to collate a single answer. It’s an interesting area.
- boomskats 3y agoGratuitously off topic - but at which point does continuing to call it Microsoft 365 become misleading given all the outages they've had this year?
- seydor 3y agoyou have to account for leap years
- layer8 3y agoBack in 2020 I was disappointed they didn’t rename it to 366.
- nahumfarchi 3y agoExcel is notoriously bad at that :) https://en.m.wikipedia.org/wiki/Leap_year_problem https://en.m.wikipedia.org/wiki/Leap_year_problem
- FirmwareBurner 3y agoSubtract 5 days/year of downtime and call it Microsoft 360, they already own that trademark anyway.
- tgv 3y agoThe Register kept track, and started calling it Microsoft 362, Microsoft 361, etc. https://www.theregister.com/2020/10/01/microsoft_exchange_online_outage/ https://www.theregister.com/2020/10/01/microsoft_exchange_on...
- nicklecompte 3y ago"We're available 24 minutes per hour, 365 hours per year!"
- alexymik2 3y agoI think that if you’re asking for information based on documents or other information in your MS instance, every answer should come with a source or snippet of where it got that info. Otherwise enabling this for the general (office) public will be a disaster.
- ryanjshaw 3y agoI recently saw a (pre-recorded) demo of M365 copilot. Here's some stuff it covered that the article doesn't attempt: * Asking Power BI to analyze some employee data and generate a Power BI report page * Asking Power BI to re-theme the page to match a different Power BI report * Asking Power BI to refine some of the page content to answer a particular question about retention * Asking Word to generate draft minutes from a Teams meeting transcript * Asking Power Apps to create an app and iteratively build out features * Asking Power Automate to map out an automation flow This article seems to be focused on a very narrow understanding of Copilot's capabilities. I'm waiting to see if the above use cases are fact or fiction.
- rf15 3y agoPeople will get fired due to hallucinated bad reports, won't they.
- ryanjshaw 3y agoMicrosoft specifically point out it's named "copilot" because it's there to assist you, not replace you. You still need to check the work and decide whether to put your name behind it or not.
- ourmandave 3y agoThis is emphasized in every "What is Copilot?" article or video I've seen. The not so fine print is Copilot may return garbage and it's up to you to confirm the results.
- latency-guy2 3y agoWell, I wouldn't really expect it. Its customer data, and it can technically be anything. Imagine making a system that is expected to be reliably excellent at generating things within that domain. It is already a very hard problem that I can confidently say is unsolved, and probably not even as good as a human can sort things. Even in a good dataset that is unstructured and unlabelled you will face issues in just a single domain (e.g. vision tasks trying to classify a variety of vehicles), imagine that being extended to a wide variety of subjects that can mix and match. I imagine at most, it'll be a reputation hit.
- huijzer 3y agoLooks like Microsoft needs to tune their "should I use my own information or ask the knowledge base (RAG)" model a bit. It missed very many cases here. The author could probably improve the results a lot by hinting that the knowledge base should be used to answer. As a side-note, OpenAI does very well on this currently in my experience. This morning, it even used search for a Rust deprecation warning that I got on some code and gave me a link to a related issue on a specific package that I was using. Quite amazing if you think about it since it (1) correctly understood the warning, (2) correctly decided to use search (Bing), (3) correctly wrote a search query, and (4) correctly gave the link to the right issue.
- hhh 3y agoI've been using Copilot for a few months, primarily in Outlook and Teams for meetings. Does a pretty good job at summarizing meetings and giving me action items. Easily good enough for me to justify the price.
- devjab 3y agoMy organisation has been using it for a month or two now, I forgot when it entered our environment as something we’re currently testing. It’s been rather good at the “office” side of things, and not very useful at doing the things we were doing with OpenAI tools. We too have found it rather good at summarising meetings, a very useful feature for any sort of Microsoft Teams meeting and much better than having some unlucky member of the meeting do it. Copilot has really shined for us, however, have been with PowerPoint, presentations in general and all sort of investor targeted material. I work in the energy sector, and we can’t exactly produce presentations and materials like a design heavy organisation might. So there are two consequences of this, one is that any internal presentation now looks good, which is basically a win-win for everyone. The other is how we’ve basically cut our external orders on design projects to a zero, which is a win did us and terrible for an entire business of graphic designers who target organisations like ours. I’ve personally mostly used Copilot to make funny pictures after it became apparent that it wasn’t anywhere near the level of OpenAI products as far as behind useful in programming and technology goes. Well, unless it’s very Microsoft documentation related, then Copilot does well at pointing you in the right direction. Well, on the frontend side of things, it’s replaced all our payments to icon libraries, as it’s capable of producing all the icons we need very well. OpenAI can do this as well, but at a cost. As far as the price goes. Copilot comes with our regular Azure and Office365 subscriptions, and similar to Microsoft teams it’ll be the obvious choice because of this. How will you ever justify paying for a competitor when your budgets show that you’re getting copilot for free? Well, you won’t in most organisations. I’m not a fan of this sort of monopoly, but it’s not like it will change unless EU or US regulation stops it.
- w3ll_w3ll_w3ll 3y agoWait, Microsoft 365 Copilot costs 30$/user per month. Or are you talking about just "Copilot" wich does not have any integration inside Microsoft 365 Apps (Office apps like Powerpoint or Outlook).
- poglet 3y agoHow much effort needs to be put into data classification before a copilot rollout?
- AshamedCaptain 3y ago> Copilot subtracted value compared to a simple Internet search. And for me this is the dealkiller aganst BingGPT or however it is called these days. It just doesn't work if you compare it to a baseline of a simple query, which in addition has the benefit it will also directly give you the source and author of the statement. BingGPT will _try_ to disclaim the sources it uses but fail miserably, as a lot of the times the assertion will not be in the source, or the _opposite_ assertion will be in the source.
- namlem 3y agoFor typical internet search Bing chat used to be way superior to a Google search before they nerfed it for cost reasons.
- ourmandave 3y agoI just wish they could nail down the product name. I thought ".NET whatever" was confusing, and now it's Copilot. In terms of the other Copilots, they all have different release dates. Here's a topline review: Copilot in Windows started rolling out on Windows 11 on September 26 through a Windows 11 update. Copilot for Microsoft 365 began rolling out for enterprise customers on November 1 and will roll out to non-enterprise users at a later date. Copilot for Sales will be available in the first quarter of 2024. Copilot for Service will be generally available in early 2024. Copilot in Viva will begin rolling out to customers "later in 2023", according to Microsoft. Github Copilot was the first tool to release, all the way back in 2021. https://www.zdnet.com/article/what-is-microsoft-copilot-heres-everything-you-need-to-know/ https://www.zdnet.com/article/what-is-microsoft-copilot-here...
- emodendroket 3y ago.NET was incoherent but "Copilot" is all LLM-based, text-generating products, right?
- tonyedgecombe 3y ago>Copilot for Sales will be available in the first quarter of 2024. I dread to think how that is going to turn out for the rest of us.
- ourmandave 3y agoCurrently I'm only using Copilot for OneDrive as a super grep for all my programming notes and gists.
- nicklecompte 3y agoWorth noting that Bing Copilot had the exact same bugs when it was announced, even though those documents were far simpler than any real corporate use case. Microsoft and OpenAI have surely spent the last year trying to improve this problem...and have seemingly made almost no useful progress. The total inability of transformer LLMs to summarize documents is not a surprise. Document summarization is an incredibly difficult task because it's impossible to do it accurately if you don't understand the semantics of the document. Current LLMs are simply too stupid to have anything but shallow, illusory understanding of human language. The illusion works when you are "kicking the tires" with simple documents, because literally thousands of data contractors have diligently trained the LLM on simple documents. But the illusion badly fails when you feed the LLM interesting documents: - novel ideas are smashed against the training set and flattened into a homogenous mush - specific facts and numbers are replaced with """statistically likely""" facts and numbers, which actually works okay for giant tables of statistics (except it destroys meaningful outliers). But this badly fails for, say, engineering specifications around aircraft engines. Or quarterly profit/loss figures, which Bing's demo screwed up. - Even code generation shits the bed if you're working in an uncommon language. Last I checked, GPT-3.5 had a terrible understanding of F# and would copy-paste dozens of lines verbatim from specific Github projects, even if that code was inappropriate for the task at hand. This illusion of competence is not meaningless if it works: GPT's Python code generation mostly works by translating English sentences into Python sentences, supplemented by plagiarism, but it's a useful tool regardless. The problem is that Copilot's illusion of competence does not work for aircraft piloting. There are not millions of lines of written aircraft logic in Github like there is with Python. The only way I can see this tool working is if Microsoft goes back and pretrains Copilot specifically on aircraft specifications, then hires data contractors for RHLF specific to Robichaux's use case. Otherwise it's simply too unreliable.
- thedougd 3y agoI had similar results trying out Amazon Q. I fed it a copy of our employee handbook and had it scrape our corporate website. My idea is that this could be a good fit for an HR bot. I asked it a few simple fact based questions, such as who our CEO is. It answered with our previous CEO based on an old press release. I asked it what year it is, and it couldn’t say. A few other specific fact based questions were either wrong or unanswered. So I asked it higher level questions about what the corporation does. It was able to answer those. Finally, I began asking questions about policies in the handbook. When I asked about harassment, it refused to provide any answers. It felt like I was filtered. Filtering will kill the utility of these products. Something like a legal practice is messy business that the filter won’t like.
- adastra22 3y agoMaybe try downloading a LLaMA model and fine tuning it with your corporate handbooks?
- _ink_ 3y agoDo you have a link on how to do such a thing?
- Multiplayer 3y agoHere's a fast and easy tool for Mac (& Linux) that I've been using: https://ollama.ai https://ollama.ai
- adastra22 3y ago[flagged]
- ForkMeOnTinder 3y agoollama is great for off-the-shelf models, but unless I missed the news I don't think it makes fine-tuning on a pile of PDFs any easier.
- mark_l_watson 3y agoI am curious how useful LLM based AI will be in Microsoft 365 and Google Gmail, Docs, YouTube, etc. I don’t have access yet to the AI extensions in Microsoft 365 but I have spent a fair amount of time with Bard + Workplace plugins. The best experience has been giving Bard a link to recipes on YouTube and getting short text summaries that are adequate for making the recipe. Sometimes queries against my GMail and Docs are useful but still a work in progress. Anyway, I have very high expectations that within 1 year, both Microsoft and Google will have nailed CoPilot-like AI tools for their cloud productivity apps.
- sbarre 3y agoHonestly, I think tools like Copilot 365 will be used less for generating copy or answering questions and more for boring yet time-intensive tasks like meeting summaries, reporting and building slides. Most people I know who work in large enterprises work in a "deck culture" where everything needs slides, and - again anecdotally - no one likes making slides. For many people it's a slow and painful process. If Copilot 365 can take a Word doc that someone wrote and mash it with a corporate slide template to produce a usable, or near-usable, deck that just needs a few tweaks before being ready to circulate or present, that will be a huge win for so many people. I saw a video demo of this recently and it was really impressive. A raw meeting transcript was turned into a meeting summary and that summary was then turned into nice-looking slides based on an existing presentation. All in under 5 minutes. I'm sure it was a controlled environment (the demo was by a Microsoft evangelist), but I didn't actually expect it to be that good. As more and more of these AI assistants appear, I think they will all slowly find their niche like this.
- zulban 3y ago> no one likes making slides. I'm not sure about that. I think there's a lot of people who actually like frequent long meetings and bad slides. That's why we have them everywhere. The rest is groupthink. I agree your use cases are the big wins though.
- FergusArgyll 3y agoI have not had much success with googles "Duet AI" for gmail and docs. It is constantly misunderstanding my emails and email threads. The generative part of it is worse in a way, I use it as a joke with friends, the output is corporate speak^10. I do find it useful when emailing ppl who appreciate obfuscated long emails, I can get company support ppl to treat me well...
- iandanforth 3y agoTLDR; It doesn't work very well except on summarizing specific documents.
- TheCaptain4815 3y agoElon just showed a post on Microsoft Word flagging the word "insane" as not being inclusive enough. These methods of "propaganda" are something the Chinese Government could only dream of (look also into Microsofts research on using Ai to blur out curse words live online). https://twitter.com/elonmusk/status/1735564647997030632 https://twitter.com/elonmusk/status/1735564647997030632
- Peanuts99 3y agoYou should probably look at some of the things they Chinese government are actually doing. Influencing spell check systems is on the tame end of these activities.
- emodendroket 3y agosorry, how is this related to Copilot?
- 93po 3y agoMaybe I'm a woke snowflake but I am entirely fine with this sort of suggestion, and is exactly the sort of feedback I'd want my for own writing. I'd also want it to suggest alternatives to someone calling something "retarded" outside the literal definition of something slowing. To me it's effectively the same thing, though "insane" is less controversial and less well known.
- ifyoubuildit 3y agoHas someone published somewhere the complete official acceptable language list? How many other less controversial and less well known words are there that we are carelessly using today? In some sense, these companies are ultimately going to be the ones to make that decision just by providing these little helpful suggestions to billions of people.
- lern_too_spel 3y agoHave you ever inadvertently offended somebody and then wondered why they disliked you? This is just a way to avoid that. Businesses want more features like this. If one employee writes something that hints that the employee reading it should do something, that might work in Japan, but in many other countries, a feature that suggests ways to make the action item clearer to the recipient would be well-received.
- JCM9 3y agoThe challenge and danger with widespread use of these tools is that they, with some frequency, continue to just confidently state answers that contain completely wrong information. These algorithms are still terrible at just saying I don’t know or here’s a possible answer but I have low confidence this is correct. They give every answer with a tone of authority despite being completely wrong with non-trivial frequency. A basic search engine search, while much more labor intensive, at least quickly reveals when there’s some inconsistency around a potential topic that allows an inquisitive individual to probe further. These new tools could be super dangerous in that they just spread false information and many folks don’t know or care enough to check if the answers are even right. We’re going to see a lot more train-wrecks like that lawyer that used ChatGPT to do research on his brief and ended up citing a bunch of cases that were made up and didn’t exist.
- deleted 3y ago[deleted]
- lukebuehler 3y agoThat’s why production RAG systems really require full blown search backends. You decompose the natural language question into a search query with meta data filtering and more. And only then do embeddings search. And only if you get some good search results, with high confidence scores, you use an LLM to summarize or glue the results together, otherwise you branch and tell the user, “I don’t know”. IMO that’s the only way to make it work right now.
- emodendroket 3y agoYeah, that's true. I was doing some research and it _seemed_ really useful how you could say "what's the difference between these two products?" and it would give you a confident answer with comparison tables and everything. At least until you checked and found the specs given were totally fictional.
- crispyambulance 3y agoHopefully, it can help with the infuriating confusion and struggles with nuances of word doc styling/formatting. Like "Clippy" but less annoying and actually useful. I am less interested in it writing stuff for me as I can do that with chat-gpt.
- emodendroket 3y agoThe secret is actually using "styles" instead of highlighting text and making ad hoc changes to it.
- probich0 3y agoLove seeing the discussion here. I wanted to test Copilot using some actual data, but not data from work. The docs I loaded are a mix of PDFs and Office documents, with embedded graphics, tables, and so on. Asking questions about e.g. stall speed is a proxy for asking fact-based questions from a corpus of "real" work docs. Some of the features I'm most excited about trying, like meeting recap, aren't available to me unless I schedule a bunch of meetings in the sandbox, which I can't do because it's a sandbox where I don't have access to invite outsiders. I will say that the integration and fit/finish of Copilot throughout M365 is quite good. For the most part, it's very easy to discover the entry points and get Copilot to do stuff. When its results are good, it's a very useful tool… but MS has some overall work to do to get consistency on that point.
- victor106 3y ago> Google Search produced the correct answer as its second result (after an ad). Bing Search didn’t show the correct answer on the first page of results and then I got tired of looking This has been my experience with a lot of my interactions using ChatGPT. The popular narrative about genAI is this ideal world where you just ask It a question and it gets you the “correct” answer every single time. The more I use it, unfortunately that is turning out NOT to be the case. I am still very optimistic about AI but this generation of AI feels like it has ways to go before it can be trusted
- mekkkkkk 3y agoA thousand times this. I'm hesitant to use ChatGPT whenever I really want the information to be correct. It's fine for getting a general outline of an historic event, for example, but not for getting the exact year and date of said event. I've been burned by way too many close-but-not-really's. I think GitHub Copilot made this very obvious. It can outline stuff really well, but you really have to double check and fix everything that it spits out as soon as complexity is above a certain (very low) threshold. ChatGPT is quite far from being the Google replacement that some people pretends/wants it to be. It's like that annoying friend who is extremely confident in announcing facts, but is often wrong. I keep giving it the side eyes.
- jmugan 3y agoI wish they would spend some time working on their spam filter. "My colleague reached out to you last week, but since we are all busy I thought I'd ..."
- sexy_seedbox 3y agoMicrosoft Defender for Office has been disappointing, replied emails with no attachments or links or dodgy keywords get quarantined for no reason at least a few times a month.
- dcreater 3y agoUnsurprising. The sota tech in the space is still not there so products at this time are under baked and are going to get found out very quickly. Especially by knowledge workers who know what they're doing.
- llamaInSouth 3y agoDid Copilot @microsoft.com (not Github) just lose its ability to write code? Just yesterday I was able to write python scripts and now it tells me that isn't able to.