4 ms·

I'm dealing with the issues discussed in the article as I roll out project management software for attorneys that has deep LLM integration. I have four differen

I'm dealing with the issues discussed in the article as I roll out project management software for attorneys that has deep LLM integration. I have four different types of LLM integration, but three of them use that same "familiar" chat request interface as their UI, and the fact that they are different interfaces for different purposes is very difficult for the end-users to reliably understand. There's the familiar "chat" interface that they are used to, and understand readily, which is used for legal R&D and new client on-boarding. But there are also traditional word processing and spreadsheet document authoring and editing, which also use that same "chat UI" - but that is not a conversational interface; requests to that interface cause modifications to the document, rewriting portions of the word processing or spreadsheet document. A third type of LLM integration is available that is a RAG type of interface for authored and uploaded documents, where the end-user is able to ask questions about the document with a generated subject matter expert in the document's topic (a word processor or spreadsheet). My fourth LLM integration has no interface, it makes an assessment of the end-user's familiarity with their goal, and adjusts the amount of help offered to the end-user. I am currently in the process of adding textual dressing around these three LLM request interfaces, but users are too used to operating interfaces on instinct, and the second and third type of LLM integrations where the LLM is co-authoring a document with the end-user is getting scant use, they simply don't realize it can co-author with them, and try to use it like a Q&A interface and not an assistant who will change the document for them.

superb_dev 2y ago
How visually distinct are the 3 interfaces?
- bsenftner 2y ago
  LLM interfaces look the same, the are a multi-line text area the end-user can resize, very much like the comment fields here on HackerNews. I'm considering changing their background color and adding other visual distinctions to make clear their different purposes. At first, I placed a simple text field on the toolbars of the word processor and the spreadsheet interfaces, but my initial testers thought they were a duplicated find/replace function and not an open ended LLM request field, despite a label and popup tool text saying otherwise. So I moved them to be separate, independent resizable text areas. The end-users I have testing the interfaces realize they are LLM interfaces now, but are not yet realizing that the word processor and spreadsheet interfaces can co-author with them. They ask the LLMs how to use questions, rather than "would you make this change" requests. To address that, I'm adding a drawer of example videos next to the LLM text area, so they have use examples showing them how to use the interface. No amount of text seems to help, people just don't read it.
afro88 2y ago
You might be trying to solve problems that they don't really have (but think they have), or you're solving them in the wrong way. For example, is a RAG chatbot the right way to get the information they need from documents, or do they want executive summaries of the document from different points of view that they almost always need to consider it from? Is a chat assistant that you have to talk to to make modifications to a document the right choice? Or would it be better to have the user highlight the text they want changed which then gives them the option to direct how to change it with natural language?
- shermantanktop 2y ago
  Sounds like this is third party product craziness.
- bsenftner 2y ago
  >Or would it be better to have the user highlight the text they want changed which then gives them the option to direct how to change it with natural language? That's exactly what I have: the user selects what they want modified, uses the "chat text area" to request the change, the entire document is sent in (for prose consistency) with the selection identified and the change is performed. If no selection is, the change request is applied to the entire document in context-output-size aware chunks. The RAG interface is not the only document query interface, but is the one preferred for finding information inside documents and discussing that information in large documents. My RAG implementation is a bit more than "just RAG", I use the document to generate a subject matter expert in the subject matter of the document, and that generated agent is who the end-user converses, along with the retrieved document information. So, in a way, my RAG is with the document, and an agent generated from the document, along with a context sensitive "professional communicator agent" whose job is to fill in the context breaks between the different RAG snippets, which reduces the fractured nature of the RAG snippets, lending towards better, deeper, more nunaced answers.
- ooterness 2y ago
  > Or would it be better to have the user highlight the text they want changed which then gives them the option to direct how to change it with natural language? Users could also highlight the text they want changed, then skip the middleman by typing in the new text themselves. Not sure how to monetize this.
htrp 2y ago
any link to mock-ups to more clearly differentiate the UX experiences?
pjc50 2y ago
> the second and third type of LLM integrations where the LLM is co-authoring a document with the end-user is getting scant use, they simply don't realize it can co-author with them This is a very new and unfamiliar paradigm, and you're going to have to "train" people in how it works.

superb_dev 2y ago

How visually distinct are the 3 interfaces?

bsenftner 2y ago
LLM interfaces look the same, the are a multi-line text area the end-user can resize, very much like the comment fields here on HackerNews. I'm considering changing their background color and adding other visual distinctions to make clear their different purposes. At first, I placed a simple text field on the toolbars of the word processor and the spreadsheet interfaces, but my initial testers thought they were a duplicated find/replace function and not an open ended LLM request field, despite a label and popup tool text saying otherwise. So I moved them to be separate, independent resizable text areas. The end-users I have testing the interfaces realize they are LLM interfaces now, but are not yet realizing that the word processor and spreadsheet interfaces can co-author with them. They ask the LLMs how to use questions, rather than "would you make this change" requests. To address that, I'm adding a drawer of example videos next to the LLM text area, so they have use examples showing them how to use the interface. No amount of text seems to help, people just don't read it.

afro88 2y ago

You might be trying to solve problems that they don't really have (but think they have), or you're solving them in the wrong way. For example, is a RAG chatbot the right way to get the information they need from documents, or do they want executive summaries of the document from different points of view that they almost always need to consider it from? Is a chat assistant that you have to talk to to make modifications to a document the right choice? Or would it be better to have the user highlight the text they want changed which then gives them the option to direct how to change it with natural language?

shermantanktop 2y ago
Sounds like this is third party product craziness.
bsenftner 2y ago
>Or would it be better to have the user highlight the text they want changed which then gives them the option to direct how to change it with natural language? That's exactly what I have: the user selects what they want modified, uses the "chat text area" to request the change, the entire document is sent in (for prose consistency) with the selection identified and the change is performed. If no selection is, the change request is applied to the entire document in context-output-size aware chunks. The RAG interface is not the only document query interface, but is the one preferred for finding information inside documents and discussing that information in large documents. My RAG implementation is a bit more than "just RAG", I use the document to generate a subject matter expert in the subject matter of the document, and that generated agent is who the end-user converses, along with the retrieved document information. So, in a way, my RAG is with the document, and an agent generated from the document, along with a context sensitive "professional communicator agent" whose job is to fill in the context breaks between the different RAG snippets, which reduces the fractured nature of the RAG snippets, lending towards better, deeper, more nunaced answers.
ooterness 2y ago
> Or would it be better to have the user highlight the text they want changed which then gives them the option to direct how to change it with natural language? Users could also highlight the text they want changed, then skip the middleman by typing in the new text themselves. Not sure how to monetize this.

htrp 2y ago

any link to mock-ups to more clearly differentiate the UX experiences?

pjc50 2y ago

> the second and third type of LLM integrations where the LLM is co-authoring a document with the end-user is getting scant use, they simply don't realize it can co-author with them This is a very new and unfamiliar paradigm, and you're going to have to "train" people in how it works.