5 ms·

The AGI party might be cancelled. But the current strengths of LLMs still are summarization, analogy, templating and search. Which can solve a lot of business p

The AGI party might be cancelled. But the current strengths of LLMs still are summarization, analogy, templating and search. Which can solve a lot of business problems already. My hot take is that UX is the biggest limiter for current LLM products other than chatGPT. Chat has too much friction for many use cases, we need to find better interfaces that are more visual and faster to interact with. Spending $100M to gain .1% on MMLU is a waste of time in comparison.

schaefer 2y ago
Pardon me, Asking for what you want (a.k.a: chat) is too much friction?!?
- mensetmanusman 2y ago
  I need an LLM to chat for me.
- arnaudsm 2y ago
  Most of the interfaces we use daily require very few clicks, while chat is multiple keystrokes. And the discoverability is the worst part of it.
- noirbot 2y ago
  It absolutely can be. Prompt construction, as well as learning what sort of things the LLM just isn't good at doing, takes work. I've had multiple times where I've asked for something, gotten complete trash back from the LLM, complained to a friend who's more into LLMs about it, and had them go "Oh, you just have to add this extra set of words at the end/phrase it this other way and it'll work." Or they'll give me some massive multi-paragraph "seed" prompt I need to add in to make it not behave stupidly sometimes, that they evidently have to update every few days or weeks to adapt to some new behavior. Until Google decided to trash their search with their own sub-par LLM, my results were often regularly better just hitting the first link on Google Search like I had for the last decade.
- fragmede 2y ago
  When you go to a sit-down restaurant, you aren't given a list of the ingredients that they have out back in the kitchen and get asked how you want to assemble them. Someone's done work to take those ingredients and form them into menu items for you to pick from. Given infinite possibilities to enter into a text box is more friction than picking from a menu in front of you. There's a lot of use for LLMs, and it's tossing the ingredients in a restaurant's kitchen together but sometimes people want a menu.
  - noirbot 2y ago
    I like that analogy, but I'd almost take it a little further. It's a restaurant with no menu and claims it has every ingredient you could ever want in the kitchen. If you ask for a sandwich, you'll get a sandwich, and it'll probably be edible. But if you want anything approaching the quality of even a mid-tier restaurant, you almost need to know how to cook it yourself in order to order it, or at least have a very broad idea of what you're willing to accept. Just saying you want a chicken florentine isn't enough. You may also have to specify that the chicken is fresh but also not still alive, and that kale isn't the same as spinach, and confirm that they have the correct recipe for mornay sauce even though every chef should already know that. And even if you do all that, you may still get a chicken cordon bleu for reasons you cannot explain and neither can the kitchen. When it's free/cheap, and I'm up for an adventure and sending stuff back to try again a few times, it's not bad. If I'm having to pay a pretty high amount for it and still treat learning the system like a second job, I start to have questions about why I'm bothering.
    - fragmede 2y ago
      Yeah. But then an eccentric celebrity comes into your restaurant. Them eating there would be great for your restaurant, so you seat them. They ask for Bubble and Squeak, which isn't on the menu and you've never made it before, though you do happen to have all the ingredients for it. It's a busy night but you've never done it before so you ask your Large Language Mise-en-place-cook to whip up one and it's... okay. You've done a lot of cooking before, so you can spot some of the problems with how the LLM whipped up the dish, and you take the one they made, fix up a few things, and ship it out the door in less time than if you didn't have your Large Language Mise-en-place-cook helping you. It was never going to get it perfectly right, but you're a busy, but expert cook, and you know its shortcomings. Rather than try to get it to work perfectly on its own, you pair with it and are able to get more dishes out than you would be without its help. You're not convinced that it's progress, but it's where the industry has moved.
- thr0w 2y ago
  Chat is about the worst possible interface for doing any sort of precise task, which presumably is what any programmer wants from a computer.
cloudking 2y ago
...or no UX, use these systems in the backend with well crafted prompts and abstract the need to instruct LLMs away from the user?
- blharr 2y ago
  I believe what you are describing is exactly the type of UX that the parent comment may have been calling for
  - cloudking 2y ago
    Makes sense, my biggest takeaway from observing folks in many different roles interact with LLMs is this: they simply don't know how to prompt effectively. There is no problem with the chat UX, if you know the right techniques to get optimal output, but writing effective prompts is a technical skill similar to programming - most folks that interact with computers are not highly technical. If we design software powered by LLMs understanding that effective prompting is a skill most users don't have, then we should build UX that abstracts prompting away. Of course there are going to be use cases where a user needs to provide input to the LLM, but that UX also can be assistive and require minimal cognitive overhead (e.g less raw text prompt input, more clicks on buttons/forms etc to a backend prompt builder).
EternalFury 2y ago
Alas, "summarization" is a crapshoot. It sometimes summarizes essential facts out of the original text. I think it's due to limits on output tokens. And chunking cannot solve this because you cannot summarize well if you don't take in the whole thing. Aggregating summarized chunks doesn't produce good results. It works very very well on Amazon reviews, though, because these are small and thin.

schaefer 2y ago

Pardon me, Asking for what you want (a.k.a: chat) is too much friction?!?

mensetmanusman 2y ago
I need an LLM to chat for me.
arnaudsm 2y ago
Most of the interfaces we use daily require very few clicks, while chat is multiple keystrokes. And the discoverability is the worst part of it.
noirbot 2y ago
It absolutely can be. Prompt construction, as well as learning what sort of things the LLM just isn't good at doing, takes work. I've had multiple times where I've asked for something, gotten complete trash back from the LLM, complained to a friend who's more into LLMs about it, and had them go "Oh, you just have to add this extra set of words at the end/phrase it this other way and it'll work." Or they'll give me some massive multi-paragraph "seed" prompt I need to add in to make it not behave stupidly sometimes, that they evidently have to update every few days or weeks to adapt to some new behavior. Until Google decided to trash their search with their own sub-par LLM, my results were often regularly better just hitting the first link on Google Search like I had for the last decade.
fragmede 2y ago
When you go to a sit-down restaurant, you aren't given a list of the ingredients that they have out back in the kitchen and get asked how you want to assemble them. Someone's done work to take those ingredients and form them into menu items for you to pick from. Given infinite possibilities to enter into a text box is more friction than picking from a menu in front of you. There's a lot of use for LLMs, and it's tossing the ingredients in a restaurant's kitchen together but sometimes people want a menu.
- noirbot 2y ago
  I like that analogy, but I'd almost take it a little further. It's a restaurant with no menu and claims it has every ingredient you could ever want in the kitchen. If you ask for a sandwich, you'll get a sandwich, and it'll probably be edible. But if you want anything approaching the quality of even a mid-tier restaurant, you almost need to know how to cook it yourself in order to order it, or at least have a very broad idea of what you're willing to accept. Just saying you want a chicken florentine isn't enough. You may also have to specify that the chicken is fresh but also not still alive, and that kale isn't the same as spinach, and confirm that they have the correct recipe for mornay sauce even though every chef should already know that. And even if you do all that, you may still get a chicken cordon bleu for reasons you cannot explain and neither can the kitchen. When it's free/cheap, and I'm up for an adventure and sending stuff back to try again a few times, it's not bad. If I'm having to pay a pretty high amount for it and still treat learning the system like a second job, I start to have questions about why I'm bothering.
  - fragmede 2y ago
    Yeah. But then an eccentric celebrity comes into your restaurant. Them eating there would be great for your restaurant, so you seat them. They ask for Bubble and Squeak, which isn't on the menu and you've never made it before, though you do happen to have all the ingredients for it. It's a busy night but you've never done it before so you ask your Large Language Mise-en-place-cook to whip up one and it's... okay. You've done a lot of cooking before, so you can spot some of the problems with how the LLM whipped up the dish, and you take the one they made, fix up a few things, and ship it out the door in less time than if you didn't have your Large Language Mise-en-place-cook helping you. It was never going to get it perfectly right, but you're a busy, but expert cook, and you know its shortcomings. Rather than try to get it to work perfectly on its own, you pair with it and are able to get more dishes out than you would be without its help. You're not convinced that it's progress, but it's where the industry has moved.
thr0w 2y ago
Chat is about the worst possible interface for doing any sort of precise task, which presumably is what any programmer wants from a computer.

cloudking 2y ago

...or no UX, use these systems in the backend with well crafted prompts and abstract the need to instruct LLMs away from the user?

blharr 2y ago
I believe what you are describing is exactly the type of UX that the parent comment may have been calling for
- cloudking 2y ago
  Makes sense, my biggest takeaway from observing folks in many different roles interact with LLMs is this: they simply don't know how to prompt effectively. There is no problem with the chat UX, if you know the right techniques to get optimal output, but writing effective prompts is a technical skill similar to programming - most folks that interact with computers are not highly technical. If we design software powered by LLMs understanding that effective prompting is a skill most users don't have, then we should build UX that abstracts prompting away. Of course there are going to be use cases where a user needs to provide input to the LLM, but that UX also can be assistive and require minimal cognitive overhead (e.g less raw text prompt input, more clicks on buttons/forms etc to a backend prompt builder).

EternalFury 2y ago

Alas, "summarization" is a crapshoot. It sometimes summarizes essential facts out of the original text. I think it's due to limits on output tokens. And chunking cannot solve this because you cannot summarize well if you don't take in the whole thing. Aggregating summarized chunks doesn't produce good results. It works very very well on Amazon reviews, though, because these are small and thin.