6 ms·

I predict there will be another six months of these sorts of articles, accompanied by a raft of LLM-powered features that aren't nearly as transformative as the

by muglug 3y ago

I predict there will be another six months of these sorts of articles, accompanied by a raft of LLM-powered features that aren't nearly as transformative as the people currently hyping AI are telling us to expect. The engineers I know whose job it is to implement LLM features are much more skeptical about the near future than the engineers who are interested in the topic but lack hands-on experience.

alaskamiller 3y ago
The last chatbot wave (when FB opened up Messenger API and when Microsoft slung SKYPE as a plausible bot platform and when Slack rebranded their app store) fizzled out after 18 months. All to figure out the singular most important thing: chat interfaces are the worst.
- muspimerol 3y ago
  I agree that chat interfaces are not great, but we shouldn't reduce LLMs to implementations in chat interfaces. For example, the "autocompletion" I get with copilot is a very useful tool that I use daily, and I think that sort of UX could be built into plenty of other interfaces. Most applications where you input some form of text could benefit from LLM AI.
  - phillipcarter 3y ago
    Yes, this exactly. That's why we didn't go with chat for our UX here, and for future product areas we likely won't either. We already have good UX for our kind of product and haven't seen much feedback or convinced via some other means that adding chat would help more than it would hurt.
    - JieJie 3y ago
      One of the weirdest parts of using Bing Chat is that it has tab-to-autocomplete function that is almost always wrong about what I want to say. I wish there was an LLM that actually was an "autocorrect on steroids" because that's honestly one of my most-anticipated features of this technology. Having an LLM spell-checker that would autocorrect my spelling as I typed, based on the context of what I was typing? That would be magnificent.
      - sheepscreek 3y ago
        My experience with GitHub Copilot has been the exact opposite. Often, it’s magical - as if it can read my mind! It does have much more context to work with (open tabs, other files) vs. a single/short chat session.
        talim 3y ago
        Yeah, Copilot serves this purpose wonderfully—I've actually started writing documentation straight in VSCode and even occasionally things like certain emails, Jira tickets, or just general notes pertaining to anything vaguely technical, solely because Copilot is quite good at acting as a technical writing assistant. Since it's just OpenAI's text completion model with a code finetune and without the chat/assistant RLHF, it works much better as an "advanced autocomplete" than ChatGPT or even OpenAI's Turbo model via their API. I can be much more surgical with how I use it (often accepting just a few words at a time), and it's good at following my usual tone.
- muglug 3y ago
  Chatbots can be really useful. At my org we use a chatbot for pull requests — you get pinged by the bot when the PR is ready to merge, with a button in the chat interface that merges the PR — no need to open GitHub and locate the big green button yourself. That won’t 10x your productivity or whatever, but it does make it slightly more pleasant.
  - cjcenizal 3y ago
    That does sound cool, but I’m not sure that’s what most folks mean by “chatbot”. My understanding was that a chatbot is an automated chat program that will generate responses to your messages, simulating a live human.
- znpy 3y ago
  > chat interfaces are the worst Can’t agree more. As an user, a chatbot makes me think the company has put some kind of dumb parrot in front of me in order to avoid giving actual support.
  - candiodari 3y ago
    This makes me think someone could "pull a Google" with LLMs though. I mean we all know the original pitch for Google, right? There were search engines. Plenty. And indexes. And ... That wasn't it. Every site had search. That also wasn't it. The problem was that these things were incredibly low quality versions. Exactly what you're complaining about now with chatbot interfaces. In fact that these were such low quality is what made Google such an incredible opportunity: centralization came almost built in. They never needed to fight books.com: their search sucked. Even now finding a book using Google's interface works better than all search on the internet, except perhaps Amazon. And Amazon's "shittifying" it's search engine too now ... Google was also never "the best". It was simply consistent quality that worked everywhere. And because all other search engines were pretty far along in their enshittification cycle, with MBA's unwilling to go back, nobody even made a serious attempt at fighting them. Except, perhaps, and very late to the party, Microsoft. Google was better, but not incredibly better (and it's been going downhill for like 5 years now). It makes me think that if you could make something that would advise 10% of the world population on how to boil eggs, that would be incredible.
dinvlad 3y ago
This will be entertaining (if dangerous) to watch, as people hopefully become disillusioned with the overselling and overhyping of this not-new-tech-but-wrapped-in-marketing-bs wave of ‘AI’. But, history shows we rarely learn from past mistakes. I also hope there will be some whistleblower within OpenAI (and others like it) that exposes its internal practices and all of the hypocricy surrounding it. And, usually the fish rots from the head, as they say.
- version_five 3y ago
  Is there something specific your second paragraph refers to? I agree with the first one but I don't see a clear basis for the second. Do you just hope they're doing something untoward? And even if there is some dirt on them, how could it possibly relate to the quality of llms overall? Unless gpt4 is really just a room full of smart people typing really quickly...
  - dinvlad 3y ago
    For a business that lies and deceives the general public from the very beginning about the capabilities of their technology, it can only mean they’re as unethical on the inside as on the outside. If they were truly honest and open, they would not be behaving the way they are. These two sorts of behaviors are simply incompatible with each other. To be brutally honest, I expected better from Sam, but he has lost all credibility in my eyes based on how they chose to roll out ChatGPT. I now see that he’s even more hawky and daring and manipulative than Zuckelberg ever was.
evrydayhustling 3y ago
The main thing LLMs can do is make products accessible/useful to a wider range of users - either by parsing their intents or by translating outputs to their needs. This might result in a sort of transformation that engineers and power users aren't geared to appreciate. You might look at a natural language log query and say, "that would actually slow me down". But if it makes Honeycomb suddenly useful to stakeholders who couldn't before, it could lead to use cases not on the radar right now.
- phillipcarter 3y ago
  I probably could have elaborated more on this in the blog post, but you can really distill a lot of Honeycomb's success as a business down to a few things: - How easily can you query stuff when you're interested - How easily can you get other people on your team to use the product too - How quickly can you narrow down a problem (e.g., during an outage) to something you can fix - How relevant is your alerting (i.e., SLOs) to the success or failure of something business critical Our bet here is that the first two could potentially be improved by using LLMs, since we hypothesized (and confirmed in some new user interviews) that there's an "expressivity gap" in our product. A lot of people who aren't already observability experts, but do have some vested interest in observability, often know what they want to look for but get confused by a UX that's tailored for people who are more familiar with these kinds of tools. It's only been 3 weeks so it's too early to tell, but we're seeing some signs that the needle is being moved a bit on some key metrics. We're not betting the farm on this stuff just yet, and it's really cool that there's technology that lets us experiment in this way without having to hire a whole ML engineering team.
  - evrydayhustling 3y ago
    Since you are here, want to say this is one of the most useful posts I've seen about pragmatic development on top of LLMs! And agreed re: development effort - compared to other hype cycles of AI, it's important for folks to understand that the results they see are coming at a fraction of the experimental budget.
tbalsam 3y ago
Am an ML engineer, was around long before LLMs. Definitely am skeptical, and I think I know where one dimension that we're missing performance is, a number of people do (in regards to steerability/controllability without losing performance), it's just something that's quite hard to do tbh. Quite hard to do indeedy. Those who figure out how to do that well will have quite a legacy under their belts, and money if they're a profit-making company and handle it well. It's not about whether or not it can be done, it's not hard to lay out the math and prove that pretty trivially if you know where to look. Just actually doing it is the hard part in a way where the inductive bias translates appropriately to the solution at hand.
- dinvlad 3y ago
  The problem is, of course, these systems are fundamentally incapable of human-level intelligence and cognition. There will be a lot of wasted effort in pursuit of this unreachable goal (with LLM technology), an effort better spent elsewhere, like solving cancer or climate change, and stealing young and naive people’s minds away from these problems.
  - simonw 3y ago
    I don't think I want or need human-level cognition in my software. It turns out what I want is fancy autocomplete that you can few-shot teach to do all manner of useful NLP things, which is what LLMs are giving me today.
    - vba616 3y ago
      Trying to analogize the bullish case for LLMs as I perceive it: If you had an intern at your beck and call, who was obedient but knew nothing about anything except how to use a search engine, would that make you a "10x" at whatever you do?
      - dinvlad 3y ago
        Exactly - if I can’t trust it (as has been my experience with 90% of queries) so that I have to still Google its answers to verify them, what good does it do? It isn’t really improving my productivity in a substantial way, or can indeed hinder it. That said, my beef is not with its use for this purpose - whatever makes people happy and feel more productive, it’s all good and fun. The problem is when the CEO of a company like OpenAI starts talking with a straight face in his Congress testimony how this not-really-new tech has the potential to run afoul and enslave us (I’m heavily paraphrasing, obviously). That is the problem - this particular CEO suddenly feels entitled enough to make a giant PR stunt by using his official testimony to endow his product with the qualities it cannot possibly have - that is the problem. The blatant lies and deception is frankly disgusting.
deet 3y ago
I suspect you're right for how people are using and deploying LLMs now: hacking all kinds of functionality out of a text-completion model that, although it encodes a ton of data and some reasoning, is fundamentally still a text completion model and when deployed via commercial APIs like today without fine tuning, are not flexible beyond prompt engineering, chaining, etc. make possible. But I think we've only scratched the surface as to what LLMs fine-tuned on specific tasks, especially for abstract reasoning over narrow domains, could do. These applications possibly won't look anything like the chat interfaces that people are getting excited about now, and fine-tuning is not as accessible as prompt engineering. But there's a whole lot more to explore.
coffeebeqn 3y ago
I tried building with LLMs but it has the basic problem that it’s totally wrong 20-50% of the time. That’s very meaningful for most business cases. It does fine when accuracy isn’t important but that’s fairly rare other than writing throwaway content
- jiggawatts 3y ago
  But you probably didn’t use GPT 4 because its API is available only to a select few. “I tried a known-bad early iteration of a technology that has already been superseded — hence it is bad and will always be bad.” is not a convincing argument. The first transistor was an ugly and impractical thing too.
  - thaanpaa 3y ago
    I have access to OpenAI's GPT4, and there appears to be no significant difference from GPT3 (or GPT3.5). It looks like we are pretty far into the curve of diminishing returns already.
    - laratied 3y ago
      This is an absurd statement. I really don't understand what you can possibly be asking it. It doesn't matter though. I would prefer you don't use it.
      - thaanpaa 3y ago
        What a strange thing to be upset about.
- toolslive 3y ago
  ideally for marketing purposes. It can generate a several slogans, or posters and you can pick the one you like most. However if it needs to generate a graph, and the graph needs to be accurate, you're out of luck.
- phillipcarter 3y ago
herval 3y ago
> The engineers I know whose job it is to implement LLM features are much more skeptical about the near future than the engineers who are interested in the topic but lack hands-on experience. Isn't that always the case?
lee101 3y ago
[dead]
NicoJuicy 3y ago
You're already wrong Photoshop: https://the-decoder.com/adobe-photoshop-can-now-modify-images-with-simple-text-prompts/ https://the-decoder.com/adobe-photoshop-can-now-modify-image... Robotics: https://youtu.be/UuKAp9a6wMs https://youtu.be/UuKAp9a6wMs Home automation: ( couldn't find a good video, but it's with home assistant)

by muglug 3y ago

alaskamiller 3y ago

The last chatbot wave (when FB opened up Messenger API and when Microsoft slung SKYPE as a plausible bot platform and when Slack rebranded their app store) fizzled out after 18 months. All to figure out the singular most important thing: chat interfaces are the worst.

muspimerol 3y ago
I agree that chat interfaces are not great, but we shouldn't reduce LLMs to implementations in chat interfaces. For example, the "autocompletion" I get with copilot is a very useful tool that I use daily, and I think that sort of UX could be built into plenty of other interfaces. Most applications where you input some form of text could benefit from LLM AI.
- phillipcarter 3y ago
  Yes, this exactly. That's why we didn't go with chat for our UX here, and for future product areas we likely won't either. We already have good UX for our kind of product and haven't seen much feedback or convinced via some other means that adding chat would help more than it would hurt.
  - JieJie 3y ago
    One of the weirdest parts of using Bing Chat is that it has tab-to-autocomplete function that is almost always wrong about what I want to say. I wish there was an LLM that actually was an "autocorrect on steroids" because that's honestly one of my most-anticipated features of this technology. Having an LLM spell-checker that would autocorrect my spelling as I typed, based on the context of what I was typing? That would be magnificent.
    - sheepscreek 3y ago
      My experience with GitHub Copilot has been the exact opposite. Often, it’s magical - as if it can read my mind! It does have much more context to work with (open tabs, other files) vs. a single/short chat session.
      - talim 3y ago
        Yeah, Copilot serves this purpose wonderfully—I've actually started writing documentation straight in VSCode and even occasionally things like certain emails, Jira tickets, or just general notes pertaining to anything vaguely technical, solely because Copilot is quite good at acting as a technical writing assistant. Since it's just OpenAI's text completion model with a code finetune and without the chat/assistant RLHF, it works much better as an "advanced autocomplete" than ChatGPT or even OpenAI's Turbo model via their API. I can be much more surgical with how I use it (often accepting just a few words at a time), and it's good at following my usual tone.
muglug 3y ago
Chatbots can be really useful. At my org we use a chatbot for pull requests — you get pinged by the bot when the PR is ready to merge, with a button in the chat interface that merges the PR — no need to open GitHub and locate the big green button yourself. That won’t 10x your productivity or whatever, but it does make it slightly more pleasant.
- cjcenizal 3y ago
  That does sound cool, but I’m not sure that’s what most folks mean by “chatbot”. My understanding was that a chatbot is an automated chat program that will generate responses to your messages, simulating a live human.
znpy 3y ago
> chat interfaces are the worst Can’t agree more. As an user, a chatbot makes me think the company has put some kind of dumb parrot in front of me in order to avoid giving actual support.
- candiodari 3y ago
  This makes me think someone could "pull a Google" with LLMs though. I mean we all know the original pitch for Google, right? There were search engines. Plenty. And indexes. And ... That wasn't it. Every site had search. That also wasn't it. The problem was that these things were incredibly low quality versions. Exactly what you're complaining about now with chatbot interfaces. In fact that these were such low quality is what made Google such an incredible opportunity: centralization came almost built in. They never needed to fight books.com: their search sucked. Even now finding a book using Google's interface works better than all search on the internet, except perhaps Amazon. And Amazon's "shittifying" it's search engine too now ... Google was also never "the best". It was simply consistent quality that worked everywhere. And because all other search engines were pretty far along in their enshittification cycle, with MBA's unwilling to go back, nobody even made a serious attempt at fighting them. Except, perhaps, and very late to the party, Microsoft. Google was better, but not incredibly better (and it's been going downhill for like 5 years now). It makes me think that if you could make something that would advise 10% of the world population on how to boil eggs, that would be incredible.

dinvlad 3y ago

This will be entertaining (if dangerous) to watch, as people hopefully become disillusioned with the overselling and overhyping of this not-new-tech-but-wrapped-in-marketing-bs wave of ‘AI’. But, history shows we rarely learn from past mistakes. I also hope there will be some whistleblower within OpenAI (and others like it) that exposes its internal practices and all of the hypocricy surrounding it. And, usually the fish rots from the head, as they say.

version_five 3y ago
Is there something specific your second paragraph refers to? I agree with the first one but I don't see a clear basis for the second. Do you just hope they're doing something untoward? And even if there is some dirt on them, how could it possibly relate to the quality of llms overall? Unless gpt4 is really just a room full of smart people typing really quickly...
- dinvlad 3y ago
  For a business that lies and deceives the general public from the very beginning about the capabilities of their technology, it can only mean they’re as unethical on the inside as on the outside. If they were truly honest and open, they would not be behaving the way they are. These two sorts of behaviors are simply incompatible with each other. To be brutally honest, I expected better from Sam, but he has lost all credibility in my eyes based on how they chose to roll out ChatGPT. I now see that he’s even more hawky and daring and manipulative than Zuckelberg ever was.

evrydayhustling 3y ago

The main thing LLMs can do is make products accessible/useful to a wider range of users - either by parsing their intents or by translating outputs to their needs. This might result in a sort of transformation that engineers and power users aren't geared to appreciate. You might look at a natural language log query and say, "that would actually slow me down". But if it makes Honeycomb suddenly useful to stakeholders who couldn't before, it could lead to use cases not on the radar right now.

phillipcarter 3y ago
I probably could have elaborated more on this in the blog post, but you can really distill a lot of Honeycomb's success as a business down to a few things: - How easily can you query stuff when you're interested - How easily can you get other people on your team to use the product too - How quickly can you narrow down a problem (e.g., during an outage) to something you can fix - How relevant is your alerting (i.e., SLOs) to the success or failure of something business critical Our bet here is that the first two could potentially be improved by using LLMs, since we hypothesized (and confirmed in some new user interviews) that there's an "expressivity gap" in our product. A lot of people who aren't already observability experts, but do have some vested interest in observability, often know what they want to look for but get confused by a UX that's tailored for people who are more familiar with these kinds of tools. It's only been 3 weeks so it's too early to tell, but we're seeing some signs that the needle is being moved a bit on some key metrics. We're not betting the farm on this stuff just yet, and it's really cool that there's technology that lets us experiment in this way without having to hire a whole ML engineering team.
- evrydayhustling 3y ago
  Since you are here, want to say this is one of the most useful posts I've seen about pragmatic development on top of LLMs! And agreed re: development effort - compared to other hype cycles of AI, it's important for folks to understand that the results they see are coming at a fraction of the experimental budget.

tbalsam 3y ago

Am an ML engineer, was around long before LLMs. Definitely am skeptical, and I think I know where one dimension that we're missing performance is, a number of people do (in regards to steerability/controllability without losing performance), it's just something that's quite hard to do tbh. Quite hard to do indeedy. Those who figure out how to do that well will have quite a legacy under their belts, and money if they're a profit-making company and handle it well. It's not about whether or not it can be done, it's not hard to lay out the math and prove that pretty trivially if you know where to look. Just actually doing it is the hard part in a way where the inductive bias translates appropriately to the solution at hand.

dinvlad 3y ago
The problem is, of course, these systems are fundamentally incapable of human-level intelligence and cognition. There will be a lot of wasted effort in pursuit of this unreachable goal (with LLM technology), an effort better spent elsewhere, like solving cancer or climate change, and stealing young and naive people’s minds away from these problems.
- simonw 3y ago
  I don't think I want or need human-level cognition in my software. It turns out what I want is fancy autocomplete that you can few-shot teach to do all manner of useful NLP things, which is what LLMs are giving me today.
  - vba616 3y ago
    Trying to analogize the bullish case for LLMs as I perceive it: If you had an intern at your beck and call, who was obedient but knew nothing about anything except how to use a search engine, would that make you a "10x" at whatever you do?
    - dinvlad 3y ago
      Exactly - if I can’t trust it (as has been my experience with 90% of queries) so that I have to still Google its answers to verify them, what good does it do? It isn’t really improving my productivity in a substantial way, or can indeed hinder it. That said, my beef is not with its use for this purpose - whatever makes people happy and feel more productive, it’s all good and fun. The problem is when the CEO of a company like OpenAI starts talking with a straight face in his Congress testimony how this not-really-new tech has the potential to run afoul and enslave us (I’m heavily paraphrasing, obviously). That is the problem - this particular CEO suddenly feels entitled enough to make a giant PR stunt by using his official testimony to endow his product with the qualities it cannot possibly have - that is the problem. The blatant lies and deception is frankly disgusting.

deet 3y ago

I suspect you're right for how people are using and deploying LLMs now: hacking all kinds of functionality out of a text-completion model that, although it encodes a ton of data and some reasoning, is fundamentally still a text completion model and when deployed via commercial APIs like today without fine tuning, are not flexible beyond prompt engineering, chaining, etc. make possible. But I think we've only scratched the surface as to what LLMs fine-tuned on specific tasks, especially for abstract reasoning over narrow domains, could do. These applications possibly won't look anything like the chat interfaces that people are getting excited about now, and fine-tuning is not as accessible as prompt engineering. But there's a whole lot more to explore.

coffeebeqn 3y ago

I tried building with LLMs but it has the basic problem that it’s totally wrong 20-50% of the time. That’s very meaningful for most business cases. It does fine when accuracy isn’t important but that’s fairly rare other than writing throwaway content

jiggawatts 3y ago
But you probably didn’t use GPT 4 because its API is available only to a select few. “I tried a known-bad early iteration of a technology that has already been superseded — hence it is bad and will always be bad.” is not a convincing argument. The first transistor was an ugly and impractical thing too.
- thaanpaa 3y ago
  I have access to OpenAI's GPT4, and there appears to be no significant difference from GPT3 (or GPT3.5). It looks like we are pretty far into the curve of diminishing returns already.
  - laratied 3y ago
    This is an absurd statement. I really don't understand what you can possibly be asking it. It doesn't matter though. I would prefer you don't use it.
    - thaanpaa 3y ago
      What a strange thing to be upset about.
toolslive 3y ago
ideally for marketing purposes. It can generate a several slogans, or posters and you can pick the one you like most. However if it needs to generate a graph, and the graph needs to be accurate, you're out of luck.
phillipcarter 3y ago

herval 3y ago

> The engineers I know whose job it is to implement LLM features are much more skeptical about the near future than the engineers who are interested in the topic but lack hands-on experience. Isn't that always the case?

lee101 3y ago

[dead]

NicoJuicy 3y ago

You're already wrong Photoshop: https://the-decoder.com/adobe-photoshop-can-now-modify-images-with-simple-text-prompts/ https://the-decoder.com/adobe-photoshop-can-now-modify-image... Robotics: https://youtu.be/UuKAp9a6wMs https://youtu.be/UuKAp9a6wMs Home automation: ( couldn't find a good video, but it's with home assistant)