6 ms·

This study had management consultants use GPT4 on a variety of different management consultant tasks. Some were more creative in nature, and some were more ana

This study had management consultants use GPT4 on a variety of different management consultant tasks. Some were more creative in nature, and some were more analytical. For example, ideas on marketing sneakers versus an analysis of store performance across a retailer's portfolio. In general they found that GPT4 was helpful for creative tasks, but didn't help much (and in fact reduced quality) for analytical questions. I think these kinds of studies are of limited use. I don't believe raw GPT4 is that helpful in the enterprise. Whether it is useful or not comes down to whether engineers can harness it within a pipeline of content and tools. For example, when engineers create a system to summarize issues a customer has had from the CRM, that can help a customer service person be more informed. Structured semantic search on a knowledge base can help them also find the right solution to common customer problems. McKinsey made a retrieval augmented generation system that searched all their research reports and created summaries of content they could use, so a consultant could quickly find prior work that would be relevant on a client project. If they built that correctly, I imagine that is pretty useful. GPT4 alone will especially not be that useful for analytical work. However, developers can make it a semi-capable analyst for some use cases by connecting it to data lakes, describing schemas, and giving it tools to do analysis. Usually this is not a generalist solution, and needs to be built for each company's application. Many of the studies so far only look at vanilla GPT4 via ChatGPT, and it seems unlikely that, if LLMs do transform the workplace, that a standalone ChatGPT is what it will look like.

ebiester 3y ago
The first generation of company-focused llm will need to be bespoke. Teams involved with this will learn the necessary abstractions and will build new companies to lower the on-ramp or join the LLM providers to do the same.
Fin_Code 3y ago
This is entirely unsurprising. A language generator is not designed for analytical analysis. The fact it codes well is what I think is throwing people. Since code is a form of language it works well for the language model. The issue with code will come from larger context where analytical thought is needed and it will break down.
- svachalek 3y ago
  People are trying really hard to make "search" or something like it the killer app for LLMs but from what I've seen, it's translation. Coding in its purest form is a kind of translation, from natural language requirements to more precise computer languages and vice versa. Keeping an LLM properly focused on that task, it's pretty astonishing what it can do. But as you say, assuming it can do anything more than that is a trap.
  - chewxy 3y ago
    > Coding in its purest form is a kind of translation, from natural language requirements to more precise computer languages Unfortunately not so true. A simple way to check this is to look at what the brain is doing. Coding barely triggers the language components of the brain (your usual suspects like the Vernicke's and Broca's areas + other miscelleneous things). In fact what you'll find is that coding triggers the IP1, IP2 and AVI pathways. These are part of the multiple demand network. We call them programming "languages" because we can represent them in a token-based game we call "language" but the act of programming is far from mere language manipulation.
    - nomel 3y ago
      > from natural language requirements to more precise computer languages This makes sense. Translation to precise code would include more of the logic/planning bits of the brain, since that's the context of the output. I suspect a person fluent in Lojban[1] would show similar areas activating during translation from English to Lojban. I also assume translating code to words, with documentation, would include more language, since that's the context of the result. That doesn't mean it's not translation. It's just translation to a precise description of intent. 1. https://en.wikipedia.org/wiki/Lojban https://en.wikipedia.org/wiki/Lojban
roveo 3y ago
> GPT4 alone will especially not be that useful for analytical work. However, developers can make it a semi-capable analyst for some use cases by connecting it to data lakes, describing schemas, and giving it tools to do analysis. Usually this is not a generalist solution, and needs to be built for each company's application. I've been thinking about this a lot lately. I'm a data analyst and all these "give GPT you data warehouse schema, let it generate SQL to answer user's queries" products completely miss the point. An analyst has value as a curator of organizational knowledge, not translator from "business" to SQL. Things like knowing that when a product manager asks us for revenue/gmv, we exclude canceled orders, but include purchases made with bonus currency or promo codes. Things like this are not documented and are decided during meetings and in Slack chats. So my idea is that in order to make LLMs truly useful, we'll create "hybrid" programming languages that are half-written by humans, and the part that's written by LLMs when translating from "human" language is simple enough for LLMs to do it reliably. I even made some weekend-prototypes with pretty interesting results.
jampekka 3y ago
> I think these kinds of studies are of limited use. I don't believe raw GPT4 is that helpful in the enterprise. I'm not so sure about this. A lot of work in and for especially large organizations is writing text that often doesn't even get read, or is at most just glanced at. LLMs are great at writing useless text.
- bitzun 3y ago
  At first I thought this was a flippant joke, but I can actually see the value in saving time writing junk without having to alter existing bureaucratic processes from which nobody gets much value.
  - jampekka 3y ago
    I am using ChatGPT to write the genetic boilerplate that e.g. many funding applications require. Ethics statements or inclusivity policies etc where the contents are largely predefined and known by everybody but still have to be written in prose instead of e.g. clicking boxes.
- jprete 3y ago
  That kind of text is valuable as a signaling exercise - the author is willing to claim the text as indicating their actual beliefs. Writing it with AI would actually damage the author's credibility since it's no longer their words - although this shouldn't be a problem if the author is carefully checking the text every single time to make sure it represents their position. The effect would be even worse the first time the author missed a mistake from the AI and had to walk it back publicly. Nobody's going to trust anything they put in writing for a while after that.
  - jampekka 3y ago
    There are genres of text that have little to do with the writer's beliefs and are a kind of formality with contents known by all interested parties beforehand. Sort of elaborate prose versions of "I Agree With the Terms and Conditions" textboxes.
- 3y ago

ebiester 3y ago

The first generation of company-focused llm will need to be bespoke. Teams involved with this will learn the necessary abstractions and will build new companies to lower the on-ramp or join the LLM providers to do the same.

Fin_Code 3y ago

This is entirely unsurprising. A language generator is not designed for analytical analysis. The fact it codes well is what I think is throwing people. Since code is a form of language it works well for the language model. The issue with code will come from larger context where analytical thought is needed and it will break down.

svachalek 3y ago
People are trying really hard to make "search" or something like it the killer app for LLMs but from what I've seen, it's translation. Coding in its purest form is a kind of translation, from natural language requirements to more precise computer languages and vice versa. Keeping an LLM properly focused on that task, it's pretty astonishing what it can do. But as you say, assuming it can do anything more than that is a trap.
- chewxy 3y ago
  > Coding in its purest form is a kind of translation, from natural language requirements to more precise computer languages Unfortunately not so true. A simple way to check this is to look at what the brain is doing. Coding barely triggers the language components of the brain (your usual suspects like the Vernicke's and Broca's areas + other miscelleneous things). In fact what you'll find is that coding triggers the IP1, IP2 and AVI pathways. These are part of the multiple demand network. We call them programming "languages" because we can represent them in a token-based game we call "language" but the act of programming is far from mere language manipulation.
  - nomel 3y ago
    > from natural language requirements to more precise computer languages This makes sense. Translation to precise code would include more of the logic/planning bits of the brain, since that's the context of the output. I suspect a person fluent in Lojban[1] would show similar areas activating during translation from English to Lojban. I also assume translating code to words, with documentation, would include more language, since that's the context of the result. That doesn't mean it's not translation. It's just translation to a precise description of intent. 1. https://en.wikipedia.org/wiki/Lojban https://en.wikipedia.org/wiki/Lojban

roveo 3y ago

> GPT4 alone will especially not be that useful for analytical work. However, developers can make it a semi-capable analyst for some use cases by connecting it to data lakes, describing schemas, and giving it tools to do analysis. Usually this is not a generalist solution, and needs to be built for each company's application. I've been thinking about this a lot lately. I'm a data analyst and all these "give GPT you data warehouse schema, let it generate SQL to answer user's queries" products completely miss the point. An analyst has value as a curator of organizational knowledge, not translator from "business" to SQL. Things like knowing that when a product manager asks us for revenue/gmv, we exclude canceled orders, but include purchases made with bonus currency or promo codes. Things like this are not documented and are decided during meetings and in Slack chats. So my idea is that in order to make LLMs truly useful, we'll create "hybrid" programming languages that are half-written by humans, and the part that's written by LLMs when translating from "human" language is simple enough for LLMs to do it reliably. I even made some weekend-prototypes with pretty interesting results.

jampekka 3y ago

> I think these kinds of studies are of limited use. I don't believe raw GPT4 is that helpful in the enterprise. I'm not so sure about this. A lot of work in and for especially large organizations is writing text that often doesn't even get read, or is at most just glanced at. LLMs are great at writing useless text.

bitzun 3y ago
At first I thought this was a flippant joke, but I can actually see the value in saving time writing junk without having to alter existing bureaucratic processes from which nobody gets much value.
- jampekka 3y ago
  I am using ChatGPT to write the genetic boilerplate that e.g. many funding applications require. Ethics statements or inclusivity policies etc where the contents are largely predefined and known by everybody but still have to be written in prose instead of e.g. clicking boxes.
jprete 3y ago
That kind of text is valuable as a signaling exercise - the author is willing to claim the text as indicating their actual beliefs. Writing it with AI would actually damage the author's credibility since it's no longer their words - although this shouldn't be a problem if the author is carefully checking the text every single time to make sure it represents their position. The effect would be even worse the first time the author missed a mistake from the AI and had to walk it back publicly. Nobody's going to trust anything they put in writing for a while after that.
- jampekka 3y ago
  There are genres of text that have little to do with the writer's beliefs and are a kind of formality with contents known by all interested parties beforehand. Sort of elaborate prose versions of "I Agree With the Terms and Conditions" textboxes.
3y ago