6 ms·

Being able to create 2048 in 6 hours has basically zero economic value. Can ChatGPT materially and positively impact the code written by big companies? Can it

Being able to create 2048 in 6 hours has basically zero economic value. Can ChatGPT materially and positively impact the code written by big companies? Can it do meaningful work in excel? Can it do meaningful PowerPoint work? Can it give effective advice on management? Right now we don’t know the answer to those questions. LLM apps can still improve in many ways - better base models, better integration with common enterprise applications, agentic processes, verifiability and so on - so there is definitely hope that there will be significant value created. Companies and people are excited because there’s huge potential. But it is really just potential right now … current systems aren’t creating real enterprise value at this moment in time

TeMPOraL 2y ago
> Can ChatGPT materially and positively impact the code written by big companies? Can it do meaningful work in excel? Can it do meaningful PowerPoint work? Can it give effective advice on management? > Right now we don’t know the answer to those questions. I know the answer to the first three. Yes, yes, and yes. I've done them all, including all of them in the past few weeks. (Which is how I learned that it's much better to ask ChatGPT to use Python evaluation mode and Pandoc and make you a PPTX, than trying to do anything with "Office 365 Copilot" in PowerPoint...) As for the fourth question - well, ChatGPT can give you better advice than most advice on management/leadership articles, so I presume the answer here is "Yes" too - but I didn't verify it in practice. > current systems aren’t creating real enterprise value at this moment in time Yes, they are. They would be creating even more value if not for the copyright and exports uncertainty, which significantly slows enterprise adoption.
- WgaqPdNr7PGLGVW 2y ago
  > I know the answer to the first three. Yes, yes, and yes. You say this but from a management perspective at a large enterprise software company I have not seen it. Some of our developers use copilot and gpt and some don't and it is incredibly difficult to see any performance difference between the groups. We aren't seeing higher overall levels of productivity. We aren't seeing the developers who start using copilot/gpt rush ahead of their peers. We aren't seeing any ability to cut back on developer spend. We aren't seeing anything positive yet and many developers have been using copilot/gpt for >1 year. In my opinion we are just regaining some of the economic value we lost when Google Search started degrading 5-10 years ago.
  - TeMPOraL 2y ago
    > We aren't seeing higher overall levels of productivity. You can't measure productivity for shit, otherwise companies would look entirely differently. Starting from me not having to do my own finances or event planning or hundred other things that are not my job description, not my specialty, and which were done by dedicated staff just a few decades ago, before tech "improved office productivity". > We aren't seeing the developers who start using copilot/gpt rush ahead of their peers. That's because individual productivity is usually constrained by team productivity. Devs rushing ahead of their teammates makes the team dysfunctional. > We aren't seeing any ability to cut back on developer spend. Devs aren't stupid. They're not going to give you an opportunity if they can avoid it. > We aren't seeing anything positive yet and many developers have been using copilot/gpt for >1 year. My belief is that's because you aren't measuring the right things. But then, no one is. This is a problem well-known to be unsolved.
    - WgaqPdNr7PGLGVW 2y ago
      > You can't measure productivity for shit We can't measure small changes and we aren't great at comparing across orgs. However, at the director level we can certainly see a 50% or 100% productivity improvement in our teams and with individuals in our teams. We aren't seeing changes of this magnitude because they don't exist.
      - WgaqPdNr7PGLGVW 2y ago
        There are other potential explanations. Perhaps developers are now slacking off. Perhaps we have added more meetings because developers have more free time. Or perhaps developers were never the bottleneck. We can see large productivity improvements when we make simple changes like having product managers join the developers daily standup meetings. We can even measure productivity improvements from Slacks/Zooms auto-summary features. Yet gpt/copilot doesn't even register.
        Vegenoid 2y ago
        > We can even measure productivity improvements from Slacks/Zooms auto-summary features. While not code generation, this auto-summary is powered by the same tech. I think using it to sift through and surface relevant information, as opposed to generation of new things, will have the biggest impact. By far the greatest value I get out of LLMs is asking them to help me understand code written by others. I feel like this is an under-appreciated use. How long has this feature been in Copilot? Since February or so? Are people using it? I do not use Copilot.
- kortilla 2y ago
  >Which is how I learned that it's much better to ask ChatGPT to use Python evaluation mode and Pandoc and make you a PPTX, than trying to do anything with "Office 365 Copilot" in PowerPoint... Can you elaborate on what this saved over just making the ppt the old fashioned way?
  - TeMPOraL 2y ago
    "I have this set of notes attached below; would you kindly group them by X and tabulate, and then use Python with Pandoc to make me a PowerPoint with that table in it, plus an extra slide with commentary from the notes?" Attach notes, paste, press Enter, wait half a minute, get back a PPTX you can build on, or just restyle[0]. Sure, it's faster to build the presentation yourself than to make ChatGPT make the whole thing for you. But the more time-consuming and boring parts, like making tables and diagrams and summaries from external data or notes, is something ChatGPT can do in a fraction of time, and can output directly into PPTX via Pandoc. (There's a lot of fun things you can do with official ChatGPT and Python integration. The other day I made it design, write and train a multi-layer perceptron for playing tic-tac-toe, because why waste my own GPU-seconds :).) -- [0] - In contrast, if you make the same request in PowerPoint's O365 Copilot, it'll barf. Last time I tried, it argued it has no capability to edit the document; the time before that, it made a new slide with text saying literally "data from the previous message".
- tempusalaria 2y ago
  I think you’re thinking about things very locally. Of course ChatGPT can help with some coding - I use it for regex quite often cause I never really learned that well. The problem is that at the average medium sized company code looks like this - you have 1mln lines of code written over a decade by a few hundred people. A big portion of the code is redundant, some of it is incomplete, much of it is undocumented. Different companies have different coding styles, different testing approaches, different development dynamics. ChatGPT does not appreciate this context. Excel has some similar problems. First of all Excel is 2 dimensional and LLMs really don’t think in 2 dimensions well. So you need to flatten the excel file for the LLM. A common approach to do this with LLMs is using pandas and then using the column and row names to index into the excel. Unfortunately, excels at companies cannot be easily read using pandas. They are illogically structured, have tons of hardcoding, intersheet referencing is weird circular ways and so on. I spent some time in finance and sell side equity research models are written by highly trained financial analysts and are substantially better organized than the average excel model at a company. Even this subset of real world models is far from suitable for a direct pandas interpretation. Parsing sell side models requires a delicate and complex interpretation before being fed into an LLM.
mdorazio 2y ago
> Can ChatGPT materially and positively impact the code written by big companies? It already has at a Fortune 100 company I contract with currently. > Can it do meaningful work in excel? We can quibble about what "meaningful" means, but it satisfactorily answered questions for two friends about how to build formulas for their datasets and is currently being used to summarize data insights from a database at a different large client (Excel =/= database, but the point stands). > Can it do meaningful PowerPoint work? I've used Midjourney multiple times a month to generate base imagery for various things in PowerPoint (usually requires modification in Photoshop, but saves me several hours each time compared to digital painting or photobashing from scratch). > Can it give effective advice on management? Again, what does "effective" mean in the context of management? I've seen VP-level individuals with hundreds of people in their orgs using AI tools for different things. It really feels like a significant chunk of the HN crowd is living in a bubble with respect to AI in the real world right now. It's absolutely invading everything. As for how much revenue that will translate into long-term vs. the investment dollars being poured into it, that's a more interesting question to discuss.
reaperman 2y ago
> Can it give effective advice on management? My friends at McKinsey say that while it can’t fine-tune reports and presentations with quite enough nuance, it does a good job sifting through lots of shit to pick out important parts they should pay more attention to, highlighting data/talking points that contradict a working hypothesis, assisting in writing emails, and other time-consuming or very nit-picky tasks. That said, no one I know has fed it real customer data, that would be a career-ending event. But self-hosted models like Gemma2 open up the possibility for using LLMs against real customer info.
- munificent 2y ago
  Hard to tell if that says more about the value of LLMs or the lack of value of McKinsey...
  - throwaway2037 2y ago
    > the lack of value of McKinsey Leaving McKinsey's specific brand value aside, people always miss the value of "hiring (business) consultants". You basically get insider knowledge about how competitors businesses and systems work. So if you ask for advice about how to build a healthcare app for smartphones, you hire McKinsey (or whomever) to tell you "about the market". But really, they are just telling you about what they saw at other competitors. For some business decisions, it is very valuable.
ern 2y ago
Can it do meaningful PowerPoint work Yes, it absolutely can. I threw together a PowerPoint presentation with a script for a low-value, high visibility meeting a couple of weeks ago with ChatGPT 4.0 and a PowerPoint plugin. Everyone loved it.
- throwaway2037 2y ago
  > low-value, high visibility meeting This is such a gem. Can you tell us more about the meeting? A senior manager kicking the tyres, or what? Any funny bike-shedding stories to tell?
  - ern 2y ago
    Corporate values. Drew the short straw, but had to present something.
nl 2y ago
> Can it do meaningful work in excel? Yes it can. But more importantly have you tried ChatGPT Data Analyst?: https://openai.com/index/improvements-to-data-analysis-in-chatgpt/ https://openai.com/index/improvements-to-data-analysis-in-ch... It drops the barrier for "pretty good data analysis" to effectively zero. > Can it do meaningful PowerPoint work? Canva and Figma are both building this and they are pretty decent right now. Better than most PowerPoints I've seen. The aforementioned Data Analyst does good presentations in a different way, too. > Can it give effective advice on management? Yes. Unfortunately can't talk about this except it is mindblowingly good.
- TeMPOraL 2y ago
  > The aforementioned Data Analyst does good presentations in a different way, too. And on top of that, it can do PowerPoint presentations too - magic keywords are "use Python and Pandoc".
fragmede 2y ago
> Being able to create 2048 in 6 hours has basically zero economic value. That's actually a really good point. In the realm of programming, things that were previously not done because they were too expensive can now be done. Prior to ChatGPT, GP could have a) done it themselves, but the cost was too high/it wasn't worth their time, b) found enough time to write a spec, found someone on upwork/etc, paid them to make it, except that costs money they didn't want to spend, or c) just not do it. Now, GP can code this thing up while watching netflix with the kids or whatever. What programs do not exist that previously did not have the economic value to exist, but now can, thanks to programming time getting cheaper? Now apply that to fields outside of programming. LLMs' ability to program is front and center here, since many of us can program, but they do other things as well.

TeMPOraL 2y ago

> Can ChatGPT materially and positively impact the code written by big companies? Can it do meaningful work in excel? Can it do meaningful PowerPoint work? Can it give effective advice on management? > Right now we don’t know the answer to those questions. I know the answer to the first three. Yes, yes, and yes. I've done them all, including all of them in the past few weeks. (Which is how I learned that it's much better to ask ChatGPT to use Python evaluation mode and Pandoc and make you a PPTX, than trying to do anything with "Office 365 Copilot" in PowerPoint...) As for the fourth question - well, ChatGPT can give you better advice than most advice on management/leadership articles, so I presume the answer here is "Yes" too - but I didn't verify it in practice. > current systems aren’t creating real enterprise value at this moment in time Yes, they are. They would be creating even more value if not for the copyright and exports uncertainty, which significantly slows enterprise adoption.

WgaqPdNr7PGLGVW 2y ago
> I know the answer to the first three. Yes, yes, and yes. You say this but from a management perspective at a large enterprise software company I have not seen it. Some of our developers use copilot and gpt and some don't and it is incredibly difficult to see any performance difference between the groups. We aren't seeing higher overall levels of productivity. We aren't seeing the developers who start using copilot/gpt rush ahead of their peers. We aren't seeing any ability to cut back on developer spend. We aren't seeing anything positive yet and many developers have been using copilot/gpt for >1 year. In my opinion we are just regaining some of the economic value we lost when Google Search started degrading 5-10 years ago.
- TeMPOraL 2y ago
  > We aren't seeing higher overall levels of productivity. You can't measure productivity for shit, otherwise companies would look entirely differently. Starting from me not having to do my own finances or event planning or hundred other things that are not my job description, not my specialty, and which were done by dedicated staff just a few decades ago, before tech "improved office productivity". > We aren't seeing the developers who start using copilot/gpt rush ahead of their peers. That's because individual productivity is usually constrained by team productivity. Devs rushing ahead of their teammates makes the team dysfunctional. > We aren't seeing any ability to cut back on developer spend. Devs aren't stupid. They're not going to give you an opportunity if they can avoid it. > We aren't seeing anything positive yet and many developers have been using copilot/gpt for >1 year. My belief is that's because you aren't measuring the right things. But then, no one is. This is a problem well-known to be unsolved.
  - WgaqPdNr7PGLGVW 2y ago
    > You can't measure productivity for shit We can't measure small changes and we aren't great at comparing across orgs. However, at the director level we can certainly see a 50% or 100% productivity improvement in our teams and with individuals in our teams. We aren't seeing changes of this magnitude because they don't exist.
    - WgaqPdNr7PGLGVW 2y ago
      There are other potential explanations. Perhaps developers are now slacking off. Perhaps we have added more meetings because developers have more free time. Or perhaps developers were never the bottleneck. We can see large productivity improvements when we make simple changes like having product managers join the developers daily standup meetings. We can even measure productivity improvements from Slacks/Zooms auto-summary features. Yet gpt/copilot doesn't even register.
      - Vegenoid 2y ago
        > We can even measure productivity improvements from Slacks/Zooms auto-summary features. While not code generation, this auto-summary is powered by the same tech. I think using it to sift through and surface relevant information, as opposed to generation of new things, will have the biggest impact. By far the greatest value I get out of LLMs is asking them to help me understand code written by others. I feel like this is an under-appreciated use. How long has this feature been in Copilot? Since February or so? Are people using it? I do not use Copilot.
kortilla 2y ago
>Which is how I learned that it's much better to ask ChatGPT to use Python evaluation mode and Pandoc and make you a PPTX, than trying to do anything with "Office 365 Copilot" in PowerPoint... Can you elaborate on what this saved over just making the ppt the old fashioned way?
- TeMPOraL 2y ago
  "I have this set of notes attached below; would you kindly group them by X and tabulate, and then use Python with Pandoc to make me a PowerPoint with that table in it, plus an extra slide with commentary from the notes?" Attach notes, paste, press Enter, wait half a minute, get back a PPTX you can build on, or just restyle[0]. Sure, it's faster to build the presentation yourself than to make ChatGPT make the whole thing for you. But the more time-consuming and boring parts, like making tables and diagrams and summaries from external data or notes, is something ChatGPT can do in a fraction of time, and can output directly into PPTX via Pandoc. (There's a lot of fun things you can do with official ChatGPT and Python integration. The other day I made it design, write and train a multi-layer perceptron for playing tic-tac-toe, because why waste my own GPU-seconds :).) -- [0] - In contrast, if you make the same request in PowerPoint's O365 Copilot, it'll barf. Last time I tried, it argued it has no capability to edit the document; the time before that, it made a new slide with text saying literally "data from the previous message".
tempusalaria 2y ago
I think you’re thinking about things very locally. Of course ChatGPT can help with some coding - I use it for regex quite often cause I never really learned that well. The problem is that at the average medium sized company code looks like this - you have 1mln lines of code written over a decade by a few hundred people. A big portion of the code is redundant, some of it is incomplete, much of it is undocumented. Different companies have different coding styles, different testing approaches, different development dynamics. ChatGPT does not appreciate this context. Excel has some similar problems. First of all Excel is 2 dimensional and LLMs really don’t think in 2 dimensions well. So you need to flatten the excel file for the LLM. A common approach to do this with LLMs is using pandas and then using the column and row names to index into the excel. Unfortunately, excels at companies cannot be easily read using pandas. They are illogically structured, have tons of hardcoding, intersheet referencing is weird circular ways and so on. I spent some time in finance and sell side equity research models are written by highly trained financial analysts and are substantially better organized than the average excel model at a company. Even this subset of real world models is far from suitable for a direct pandas interpretation. Parsing sell side models requires a delicate and complex interpretation before being fed into an LLM.

mdorazio 2y ago

> Can ChatGPT materially and positively impact the code written by big companies? It already has at a Fortune 100 company I contract with currently. > Can it do meaningful work in excel? We can quibble about what "meaningful" means, but it satisfactorily answered questions for two friends about how to build formulas for their datasets and is currently being used to summarize data insights from a database at a different large client (Excel =/= database, but the point stands). > Can it do meaningful PowerPoint work? I've used Midjourney multiple times a month to generate base imagery for various things in PowerPoint (usually requires modification in Photoshop, but saves me several hours each time compared to digital painting or photobashing from scratch). > Can it give effective advice on management? Again, what does "effective" mean in the context of management? I've seen VP-level individuals with hundreds of people in their orgs using AI tools for different things. It really feels like a significant chunk of the HN crowd is living in a bubble with respect to AI in the real world right now. It's absolutely invading everything. As for how much revenue that will translate into long-term vs. the investment dollars being poured into it, that's a more interesting question to discuss.

reaperman 2y ago

> Can it give effective advice on management? My friends at McKinsey say that while it can’t fine-tune reports and presentations with quite enough nuance, it does a good job sifting through lots of shit to pick out important parts they should pay more attention to, highlighting data/talking points that contradict a working hypothesis, assisting in writing emails, and other time-consuming or very nit-picky tasks. That said, no one I know has fed it real customer data, that would be a career-ending event. But self-hosted models like Gemma2 open up the possibility for using LLMs against real customer info.

munificent 2y ago
Hard to tell if that says more about the value of LLMs or the lack of value of McKinsey...
- throwaway2037 2y ago
  > the lack of value of McKinsey Leaving McKinsey's specific brand value aside, people always miss the value of "hiring (business) consultants". You basically get insider knowledge about how competitors businesses and systems work. So if you ask for advice about how to build a healthcare app for smartphones, you hire McKinsey (or whomever) to tell you "about the market". But really, they are just telling you about what they saw at other competitors. For some business decisions, it is very valuable.

ern 2y ago

Can it do meaningful PowerPoint work Yes, it absolutely can. I threw together a PowerPoint presentation with a script for a low-value, high visibility meeting a couple of weeks ago with ChatGPT 4.0 and a PowerPoint plugin. Everyone loved it.

throwaway2037 2y ago
> low-value, high visibility meeting This is such a gem. Can you tell us more about the meeting? A senior manager kicking the tyres, or what? Any funny bike-shedding stories to tell?
- ern 2y ago
  Corporate values. Drew the short straw, but had to present something.

nl 2y ago

> Can it do meaningful work in excel? Yes it can. But more importantly have you tried ChatGPT Data Analyst?: https://openai.com/index/improvements-to-data-analysis-in-chatgpt/ https://openai.com/index/improvements-to-data-analysis-in-ch... It drops the barrier for "pretty good data analysis" to effectively zero. > Can it do meaningful PowerPoint work? Canva and Figma are both building this and they are pretty decent right now. Better than most PowerPoints I've seen. The aforementioned Data Analyst does good presentations in a different way, too. > Can it give effective advice on management? Yes. Unfortunately can't talk about this except it is mindblowingly good.

TeMPOraL 2y ago
> The aforementioned Data Analyst does good presentations in a different way, too. And on top of that, it can do PowerPoint presentations too - magic keywords are "use Python and Pandoc".

fragmede 2y ago

> Being able to create 2048 in 6 hours has basically zero economic value. That's actually a really good point. In the realm of programming, things that were previously not done because they were too expensive can now be done. Prior to ChatGPT, GP could have a) done it themselves, but the cost was too high/it wasn't worth their time, b) found enough time to write a spec, found someone on upwork/etc, paid them to make it, except that costs money they didn't want to spend, or c) just not do it. Now, GP can code this thing up while watching netflix with the kids or whatever. What programs do not exist that previously did not have the economic value to exist, but now can, thanks to programming time getting cheaper? Now apply that to fields outside of programming. LLMs' ability to program is front and center here, since many of us can program, but they do other things as well.