4 ms·

We need to check the gen AI hype and get back to reality

12 points by mpweiher 2y ago

Madmallard 2y ago
We need to talk about something else
tracerbulletx 2y ago
This point of view will be wrong in the long term. There are about 10 billion obvious things you can do with transformers as they are today that just haven't been done yet, and it's pretty likely one of these labs will iterate on transformers and make them much more powerful.
- boppo1 2y ago
  Can you elaborate?
  - tracerbulletx 2y ago
    The power for transformers is the generalizability of being able to learn a function to transform an arbitrary input signal in any modality into an arbitrary output signal in any other modality. Chat is fine, image generators fine, universal translation was more than fine. But any thing that can have a symbolic representation is a target for creating a new product using transformers. I think the big value will come from some random obscure places like CAD or real time graphics pipelines. I just think it will be a victory of a million paper cuts where it gets more cost effective than the alternatives in a million various industries. It will just be a ubiquitous technique when solving computing problems and people are going to create more specific datasets for specific problems and continue to get each industrial usage to a level where it's good enough at each task to change each market.
    - benterix 2y ago
      People are already trying that with CAD and so on. Still, the main challenges are still there: enormous power needed to train the model (within reasonable time) and relatively powerful hardware to actually use it, inconsistency in output, bugs that you can't meaningfully debug. You might be right about many things above but so far traditional algorithms remain a far better option for many of these use cases (deterministic and efficient).
cynicalsecurity 2y ago
The article is nothing new, and it doesn't correspond to its title. Clickbait.
say_it_as_it_is 2y ago
> This study evaluates the impact of generative AI on software developer productivity by analyzing data from three randomized controlled trials conducted at Microsoft, Accenture, and an anonymous Fortune 100 electronics manufacturing company. These field experiments, which were run by the companies as part of their ordinary course of business, provided a randomly selected subset of developers with access to GitHub Copilot, an AI-based coding assistant that suggests intelligent code completions. Though each separate experiment is noisy, combined across all three experiments and 4,867 software developers, our analysis reveals a 26.08% increase (SE: 10.3%) in the number of completed tasks among developers using the AI tool. Notably, less experienced developers showed higher adoption rates and greater productivity gains. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4945566 https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4945566
- lumost 2y ago
  A 26% productivity increase is... pretty big? There is obviously some noise in the data. But given copilots are pretty new pieces of tech, it does not seem impossible that there could be at least a 2x productivity growth available within the next 6-8 years. Given that SWE work is not particularly differentiated from CAD and similar types of knowledge work... is it really that unreasonable that AI could double the productivity of knowledge workers by 2030? That's certainly worth a few trillion dollars in economic growth.
- svachalek 2y ago
  The last statement rings very true for me. When I'm coding up something in my bread and butter language and libraries, current LLMs can rarely help me do something faster than I can do it myself. But when I'm trying to write a script in an unfamiliar language, it's much easier to let an LLM do it and fix the parts it got wrong, than to figure out how to write it all in the first place.
abletonlive 2y ago
Meh. The first two points the author fails to explain how that’s different from any human giving you information. The last point fails to acknowledge that people actually like Uber over cabs for the most part, despite what they might cost.
- rtkwe 2y ago
  For the first the big difference between genAI hallucinations and people imo is that you can meaningfully reason with a person, address where they may be getting incorrect information, and assuming they're willing get them to do better in the future. None of that is really possible with things like ChatGPT at the moment. You can't teach it anything you have to wait for a new version. You can give some preambles that get it to work a little better, thinking logically or showing step by step, but it still can't count the number of Rs in strawberry.
  - gaganyaan 2y ago
    The r's in strawberry thing is an artifact of tokenization. Ask it to spell it out with dashes between the letters or something first and it'll do fine.
    - lostmsu 2y ago
      it's not even that, the question is ambiguous out of context.
tsurba 2y ago
The writer should stick with nontechnical points since they clearly don’t understand why the models currently used are not deterministic. It’s trivial to make them deterministic.
- Techtsunami 2y ago
  That's interesting! Any good examples of LLMs that are deterministic?
  - xnx 2y ago
    Any LLM that supports an input seed will return the same outputs given the same inputs (prompt, temperature, etc.)
  - auggierose 2y ago
    ChatGPT is. Use it through the API.
baggachipz 2y ago
Thus begins the trough of disillusionment in the Gartner Hype Cycle.
JeremyHerrman 2y ago
> remember how cheap Uber was in 2008? ah yes, uber was so cheap checks notes... 2 years before it raised its $1.25m seed round in 2010...
Haugsevje 2y ago
On the contrary! My gut feeling is that the AI hype has already paid itself in saved work hours and more. He joins the ranks of middle aged men scared that their know-how is loosing value (which it is).
dangmustdie 2y ago
[flagged]

12 points by mpweiher 2y ago

We need to talk about something else

This point of view will be wrong in the long term. There are about 10 billion obvious things you can do with transformers as they are today that just haven't been done yet, and it's pretty likely one of these labs will iterate on transformers and make them much more powerful.

boppo1 2y ago
Can you elaborate?
- tracerbulletx 2y ago
  The power for transformers is the generalizability of being able to learn a function to transform an arbitrary input signal in any modality into an arbitrary output signal in any other modality. Chat is fine, image generators fine, universal translation was more than fine. But any thing that can have a symbolic representation is a target for creating a new product using transformers. I think the big value will come from some random obscure places like CAD or real time graphics pipelines. I just think it will be a victory of a million paper cuts where it gets more cost effective than the alternatives in a million various industries. It will just be a ubiquitous technique when solving computing problems and people are going to create more specific datasets for specific problems and continue to get each industrial usage to a level where it's good enough at each task to change each market.
  - benterix 2y ago
    People are already trying that with CAD and so on. Still, the main challenges are still there: enormous power needed to train the model (within reasonable time) and relatively powerful hardware to actually use it, inconsistency in output, bugs that you can't meaningfully debug. You might be right about many things above but so far traditional algorithms remain a far better option for many of these use cases (deterministic and efficient).

cynicalsecurity 2y ago

The article is nothing new, and it doesn't correspond to its title. Clickbait.

say_it_as_it_is 2y ago

> This study evaluates the impact of generative AI on software developer productivity by analyzing data from three randomized controlled trials conducted at Microsoft, Accenture, and an anonymous Fortune 100 electronics manufacturing company. These field experiments, which were run by the companies as part of their ordinary course of business, provided a randomly selected subset of developers with access to GitHub Copilot, an AI-based coding assistant that suggests intelligent code completions. Though each separate experiment is noisy, combined across all three experiments and 4,867 software developers, our analysis reveals a 26.08% increase (SE: 10.3%) in the number of completed tasks among developers using the AI tool. Notably, less experienced developers showed higher adoption rates and greater productivity gains. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4945566 https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4945566

lumost 2y ago
A 26% productivity increase is... pretty big? There is obviously some noise in the data. But given copilots are pretty new pieces of tech, it does not seem impossible that there could be at least a 2x productivity growth available within the next 6-8 years. Given that SWE work is not particularly differentiated from CAD and similar types of knowledge work... is it really that unreasonable that AI could double the productivity of knowledge workers by 2030? That's certainly worth a few trillion dollars in economic growth.
svachalek 2y ago
The last statement rings very true for me. When I'm coding up something in my bread and butter language and libraries, current LLMs can rarely help me do something faster than I can do it myself. But when I'm trying to write a script in an unfamiliar language, it's much easier to let an LLM do it and fix the parts it got wrong, than to figure out how to write it all in the first place.

abletonlive 2y ago

Meh. The first two points the author fails to explain how that’s different from any human giving you information. The last point fails to acknowledge that people actually like Uber over cabs for the most part, despite what they might cost.

rtkwe 2y ago
For the first the big difference between genAI hallucinations and people imo is that you can meaningfully reason with a person, address where they may be getting incorrect information, and assuming they're willing get them to do better in the future. None of that is really possible with things like ChatGPT at the moment. You can't teach it anything you have to wait for a new version. You can give some preambles that get it to work a little better, thinking logically or showing step by step, but it still can't count the number of Rs in strawberry.
- gaganyaan 2y ago
  The r's in strawberry thing is an artifact of tokenization. Ask it to spell it out with dashes between the letters or something first and it'll do fine.
  - lostmsu 2y ago
    it's not even that, the question is ambiguous out of context.

tsurba 2y ago

The writer should stick with nontechnical points since they clearly don’t understand why the models currently used are not deterministic. It’s trivial to make them deterministic.

Techtsunami 2y ago
That's interesting! Any good examples of LLMs that are deterministic?
- xnx 2y ago
  Any LLM that supports an input seed will return the same outputs given the same inputs (prompt, temperature, etc.)
- auggierose 2y ago
  ChatGPT is. Use it through the API.

baggachipz 2y ago

Thus begins the trough of disillusionment in the Gartner Hype Cycle.

JeremyHerrman 2y ago

> remember how cheap Uber was in 2008? ah yes, uber was so cheap checks notes... 2 years before it raised its $1.25m seed round in 2010...

Haugsevje 2y ago

On the contrary! My gut feeling is that the AI hype has already paid itself in saved work hours and more. He joins the ranks of middle aged men scared that their know-how is loosing value (which it is).

dangmustdie 2y ago

[flagged]