4 ms·

This feels fairly naive, ignoring how much progress has happened over the (short) span of one year. This doesn't sound like that tough of a gap to close in anot

This feels fairly naive, ignoring how much progress has happened over the (short) span of one year. This doesn't sound like that tough of a gap to close in another year (again, projecting based off recent progress).

lukev 3y ago
The bitter truth though is that the last 20% always takes 99% of the time and effort. LLMs are a huge breakthrough and we are absolutely speedrunning the 80%, but I see no reason to believe the usual pattern won’t hold when the easy wins are exploited.
- realce 3y ago
  Some insurance against this dam breaking wouldn't be so bad however
TheRoque 3y ago
The opposite is fairly naive. Software development is not only dumping tokens into a text file. To have a significant impact on the market, it should do much, much, much more: compile and test code, automatically assess the quality of what its done, be aware of the current design trends (if in UI/UX), ideally innovate, it should also be able to run a debugger, inspect all the variables, and deduce from there how it got something wrong, sometimes with tiny clues that I don't even know how it would get its information (e.g. in graphics programming where you have to actually see at a high frame rate). Oh snap a library is broken ? The AI needs to search online why it's broken, then find a fix (log onto a website to communicate with support, install a missing dep...). It can't be fixed ? Then the AI needs to explain this to the manager, good luck for that. It would need to think and feel like a human, otherwise producing uncanny content that will be either boring, either creepy. You can think about your daily job and break down all the tasks, and you'll quickly realize that replacing all this is just a monstrous task.
- jader201 3y ago
  Yeah, I definitely am not on team “We’re Doomed”, but I also can’t say definitively that I’m on team “We’re Fine” either. I think there are merits to both arguments, and I think it’s possible that we’ll see things move towards either direction in the next 1/5/10 years. My point is, I don’t think we can rule out the possibility of some jobs being at risk within the next 1/5/10 years.
  - TheRoque 3y ago
    Some jobs are definitely at risk, I was just making the case for software development. But just like you, even after writing all this, there's still some anxiety.
- famouswaffles 3y ago
  https://arxiv.org/abs/2309.12499 https://arxiv.org/abs/2309.12499
- jabradoodle 3y ago
  In a market you don't need to replace it, you replace millions of hours of building/testing/documenting apps and you've effectively reduced demand for SWE labour (ignoring the potential for new job creation).
WendyTheWillow 3y ago
What actually was the innovation in LLMs that produced the kind of AI we're seeing now? Is that innovation ongoing or did it happen, and now we're seeing the various optimizations of that innovation? Is voice and image integration with ChatGPT a whole new capability of LLMs or is the "product" here a clean and intuitive interface through which to use the already existent technology? The difference between GPT 3, 3.5, and 4 is substantially smaller than the difference between GPT 2 and GPT 3, and Sam Altman has directly said there are no plans for a GPT 5. I don't think progress is linear here. Rather, it seems more likely that we made the leap about a year or so ago, and are currently in the process of applying that leap in many different ways. But the leap happened, and there isn't seemingly another one coming.
- famouswaffles 3y ago
  >What actually was the innovation in LLMs that produced the kind of AI we're seeing now? Is that innovation ongoing or did it happen, and now we're seeing the various optimizations of that innovation? Past the introduction of the transformer in 2017, There is no big "innovation". It is just scale. Bigger models are better. The last 4 years can be summed up that simply. >Is voice and image integration with ChatGPT a whole new capability of LLMs or is the "product" here a clean and intuitive interface through which to use the already existent technology? What is existing technology here ? Open ai aren't doing anything so alien you couldn't guess at if you knew what you were doing but image training at the scale of GPT-4 is new and it's not even the cleanest way to do it. We still don't have a "trained from scratch" large scale multimodal LLM yet. >The difference between GPT 3, 3.5, and 4 is substantially smaller than the difference between GPT 2 and GPT 3 Definitely not lol. The OG GPT-3 was pulling sub 50 on MMLU. Even benchmarks aside, there is a massive gap in utility between 3.5 and 4, never mind 3. 4 was finished training august 2022. It's only 2 years apart from 3. >I don't think progress is linear here. Rather, it seems more likely that we made the leap about a year or so ago, and are currently in the process of applying that leap in many different ways. But the leap happened, and there isn't seemingly another one coming. There was no special leap (in terms of theory and engineering). This is scale plainly laid out and there's more of it to go. >and Sam Altman has directly said there are no plans for a GPT 5. the same that sat on 4 for 8 months and said absolutely nothing about it ? Take anything altman says about new iterations with a grain of salt.

lukev 3y ago

The bitter truth though is that the last 20% always takes 99% of the time and effort. LLMs are a huge breakthrough and we are absolutely speedrunning the 80%, but I see no reason to believe the usual pattern won’t hold when the easy wins are exploited.

realce 3y ago
Some insurance against this dam breaking wouldn't be so bad however

TheRoque 3y ago

The opposite is fairly naive. Software development is not only dumping tokens into a text file. To have a significant impact on the market, it should do much, much, much more: compile and test code, automatically assess the quality of what its done, be aware of the current design trends (if in UI/UX), ideally innovate, it should also be able to run a debugger, inspect all the variables, and deduce from there how it got something wrong, sometimes with tiny clues that I don't even know how it would get its information (e.g. in graphics programming where you have to actually see at a high frame rate). Oh snap a library is broken ? The AI needs to search online why it's broken, then find a fix (log onto a website to communicate with support, install a missing dep...). It can't be fixed ? Then the AI needs to explain this to the manager, good luck for that. It would need to think and feel like a human, otherwise producing uncanny content that will be either boring, either creepy. You can think about your daily job and break down all the tasks, and you'll quickly realize that replacing all this is just a monstrous task.

jader201 3y ago
Yeah, I definitely am not on team “We’re Doomed”, but I also can’t say definitively that I’m on team “We’re Fine” either. I think there are merits to both arguments, and I think it’s possible that we’ll see things move towards either direction in the next 1/5/10 years. My point is, I don’t think we can rule out the possibility of some jobs being at risk within the next 1/5/10 years.
- TheRoque 3y ago
  Some jobs are definitely at risk, I was just making the case for software development. But just like you, even after writing all this, there's still some anxiety.
famouswaffles 3y ago
https://arxiv.org/abs/2309.12499 https://arxiv.org/abs/2309.12499
jabradoodle 3y ago
In a market you don't need to replace it, you replace millions of hours of building/testing/documenting apps and you've effectively reduced demand for SWE labour (ignoring the potential for new job creation).

WendyTheWillow 3y ago

What actually was the innovation in LLMs that produced the kind of AI we're seeing now? Is that innovation ongoing or did it happen, and now we're seeing the various optimizations of that innovation? Is voice and image integration with ChatGPT a whole new capability of LLMs or is the "product" here a clean and intuitive interface through which to use the already existent technology? The difference between GPT 3, 3.5, and 4 is substantially smaller than the difference between GPT 2 and GPT 3, and Sam Altman has directly said there are no plans for a GPT 5. I don't think progress is linear here. Rather, it seems more likely that we made the leap about a year or so ago, and are currently in the process of applying that leap in many different ways. But the leap happened, and there isn't seemingly another one coming.

famouswaffles 3y ago
>What actually was the innovation in LLMs that produced the kind of AI we're seeing now? Is that innovation ongoing or did it happen, and now we're seeing the various optimizations of that innovation? Past the introduction of the transformer in 2017, There is no big "innovation". It is just scale. Bigger models are better. The last 4 years can be summed up that simply. >Is voice and image integration with ChatGPT a whole new capability of LLMs or is the "product" here a clean and intuitive interface through which to use the already existent technology? What is existing technology here ? Open ai aren't doing anything so alien you couldn't guess at if you knew what you were doing but image training at the scale of GPT-4 is new and it's not even the cleanest way to do it. We still don't have a "trained from scratch" large scale multimodal LLM yet. >The difference between GPT 3, 3.5, and 4 is substantially smaller than the difference between GPT 2 and GPT 3 Definitely not lol. The OG GPT-3 was pulling sub 50 on MMLU. Even benchmarks aside, there is a massive gap in utility between 3.5 and 4, never mind 3. 4 was finished training august 2022. It's only 2 years apart from 3. >I don't think progress is linear here. Rather, it seems more likely that we made the leap about a year or so ago, and are currently in the process of applying that leap in many different ways. But the leap happened, and there isn't seemingly another one coming. There was no special leap (in terms of theory and engineering). This is scale plainly laid out and there's more of it to go. >and Sam Altman has directly said there are no plans for a GPT 5. the same that sat on 4 for 8 months and said absolutely nothing about it ? Take anything altman says about new iterations with a grain of salt.