4 ms·

Incidentally I was about to say that even speculation based on present utility can get a little bit out there. Modern AI is already quite good at generating co

Incidentally I was about to say that even speculation based on present utility can get a little bit out there. Modern AI is already quite good at generating code snippets, and it already increases developer productivity, but just how much is that going to really change things? Just how much more productive is an AI-augmented software developer? Are LLMs really the thing that was missing from no-code website / app builders? I wouldn't say it's obvious that this stuff hasn't been scaled to the limit. I mean, sure, you could make bigger LLMs, but I don't think it's a given that LLM performance will increase in performance in a predictable fashion. If anything I think that performance probably resembles an S curve where performance matches improvements in scale before sharply plateauing. When's the plateau? GPT-4 is a clear improvement over GPT-3, but it's also 6x larger, and so far I haven't quite seen any real evidence to suggest that it's 6x better, other than the fact that it scored better on tests.

int_19h 4y ago
It's hard to say what "6x better" even means, but there are tasks that GPT-3.5 consistently fails hard at that GPT-4 is much better at (and has improved over time). I think the next big leap will be the context size. Going from 4k to 32k tokens means that the model can see a lot more when generating the answer. For code especially this makes a big difference in the ability to reason about it. As to the limits, it's hard to predict because what we're seeing is more of a series of sudden leaps. But let me put it this way - so far we've seen noticeable improvements on every iteration of the new LLMs. So given this, why should the default assumption be that the limit past which nothing new will emerge, whatever it is, is close to the present size of the publicly available models?
- throwaway2037 4y ago
  Is anyone doing static code analysis with GPT? It seems like an excellent idea. I think a combination of formal analysis methods together with blackbox GPT magic will be a killer combination. That said, the market is tiny.
MacsHeadroom 4y ago
There is zero evidence that GPT-4 is a trillion parameter model and some evidence that it is the same parameter size as GPT-3. Newer research and models out of DeepMind etc. show that GPT-4 performance is attainable at 90% the size of GPT-3. It's well known by now that GPT-3 was hugely over parameterized.

int_19h 4y ago

It's hard to say what "6x better" even means, but there are tasks that GPT-3.5 consistently fails hard at that GPT-4 is much better at (and has improved over time). I think the next big leap will be the context size. Going from 4k to 32k tokens means that the model can see a lot more when generating the answer. For code especially this makes a big difference in the ability to reason about it. As to the limits, it's hard to predict because what we're seeing is more of a series of sudden leaps. But let me put it this way - so far we've seen noticeable improvements on every iteration of the new LLMs. So given this, why should the default assumption be that the limit past which nothing new will emerge, whatever it is, is close to the present size of the publicly available models?

throwaway2037 4y ago
Is anyone doing static code analysis with GPT? It seems like an excellent idea. I think a combination of formal analysis methods together with blackbox GPT magic will be a killer combination. That said, the market is tiny.

MacsHeadroom 4y ago

There is zero evidence that GPT-4 is a trillion parameter model and some evidence that it is the same parameter size as GPT-3. Newer research and models out of DeepMind etc. show that GPT-4 performance is attainable at 90% the size of GPT-3. It's well known by now that GPT-3 was hugely over parameterized.