6 ms·

I do not intend to take anything away from the technical achievement of the team. However, as Satya opined some weeks back, these benchmarks do not mean a lot i

I do not intend to take anything away from the technical achievement of the team. However, as Satya opined some weeks back, these benchmarks do not mean a lot if we do not see a comparable increase in productivity. But then there are two questions. First, are the white collar workers specifically consultants, engineers responsible for increase in productivity? Or is the white collar workers at the very right tail e.g., scientists? I think consultants and engineers are using these technologies a lot. I think biologists at least are using these models a lot. But then where is the productivity increases?

ZeroTalent 2y ago
Also, why not compare to GPT-o3 in the benchmarks?
- KoolKat23 2y ago
  The models not really available.
  - ZeroTalent 2y ago
    they have access to o3, I do. Thousands of people do(tens of thousands at this point?). Come on. Compare to SOTA, when you're saying it's the best AI you have.
    - KoolKat23 2y ago
      I honestly wasn't aware it's available to the few outside of pro.
KoolKat23 2y ago
As it stands currently I'd say this is difficult to measure. They're not baked into workflows where the measurable output is attributed easily to the model use. Productivity in its current form is transformative in the sense that the use case and gain differs for the individual (who even provide different prompts). So some are keeping the gains for themselves, others are using it to improve quality rather than quantity. It'll come in time, it's important to remember gpt 4 was released 2 years ago this month. The newer models are more reliable and could probably be introduced into workflows more frequently. Today I spoke to a company who are looking to use it to reduce cost in the next year.
danpalmer 2y ago
That’s true, but productivity has many factors and takes a long time to get confidence on. Any productivity value that could be stated clearly would have similar downsides to a benchmark, and take far longer. Benchmarks are useful as leading indicators. Early warning signs. If there’s no relation to the eventual productivity then hopefully that benchmark will disappear as it’s not useful. In a fast moving space like this it’s reasonable to make use of leading indicators.
weatherlite 2y ago
It's a complex proposition. I think Satya was talking about actual gdp growth right ? In theory lets say all knowledge work is now 50% faster due to A.I. Well then I would assume this should affect civil society as well - planning a bridge, a railway etc should happen faster and more efficiently (the actual building of thins won't, but a lot of time is spent on planning a red tape). Healthcare in general should become way more efficient with people getting better treatment; this should have a positive economic effect. It does seem to me like it should be able to speed things up in the real world but of course a lot will have to do with how well the models can reason / how often they make catastrophic mistakes + the will of the governments and people to start using them seriously. But its more complex than that - if many people start losing their jobs we all take a hit on gdp because they can't consume as much anymore, so it could take perhaps a long time until gdp actually sees meaningful gains. And one last thought - Satya likely hasn't spent much time thinking about gdp, it's just not his field. He's a smart guy for sure but this isn't what he does.
- eru 2y ago
  The problem is slightly different. Unemployment hasn't really picked up, and is unlikely to do so, unless the central bank is incompetent. (They have been from time to time.) However, some advances don't show up in GDP. Eg Wikipedia is a tremendous achievement. But nobody pays for it, so it doesn't show up in GDP statistics.
  - weatherlite 2y ago
    > Unemployment hasn't really picked up, and is unlikely to do so That's an important assessment. I don't know if you're right. If the models are going to continue to get more capable I'm expecting unemployment to rise , I don't see how it won't (sure we are promised A.I to create tons of new jobs no one has imagined yet, I haven't seen a reliable clue for such jobs yet).
    - eru 2y ago
      No, it won't (necessarily) be AI that's creating the new jobs. In general, when a new technology comes along and automates away some jobs, you can't expect the same technology to provide the new jobs. To give an example from the recent past: 'hipster' baristas that make you a five dollar coffee are a fairly new job. At least at scale. But I doubt you'll be able to find any technology that automated some other job but created barista jobs. It's just that the market will find stuff for people to do for money, unless prevented to do so by incompetent central bank policy or (too) onerous labour market regulation. (The labour market can take quite a lot of regulation, and still be able to get people jobs. Have a look at Germany today for an example.)

ZeroTalent 2y ago

Also, why not compare to GPT-o3 in the benchmarks?

KoolKat23 2y ago
The models not really available.
- ZeroTalent 2y ago
  they have access to o3, I do. Thousands of people do(tens of thousands at this point?). Come on. Compare to SOTA, when you're saying it's the best AI you have.
  - KoolKat23 2y ago
    I honestly wasn't aware it's available to the few outside of pro.

KoolKat23 2y ago

As it stands currently I'd say this is difficult to measure. They're not baked into workflows where the measurable output is attributed easily to the model use. Productivity in its current form is transformative in the sense that the use case and gain differs for the individual (who even provide different prompts). So some are keeping the gains for themselves, others are using it to improve quality rather than quantity. It'll come in time, it's important to remember gpt 4 was released 2 years ago this month. The newer models are more reliable and could probably be introduced into workflows more frequently. Today I spoke to a company who are looking to use it to reduce cost in the next year.

danpalmer 2y ago

That’s true, but productivity has many factors and takes a long time to get confidence on. Any productivity value that could be stated clearly would have similar downsides to a benchmark, and take far longer. Benchmarks are useful as leading indicators. Early warning signs. If there’s no relation to the eventual productivity then hopefully that benchmark will disappear as it’s not useful. In a fast moving space like this it’s reasonable to make use of leading indicators.

weatherlite 2y ago

It's a complex proposition. I think Satya was talking about actual gdp growth right ? In theory lets say all knowledge work is now 50% faster due to A.I. Well then I would assume this should affect civil society as well - planning a bridge, a railway etc should happen faster and more efficiently (the actual building of thins won't, but a lot of time is spent on planning a red tape). Healthcare in general should become way more efficient with people getting better treatment; this should have a positive economic effect. It does seem to me like it should be able to speed things up in the real world but of course a lot will have to do with how well the models can reason / how often they make catastrophic mistakes + the will of the governments and people to start using them seriously. But its more complex than that - if many people start losing their jobs we all take a hit on gdp because they can't consume as much anymore, so it could take perhaps a long time until gdp actually sees meaningful gains. And one last thought - Satya likely hasn't spent much time thinking about gdp, it's just not his field. He's a smart guy for sure but this isn't what he does.

eru 2y ago
The problem is slightly different. Unemployment hasn't really picked up, and is unlikely to do so, unless the central bank is incompetent. (They have been from time to time.) However, some advances don't show up in GDP. Eg Wikipedia is a tremendous achievement. But nobody pays for it, so it doesn't show up in GDP statistics.
- weatherlite 2y ago
  > Unemployment hasn't really picked up, and is unlikely to do so That's an important assessment. I don't know if you're right. If the models are going to continue to get more capable I'm expecting unemployment to rise , I don't see how it won't (sure we are promised A.I to create tons of new jobs no one has imagined yet, I haven't seen a reliable clue for such jobs yet).
  - eru 2y ago
    No, it won't (necessarily) be AI that's creating the new jobs. In general, when a new technology comes along and automates away some jobs, you can't expect the same technology to provide the new jobs. To give an example from the recent past: 'hipster' baristas that make you a five dollar coffee are a fairly new job. At least at scale. But I doubt you'll be able to find any technology that automated some other job but created barista jobs. It's just that the market will find stuff for people to do for money, unless prevented to do so by incompetent central bank policy or (too) onerous labour market regulation. (The labour market can take quite a lot of regulation, and still be able to get people jobs. Have a look at Germany today for an example.)