5 ms·

Two things mentioned in the abstract that are worth pointing out. > For each one of a set of 18 realistic consulting tasks within the frontier of AI capabiliti

Two things mentioned in the abstract that are worth pointing out. > For each one of a set of 18 realistic consulting tasks within the frontier of AI capabilities They specifically picked tasks that GPT-4 was capable of doing. GPT-4 could not do many tasks, so when we say that performance was significantly increased this is only for tasks GPT-4 is well suited to. There is still value here but let's put these results into context. > Consultants across the skills distribution benefited significantly from having AI augmentation, with those below the average performance threshold increasing by 43% and those above increasing by 17% compared to their own scores Even when cherry-picking tasks that GPT-4 is particularly suited for, above average performers only increased performance by 17%. This increase is still impressive, were it to be seen across the board. But I do think that 17% is a lot less than some people are trying to sell.

ellyagg 3y ago
You're underestimating because it compounds. Small gains in efficency lead to huge advantages in long term growth. 17% would be absolutely monumental improvement.
- visarga 3y ago
  yet insignificant compared to how full automation would look, not 1.17x but 1000x but I can't find any automated AI tasks of critical importance, they all need human support
denton-scratch 3y ago
Hmmm. Perhaps below-average performers are more likely to take GPT output at face-value, being less competent to review and edit it. And above-average performers are more likely to hack the GPT output around, because they're confident in their own abilities. Therefore below-average types will produce finished output more quickly; and this was a time-constrained test, so velocity matters. ChatGPT is very good at waffling, and marketing-speak and inspirational messages are essentially waffle. IOW, the tasks were tailor-made for unaided ChatGPT, so high-performers were penalized.

ellyagg 3y ago

You're underestimating because it compounds. Small gains in efficency lead to huge advantages in long term growth. 17% would be absolutely monumental improvement.

visarga 3y ago
yet insignificant compared to how full automation would look, not 1.17x but 1000x but I can't find any automated AI tasks of critical importance, they all need human support

denton-scratch 3y ago

Hmmm. Perhaps below-average performers are more likely to take GPT output at face-value, being less competent to review and edit it. And above-average performers are more likely to hack the GPT output around, because they're confident in their own abilities. Therefore below-average types will produce finished output more quickly; and this was a time-constrained test, so velocity matters. ChatGPT is very good at waffling, and marketing-speak and inspirational messages are essentially waffle. IOW, the tasks were tailor-made for unaided ChatGPT, so high-performers were penalized.