3 ms·

I've used GPT-4 before it changed, and though impressive, writing code has never been the bottleneck for me personally. When these things can understand the bus

I've used GPT-4 before it changed, and though impressive, writing code has never been the bottleneck for me personally. When these things can understand the business requirements and tell me what I should be building and why, with detailed sensible reasoning then I'll be hyped.

Takennickname 3y ago
You're missing out. 90% of my boilerplate code is automatically customized and inserted in my codebase for me by copilot. Maybe "manual labor" makes you happy, that's up to you.
Swizec 3y ago
> When these things can understand the business requirements I’ve been looking into this. Nothing definitive yet, but my hunch is that current LLM’s struggle because they lack curiosity. They will answer your question, however vague, with the first most obvious answer they can think of. This is great, if you’re a junior team member. Super talented, very eager. But a more senior engineer approaches the problem differently. They ask more questions than provide answers. They’ll happily spend the first 25min of a 30min meeting asking heaps upon oodles of really dumb sounding questions. Then the last 5min, that’s the magic. They now have a solution perfectly tailored to the problem at hand, with all sorts of edge cases either explored or eliminated through questioning. The questions they kept asking weren’t dumb after all, they were pruning a decision/option tree in their head of all possible solutions until they landed on the most optimal solution for a set of known constraints. With further options to dig and improve. I think you can make an LLM do this (i’m trying), but it’s very very slow still.
- agentcoops 3y ago
  I think you’re still anthropomorphizing LLMs too much by saying the problem is a lack of curiosity. It’s an engineering problem: you haven’t figured out the correct “context” from which business requirements would follow. (And why Microsoft is so incredibly well positioned for the future)
  - Swizec 3y ago
    Curiosity is how you build that context.
    - agentcoops 3y ago
      I'm sure in your career you've seen countless attempts by 1000+ employee firms to spinup an ops team that must construct a giant spreadsheet of product priorities between customers, PMs and engineering teams -- versus having all of Salesforce, Slack, emails, meeting transcripts, user support messages, vectorised and fed into an LLM that any employee can converse with. The latter is not a trivial problem by any means and many, many implementation details, but I'm sure it's such an exciting time to be at Microsoft. Stratechery had a fantastic article about this. If I had to make a bet, I suspect Google and Meta will royally lose this generation: Apple will dominate personal access to LLMs, Microsoft will dominate business access to LLMs -- and honestly Palantir for the first time terrifies me. The current generation of LLMs are not about automation as such, but helping individuals accomplish self-directed goals (as possible in language and code).
- pydry 3y ago
  Whatever it is it goes beyond curiosity - it doesnt just come up with the laziest answer it often invents it.
- pixl97 3y ago
  >because they lack curiosity. This is because curiosity requires free energy, hence it is very expensive when you're limited on very expensive compute. This is what tree of thought is attempting to simulate in some ways. Build a set of multiple questions around the original question and then build on and prune that list based on a 'show your work' set of steps, and then keep iterating. Humans naturally solve the halting problem when thinking about things... we work on something long enough without a break and we'll pass out. Maybe when we wake, eat, and go to work we'll stop working on the same problem. But an LLM never sleeps. In theory with TOT and no time limit, you could find out your AutoGPT spent 10 million in computing resources contemplating navel lint. So, there are a number of unsolved problems there. What really becomes concerning is if Nvidia achieves its goals of speeding up training/inference by 1 million times in the next few years, and if the amount of compute we produce increases by a few million times. You and me simply can't use hundreds of minds thinking for years straight and machines could.
naiv 3y ago
what do you mean with 'changed'?
- unshavedyak 3y ago
  A lot of people think GPT4 got worse with a semi-recent change. Worse as measured by intelligence or accuracy on complex inputs. Most notably i think the change was around the time they significantly increased the speed of GPT4. I don't have much opinion on this though, i just know i see that description of GPT4 a lot. I use GPT4 daily, but i think i don't really challenge it most days, as i've never trusted it enough to do so hah.

Takennickname 3y ago

You're missing out. 90% of my boilerplate code is automatically customized and inserted in my codebase for me by copilot. Maybe "manual labor" makes you happy, that's up to you.

Swizec 3y ago

> When these things can understand the business requirements I’ve been looking into this. Nothing definitive yet, but my hunch is that current LLM’s struggle because they lack curiosity. They will answer your question, however vague, with the first most obvious answer they can think of. This is great, if you’re a junior team member. Super talented, very eager. But a more senior engineer approaches the problem differently. They ask more questions than provide answers. They’ll happily spend the first 25min of a 30min meeting asking heaps upon oodles of really dumb sounding questions. Then the last 5min, that’s the magic. They now have a solution perfectly tailored to the problem at hand, with all sorts of edge cases either explored or eliminated through questioning. The questions they kept asking weren’t dumb after all, they were pruning a decision/option tree in their head of all possible solutions until they landed on the most optimal solution for a set of known constraints. With further options to dig and improve. I think you can make an LLM do this (i’m trying), but it’s very very slow still.

agentcoops 3y ago
I think you’re still anthropomorphizing LLMs too much by saying the problem is a lack of curiosity. It’s an engineering problem: you haven’t figured out the correct “context” from which business requirements would follow. (And why Microsoft is so incredibly well positioned for the future)
- Swizec 3y ago
  Curiosity is how you build that context.
  - agentcoops 3y ago
    I'm sure in your career you've seen countless attempts by 1000+ employee firms to spinup an ops team that must construct a giant spreadsheet of product priorities between customers, PMs and engineering teams -- versus having all of Salesforce, Slack, emails, meeting transcripts, user support messages, vectorised and fed into an LLM that any employee can converse with. The latter is not a trivial problem by any means and many, many implementation details, but I'm sure it's such an exciting time to be at Microsoft. Stratechery had a fantastic article about this. If I had to make a bet, I suspect Google and Meta will royally lose this generation: Apple will dominate personal access to LLMs, Microsoft will dominate business access to LLMs -- and honestly Palantir for the first time terrifies me. The current generation of LLMs are not about automation as such, but helping individuals accomplish self-directed goals (as possible in language and code).
pydry 3y ago
Whatever it is it goes beyond curiosity - it doesnt just come up with the laziest answer it often invents it.
pixl97 3y ago
>because they lack curiosity. This is because curiosity requires free energy, hence it is very expensive when you're limited on very expensive compute. This is what tree of thought is attempting to simulate in some ways. Build a set of multiple questions around the original question and then build on and prune that list based on a 'show your work' set of steps, and then keep iterating. Humans naturally solve the halting problem when thinking about things... we work on something long enough without a break and we'll pass out. Maybe when we wake, eat, and go to work we'll stop working on the same problem. But an LLM never sleeps. In theory with TOT and no time limit, you could find out your AutoGPT spent 10 million in computing resources contemplating navel lint. So, there are a number of unsolved problems there. What really becomes concerning is if Nvidia achieves its goals of speeding up training/inference by 1 million times in the next few years, and if the amount of compute we produce increases by a few million times. You and me simply can't use hundreds of minds thinking for years straight and machines could.

naiv 3y ago

what do you mean with 'changed'?

unshavedyak 3y ago
A lot of people think GPT4 got worse with a semi-recent change. Worse as measured by intelligence or accuracy on complex inputs. Most notably i think the change was around the time they significantly increased the speed of GPT4. I don't have much opinion on this though, i just know i see that description of GPT4 a lot. I use GPT4 daily, but i think i don't really challenge it most days, as i've never trusted it enough to do so hah.