8 ms·

Again, this is a misunderstanding of what LLMs are capable of. These aren't chained, you can run parallel prompts of 15 personas with X diversity of perspective

Again, this is a misunderstanding of what LLMs are capable of. These aren't chained, you can run parallel prompts of 15 personas with X diversity of perspective, that reason on a singular request, string, input or variable, they provide output plus audit explanation. You then run an amalgamation or committee decision (sort of like mixture of experts) on it to output variable or string. Then run parallel simulation or reflection prompts based on X different context personas to double check their application to outside cases, unconsidered context, etc. It's pretty effective on complex problems like Spam, Trust and Safety, etc. And the applications of these sort of reasoning atomic configurations I think are unlimited. It's not just 'talking fancy' to an AI, its building processes that systematically improve reasoning to different very hard applied problems.

ironborn123 3y ago
Brave are those who have set out to make prompt engg an entire industry, with gpt-5 and gemini lurking on the horizon.
- tsunamifury 3y ago
  Sam has specifically stated it's unlikely there will be a GPT5, and likely GPT4 is just a deeply prompt optimized and multimode version of GPT3. But overall, hasn't that theme been true for like... all tech ever? You have to set up and build your own innovation path at some point.
  - matmulbro 3y ago
    [flagged]
dlkf 3y ago
> the applications of these sort of reasoning atomic configurations I think are unlimited They are limited to applications in which the latency slo is O(seconds), knowledge of 2021-present doesn’t matter, and you’re allowed to make things up when you don’t know the answer. There are, to be fair, many such applications. But it’s not unlimited.
- tsunamifury 3y ago
  Bounded =/= not unlimited
carlitossway 3y ago
Interesting. Were you inspired by the Delphi decision algorithm, or did you happen to rediscover it? I agree it could provide a moat, though not a very wide one. All it takes is for your competition to talk to non-engineers, and all sorts of clever usages become possible. See https://en.wikipedia.org/wiki/Delphi_method https://en.wikipedia.org/wiki/Delphi_method
- tsunamifury 3y ago
  Accidentally discovered its effectiveness when configured with LLMs, just applied with new technology. And it’s just one configuration example. But in general many configurations are possible, but also you need to refine the personas a great deal to get it to work well.
alfalfasprout 3y ago
> These aren't chained, you can run parallel prompts of 15 personas with X diversity of perspective, that reason on a singular request, string, input or variable, they provide output plus audit explanation. You then run an amalgamation or committee decision (sort of like mixture of experts) on it to output variable or string. Then run parallel simulation or reflection prompts based on X different context personas to double check their application to outside cases, unconsidered context, etc. Sure, so you ensemble some results. You're back to the classical "hyperparameter" problem though that's faced ML for a long time-- what those personas are, what those subsequent prompts are, etc. require a fair amount of manual verification and tuning. And the search space is extremely vast. Not to mention that something like this is likely to be very unperformant.
- tsunamifury 3y ago
  We’ve found way that works for us. Nothing magical, but it works in a generalized sense. And doesn’t have this need for significant manual tuning. Search space is vast, but since they can write their own prompts or select strategies, you can have then tree nav a library of strategies and they do pretty well.
FridgeSeal 3y ago
That’s a lot of compute power you’re expending for funny computer parrot to hallucinate some text for you. Seems not even machines get to escape “decision by committee” lol.
- __loam 3y ago
  Decision by committee is actually a pretty good technique for improving the performance of ml models but there's probably cheaper ways of doing it than hitting OpenAI's api 10 times a request
__loam 3y ago
Okay that actually sounds like ensemble learning which is a known ml technique
- tsunamifury 3y ago
  Been working on ML for 10 years at goog. The difference is the context, subtlety and complexity of the problems you can feed this vs older models. Delphi, ensemble, whatever before lacked audit ability and explain ability —- LLMs excel at this. It allows you to apply it to much softer problems.

ironborn123 3y ago

Brave are those who have set out to make prompt engg an entire industry, with gpt-5 and gemini lurking on the horizon.

tsunamifury 3y ago
Sam has specifically stated it's unlikely there will be a GPT5, and likely GPT4 is just a deeply prompt optimized and multimode version of GPT3. But overall, hasn't that theme been true for like... all tech ever? You have to set up and build your own innovation path at some point.
- matmulbro 3y ago
  [flagged]

dlkf 3y ago

> the applications of these sort of reasoning atomic configurations I think are unlimited They are limited to applications in which the latency slo is O(seconds), knowledge of 2021-present doesn’t matter, and you’re allowed to make things up when you don’t know the answer. There are, to be fair, many such applications. But it’s not unlimited.

tsunamifury 3y ago
Bounded =/= not unlimited

carlitossway 3y ago

Interesting. Were you inspired by the Delphi decision algorithm, or did you happen to rediscover it? I agree it could provide a moat, though not a very wide one. All it takes is for your competition to talk to non-engineers, and all sorts of clever usages become possible. See https://en.wikipedia.org/wiki/Delphi_method https://en.wikipedia.org/wiki/Delphi_method

tsunamifury 3y ago
Accidentally discovered its effectiveness when configured with LLMs, just applied with new technology. And it’s just one configuration example. But in general many configurations are possible, but also you need to refine the personas a great deal to get it to work well.

alfalfasprout 3y ago

> These aren't chained, you can run parallel prompts of 15 personas with X diversity of perspective, that reason on a singular request, string, input or variable, they provide output plus audit explanation. You then run an amalgamation or committee decision (sort of like mixture of experts) on it to output variable or string. Then run parallel simulation or reflection prompts based on X different context personas to double check their application to outside cases, unconsidered context, etc. Sure, so you ensemble some results. You're back to the classical "hyperparameter" problem though that's faced ML for a long time-- what those personas are, what those subsequent prompts are, etc. require a fair amount of manual verification and tuning. And the search space is extremely vast. Not to mention that something like this is likely to be very unperformant.

tsunamifury 3y ago
We’ve found way that works for us. Nothing magical, but it works in a generalized sense. And doesn’t have this need for significant manual tuning. Search space is vast, but since they can write their own prompts or select strategies, you can have then tree nav a library of strategies and they do pretty well.

FridgeSeal 3y ago

That’s a lot of compute power you’re expending for funny computer parrot to hallucinate some text for you. Seems not even machines get to escape “decision by committee” lol.

__loam 3y ago
Decision by committee is actually a pretty good technique for improving the performance of ml models but there's probably cheaper ways of doing it than hitting OpenAI's api 10 times a request

__loam 3y ago

Okay that actually sounds like ensemble learning which is a known ml technique

tsunamifury 3y ago
Been working on ML for 10 years at goog. The difference is the context, subtlety and complexity of the problems you can feed this vs older models. Delphi, ensemble, whatever before lacked audit ability and explain ability —- LLMs excel at this. It allows you to apply it to much softer problems.