3 ms·

One thing that I really like about the approach you took with clownfish is that it doesn't constrain or modify the structure of the prompt. One of the primary

One thing that I really like about the approach you took with clownfish is that it doesn't constrain or modify the structure of the prompt. One of the primary difficulties with writing LLM applications is that prompts are basically not composable, and any LLM library that modifies your prompt is going to be a nightmare to work with.

killthebuddha 3y ago
Follow-up thought I just had: It seems that prompt structure standards are going to have to emerge if any of these tools have a shot at interoperability. I don't have hard data, but IME if a prompt is structured MEMORY EXAMPLE INSTRUCTION [COMPLETION] it will basically not work to wrap it in a prompt that's structured INSTRUCTION MEMORY EXAMPLE [COMPLETION]
- ianbutler 3y ago
  Interoperability can also be achieved with small adapters written for the prompting style of the particular model being interfaced with, I'd be surprised if like LangChain or AutoGPT don't already do something like this in their systems. I'm currently building something that leverages an ensemble of different LLMs depending on the difficulty of a task and ran into this issue. Dolly V2 takes "###Instruction: <your stuff> ###Response" as the structure fed to the model where as GPT3.5 Turbo wasn't trained to treat that particular structure as important. The nice thing is that GPT3.5 Turbo will just roll with the prompt structure Dolly uses but that only works in very large LLMs, I'd imagine I wouldn't get away with it in other 12BN parameter models. But realistically this could look like taking the "INSTRUCTION MEMORY EXAMPLE [COMPLETION]" schema represented in a library and each adapter would transform it into "MEMORY EXAMPLE INSTRUCTION [COMPLETION]" schema or whatever is needed by the different model.
  - killthebuddha 3y ago
    I think I agree, and am doing something similar as well. Even the adapters approach does require some amount of consistency across prompts. For example, if you have one prompt that says “you are a very helpful assistant” and one prompt that says “you are a very lazy assistant”, then even if those prompts are otherwise written to be as orthogonal as possible you will still probably see degradation in completion quality.
- newhouseb 3y ago
  I haven't spent time going deep here but my current hypothesis is that interoperability will more or less end up looking like toolformer where the "tools" are just separate LLM runs with task-specific context. So for example: > Instruction: Write a poem and them emit a structure that follows a schema named X. > Completion: [map-schema X "roses are red, violets are blue"] Conceptually this is basically just a function call where context is local to the function.

killthebuddha 3y ago

Follow-up thought I just had: It seems that prompt structure standards are going to have to emerge if any of these tools have a shot at interoperability. I don't have hard data, but IME if a prompt is structured MEMORY EXAMPLE INSTRUCTION [COMPLETION] it will basically not work to wrap it in a prompt that's structured INSTRUCTION MEMORY EXAMPLE [COMPLETION]

ianbutler 3y ago
Interoperability can also be achieved with small adapters written for the prompting style of the particular model being interfaced with, I'd be surprised if like LangChain or AutoGPT don't already do something like this in their systems. I'm currently building something that leverages an ensemble of different LLMs depending on the difficulty of a task and ran into this issue. Dolly V2 takes "###Instruction: <your stuff> ###Response" as the structure fed to the model where as GPT3.5 Turbo wasn't trained to treat that particular structure as important. The nice thing is that GPT3.5 Turbo will just roll with the prompt structure Dolly uses but that only works in very large LLMs, I'd imagine I wouldn't get away with it in other 12BN parameter models. But realistically this could look like taking the "INSTRUCTION MEMORY EXAMPLE [COMPLETION]" schema represented in a library and each adapter would transform it into "MEMORY EXAMPLE INSTRUCTION [COMPLETION]" schema or whatever is needed by the different model.
- killthebuddha 3y ago
  I think I agree, and am doing something similar as well. Even the adapters approach does require some amount of consistency across prompts. For example, if you have one prompt that says “you are a very helpful assistant” and one prompt that says “you are a very lazy assistant”, then even if those prompts are otherwise written to be as orthogonal as possible you will still probably see degradation in completion quality.
newhouseb 3y ago
I haven't spent time going deep here but my current hypothesis is that interoperability will more or less end up looking like toolformer where the "tools" are just separate LLM runs with task-specific context. So for example: > Instruction: Write a poem and them emit a structure that follows a schema named X. > Completion: [map-schema X "roses are red, violets are blue"] Conceptually this is basically just a function call where context is local to the function.