3 ms·

I would argue a more appropriate title would be something about integrating LLMs into complex products. A lot of the problems are much more easily solved when

I would argue a more appropriate title would be something about integrating LLMs into complex products. A lot of the problems are much more easily solved when you're working on a new product from scratch. Also, I think LLMs work much better with structured data when you use them as a selector instead of a generator. Asking an LLM to generate a structured schema is a bad idea. Ask it to pick from a set of pre-defined schemas instead, for example. You're not using LLMs for their schema generating ability, you're using them for their intelligence and creativity. Don't make them do things they're not good at.

selalipop 3y ago
That's how I built notionsmith.ai I don't go directly to an LLM asking for structured data, or even a final answer, so you can type literally anything into the entry field and get a useful result People are trying to treat them as conversational, but I'd say for most products it'll be rare to ever want more than one response for a given system prompt, and instead you'll want to build a final answer procedurally.
- HalcyonCowboy 3y ago
  Just wanted to say that I checked out your app, and it’s really impressive! When building it, did you bootstrap by asking it what developers like me would want out of a site like that?
  - selalipop 3y ago
    It actually came of my own use of the default ChatGPT interface: I was working on an indie game in my spare time and using it to spitball new mechanics with personas But it was really tedious to prompt ChatGPT into being properly critical about an idea that doesn't exist: A basic "make me a persona" prompt will give you an answer, but if you can really break down the generation of the persona (ie. instead of asking for the whole thing, ask who are the people likely to use X, what's the range of incomes they have, etc) you get a much better answer The site just automates that process and presents chats that are seeded with the result of that process so the LLM is more willing to imagine things. For example, if a persona complains about a feature, when can hit 'Chat with X' and interrogate them about it instead of running into 'As a LLM' you should get an actual answer
- jonchurch_ 3y ago
  Also wanted to say this is a really cool tool, ty for mentioning it. I fall into the category of developers using LLMs every single day, for both answering questions while working, and also for more exploratory “bounce ideas off the wall” exercises. Everytime I find a new way to explain to the LLM how I want us to work together, I feel like Ive unlocked new abilities and usecases I didnt expect the model to have. Some examples for those curious: * I am interested in learning more about X because I want to achieve Y. Please give me an overview of concepts that would be useful to learn about X to achieve Y. [then after going back and forth fleshing out what Im interested in learning] Please create a syllabus for me to begin learning about X based on the information you've given me. Provide examples of additional materials I can study which already exist, and some exercises to test and operationalize my knowledge. * [I find that the above can often make the model attempt to squeeze all the info into a single response, which compresses the fidelity of the knowledge and tends towards big shallow lists, so I will employ this trick] I want you to go deeper into each topic you have listed, one at a time. When I say “next” move onto the next topic * You are my personal coach for X, here is context about the problem I want to work on and my goals. This is our first coaching session, ask me any questions you need to gather more information, but never more than 3 at once. Where should we start?
- Ambix 3y ago
  Wow, looks like fresh idea and will be useful for me! Will wait for new releases
phillipcarter 3y ago
Something we're looking to experiment with is asking the LLM to produce pieces of things that we then construct a query from, rather than ask it to also assemble it. The hypothesis is that it's more likely to produce things we can "work with" that are also "interesting" or "useful" to users. FWIW we have about a ~7% failure rate (meaning it fails to produce a valid, runnable query) after some work done to correct what we consider correctable outputs. Not terrible, but we think the above idea could help with that.
- ZephyrBlu 3y ago
  Based on my personal experience I think that's a much better approach, so I wish you luck with it. Maybe somewhat counter-intuitively to how most people view LLMs, I strongly believe they're better when you constrain them a bit with some guardrails (E.g. pieces of a query, a bunch of existing queries, etc). Happily surprised you guys managed to get it down to only a 7% failure rate though! For how temperamental LLMs are and the seeming complexity of the task that's impressive.
  - phillipcarter 3y ago
    > Happily surprised you guys managed to get it down to only a 7% failure rate though! Thanks! It, uhh, was quite a bit higher before we did some of that work though, heh. Since we can take a query and attempt to run it, we get good errors for anything that's ill-specified, and we can track it. Ideally we'd address everything with better prompt engineering, but it's certainly quicker to just fix stuff up after the fact when we know how to.
  - Der_Einzige 3y ago
    Re: constraints, it turns out that banning tokens in a vocabulary is a great way to force models to be creative and follow syntactic or semantic constraints without errors: https://github.com/hellisotherpeople/constrained-text-generation-studio https://github.com/hellisotherpeople/constrained-text-genera...

selalipop 3y ago

That's how I built notionsmith.ai I don't go directly to an LLM asking for structured data, or even a final answer, so you can type literally anything into the entry field and get a useful result People are trying to treat them as conversational, but I'd say for most products it'll be rare to ever want more than one response for a given system prompt, and instead you'll want to build a final answer procedurally.

HalcyonCowboy 3y ago
Just wanted to say that I checked out your app, and it’s really impressive! When building it, did you bootstrap by asking it what developers like me would want out of a site like that?
- selalipop 3y ago
  It actually came of my own use of the default ChatGPT interface: I was working on an indie game in my spare time and using it to spitball new mechanics with personas But it was really tedious to prompt ChatGPT into being properly critical about an idea that doesn't exist: A basic "make me a persona" prompt will give you an answer, but if you can really break down the generation of the persona (ie. instead of asking for the whole thing, ask who are the people likely to use X, what's the range of incomes they have, etc) you get a much better answer The site just automates that process and presents chats that are seeded with the result of that process so the LLM is more willing to imagine things. For example, if a persona complains about a feature, when can hit 'Chat with X' and interrogate them about it instead of running into 'As a LLM' you should get an actual answer
jonchurch_ 3y ago
Also wanted to say this is a really cool tool, ty for mentioning it. I fall into the category of developers using LLMs every single day, for both answering questions while working, and also for more exploratory “bounce ideas off the wall” exercises. Everytime I find a new way to explain to the LLM how I want us to work together, I feel like Ive unlocked new abilities and usecases I didnt expect the model to have. Some examples for those curious: * I am interested in learning more about X because I want to achieve Y. Please give me an overview of concepts that would be useful to learn about X to achieve Y. [then after going back and forth fleshing out what Im interested in learning] Please create a syllabus for me to begin learning about X based on the information you've given me. Provide examples of additional materials I can study which already exist, and some exercises to test and operationalize my knowledge. * [I find that the above can often make the model attempt to squeeze all the info into a single response, which compresses the fidelity of the knowledge and tends towards big shallow lists, so I will employ this trick] I want you to go deeper into each topic you have listed, one at a time. When I say “next” move onto the next topic * You are my personal coach for X, here is context about the problem I want to work on and my goals. This is our first coaching session, ask me any questions you need to gather more information, but never more than 3 at once. Where should we start?
Ambix 3y ago
Wow, looks like fresh idea and will be useful for me! Will wait for new releases

phillipcarter 3y ago

Something we're looking to experiment with is asking the LLM to produce pieces of things that we then construct a query from, rather than ask it to also assemble it. The hypothesis is that it's more likely to produce things we can "work with" that are also "interesting" or "useful" to users. FWIW we have about a ~7% failure rate (meaning it fails to produce a valid, runnable query) after some work done to correct what we consider correctable outputs. Not terrible, but we think the above idea could help with that.

ZephyrBlu 3y ago
Based on my personal experience I think that's a much better approach, so I wish you luck with it. Maybe somewhat counter-intuitively to how most people view LLMs, I strongly believe they're better when you constrain them a bit with some guardrails (E.g. pieces of a query, a bunch of existing queries, etc). Happily surprised you guys managed to get it down to only a 7% failure rate though! For how temperamental LLMs are and the seeming complexity of the task that's impressive.
- phillipcarter 3y ago
  > Happily surprised you guys managed to get it down to only a 7% failure rate though! Thanks! It, uhh, was quite a bit higher before we did some of that work though, heh. Since we can take a query and attempt to run it, we get good errors for anything that's ill-specified, and we can track it. Ideally we'd address everything with better prompt engineering, but it's certainly quicker to just fix stuff up after the fact when we know how to.
- Der_Einzige 3y ago
  Re: constraints, it turns out that banning tokens in a vocabulary is a great way to force models to be creative and follow syntactic or semantic constraints without errors: https://github.com/hellisotherpeople/constrained-text-generation-studio https://github.com/hellisotherpeople/constrained-text-genera...