9 ms·

Building Boba AI: Lessons learnt in building an LLM-powered application

171 points by nalgeon 3y ago

akiselev 3y ago
> Along the way, we’ve learned some useful lessons on how to build these kinds of applications, which we’ve formulated in terms of patterns. * Use a text template to enrich a prompt with context and structure * Tell the LLM to respond in a structured data format * Stream the response to the UI so users can monitor progress * Capture and add relevant context information to subsequent action * Allow direct conversation with the LLM within a context. * Tell LLM to generate intermediate results while answering * Provide affordances for the user to have a back-and-forth interaction with the co-pilot * Combine LLM with other information sources to access data beyond the LLM's training set
- frankgrecojr 3y ago
  > Stream the response to the UI so users can monitor progress This is a game changer to the UX
  - senko 3y ago
    It's a crutch to minimize the user annoyance at having to wait up to a minute for the response. It sure beats the spinner but it's still a crutch.
    - thelittleone 3y ago
      Really? I feel like in most uses the stream is close to, if not slightly faster than my reading. I actually prefer that over an instant full-page response. It helps me keep my place in the text and feels like reduced cognitive load.
      - golergka 3y ago
        Also, the vibrations when LLM “types” in OpenAI iOS app are so satisfying for some reason. It may be a stupid subjective thing, but from my experience in game dev, that's exactly the kind of detail that creates the overall user experience feeling.
  - behnamoh 3y ago
    Actually, it's annoying because as you start reading the first lines, the content keeps scrolling (often with jagged movements). I always have to scroll up immediately after the stream begins to disable this behavior.
    - tobr 3y ago
      That’s totally fixable, though. ReadyRunner handles it simply by scrolling all the way from the start, leaving space for the message to grow.
      - trafnar 3y ago
        Hey, that's my app! https://www.readyrunner.ai https://www.readyrunner.ai
        darkteflon 3y ago
        That looks really nice. I do wish you would consider one-time pricing for those bringing their own API key on the “Dev” plan, though :) I’d pay ~$20-30 for a nice desktop app like this but won’t enter into another $48/year sub.
  - huydotnet 3y ago
    Still not a reasonable way if you're expecting a structured data in the response, like JSON or something that you're required to parse before showing to the user.
    - jelled 3y ago
      On one of my apps I ask for a plain text response first and then make a second call to parse the original response to json.
  - jamifsud 3y ago
    Anyone know of any good “tolerant” JSON parsers? I’d love to be able to stream a JSON response down to the client and have it be able to parse the JSON as it goes and handle the formatting errors that we sometimes see.
    - icyfox 3y ago
      There's no bulletproof solution to this. JSON5 (https://www.npmjs.com/package/json5 https://www.npmjs.com/package/json5) gets you slightly more leniency, as does plugging the currently streamed content into another smaller LLM. I also wrote a deterministic parser more tailored towards these partially-complete LM outputs. Not perfect certainly but handles the 99% of cases well: https://github.com/piercefreeman/gpt-json https://github.com/piercefreeman/gpt-json. In particular the "streaming" functionality here might be of interest to you.
      - darkteflon 3y ago
        This looks really cool, thanks for open-sourcing this. I’ve been similarly parsing and validating output from OpenAI’s new functions using a schema defined on a custom Pydantic class, but I can see that your code has a lot of niceties coming from proper battle-testing, including elegant error handling, transformations and good docs. I’d like to incorporate this in a production workflow for generating schema-compliant test data for use in few-shot promoting - would you mind saying a few words about your medium term plans for the library? The LangChain API is changing all the time at the moment so we’re trying to figure out where it’s safe to stand. No expectations, of course, just curious.
        icyfox 3y ago
        Sure - I'm using it in a few different internal tools and know others are using it in production. The API should be relatively stable at this point since I intentionally kept the scope pretty limited. The main changes over time will be improved robustness and error correction as issues report different JSON schema breaks that we can fix automatically. Let me know if you see more cases that can be addressed here, would love to collaborate on it.
- manojlds 3y ago
  The short courses from dl.ai are better at driving these points - https://www.deeplearning.ai/short-courses/ https://www.deeplearning.ai/short-courses/
selalipop 3y ago
I worked on something very much in this vein (notionsmith.ai) and feel like I should do a write up after reading this! I think a lot of people are learning these lessons in isolation, I do wish there was a centralized place where people working on UX-focused LLM based apps were exchanging lessons
- tinco 3y ago
  I think a lot of us are working heads down in isolation because we don't have a shareworthy project yet. In a week or two I think my system will be fancy enough to write a blog post about and maybe make open source. HN has been a pretty good source of exchanging knowledge so far, every couple days or so there's a write up like this that has some new tidbits or confirmations of ideas. If everyone keeps doing that we're doing great in my opinion. Looking forward to seeing your write up on here!
- ignoramous 3y ago
  Things on the LLM front for utility apps are fairly nascent and by OpenAI's own admission, the current limitations are fleeting, as in, as a developer, you will soon not need the workarounds used today. Multi-modal models are going to change things even further.
sgt101 3y ago
I find the whole idea of adding text into text to drive a outcome pretty worrying if I have to rely on the output. If the probability of the model spitting out something bad is 0.01% will my testing find it? Probably not.. but my users certainly will.
- phillipcarter 3y ago
  Well, it's a tool for ideation, not a strategy emitter. You don't rely on the output, you rely on the people who finalize and commit to a strategy.
  - sgt101 3y ago
    Yeah - for an application like this I get it. But no one is getting rich or shifting the dial on scientific progress with this sort of thing.
  - mark_l_watson 3y ago
    Ideation is a prime application. At least once a week, I work out an idea for something with a long “conversation” chain staring with “I want to design a system that…” and asking for more detail, suggesting resources, etc. that said, being able to do this is not life changing for me. It just saves some time.
m3kw9 3y ago
LLM latency is a huge no go for most apps except for chat apps. I’ve try to build apps based on OpenAI and that itself creates a bad experience no matter how much elevator music/mirrors/spinners you place. Then you need proper error correction when dealing with structured responses/occasional hallucinations
- PUSH_AX 3y ago
  Latency is acceptable when the value of the result outweighs the time you believe to be excess. If AI can build me a useful marketing plan in 15 mins with multiple agents doing work this seems fine to me. It’s going to take much longer to get a human involved.
  - m3kw9 3y ago
    You are assuming it gets you the result first time, it rarely does and you always need many back and forths. It’s still ok in your case if the results can kick start something, but you’d be nuts to assume you can use that all the way
    - PUSH_AX 3y ago
      Ultimately the same concept applies in my opinion, ROI for both effort and time to be clear. Even today it's in a good place for a lot of things, and will only get better.
daviding 3y ago
This is an interesting article, and a bit of a mish mash of UI conventions, application use ideas for GPT and actual patterns for LLMs. I really do miss Martin Fowler's actual take on these things, but using his name as some sort of gestalt brain for Thoughtworks works too. It still feels like a bit of a Wild West for patterns in this area as yet, with a lot of people trying lots of things and it might be too soon for defining terms. A useful resource is still things like the OpenAI Cookbook, that is a decent collection of a lot of the things in this article but with a more implementation bent.[1] The area that seems to get a lot of idea duplication currently is in providing either a 'session' or a longer term context for GPT, be it with embeddings or rolling prompts for these apps. The use of vector search and embedded chunks is something that seems to be missing so far from vendors like OpenAI, and you can't help but wonder that they'll move it behind their API eventually with a 'session id' in the end. I think that was mentioned as on their roadmap for this year too. The lack of GPT-4 fine tuning options just seems to push people more to look at the Pinecone, Weaviates etc stores and chaining up their own sequences to achieve some sort of memory. I've implemented features with GPT-4 and functions and so far it's feeling useful for 'data model' like use (where you're bringing json into the prompt about a domain noun, e.g. 'Tasks') but is pretty hairy when it comes to pure functions - the tuning they've done to get it to pick which function and which parameters to use is still hard going to get right, which means there doesn't feel like a lot of trust that it is going to be usable. It's like there needs to be a set of patterns or categories for 'business apps' that are heavily siloed into just a subset of available functions it can work with, making it more task-specific rather than as a general chat agent we see a lot of. The difference in approach between LangChain's Chain of Thought pattern and just using OpenAI functions is sort of up in the air as well. Like I said, it still all feels like we're in wild west times, at least as an app developer. [1] https://github.com/openai/openai-cookbook https://github.com/openai/openai-cookbook
- ignoramous 3y ago
  > A useful resource is still things like the OpenAI Cookbook, that is a decent collection of a lot of the things in this article By far, the best resource I've found is the Prompt Engineering Guide: https://www.promptingguide.ai/ https://www.promptingguide.ai/ > you can't help but wonder that they'll move it behind their API eventually with a 'session id' in the end For in-context learning, I think it is fair to expect 100k to 500k context windows sooner. OpenAI is already at 32k.
mvdtnz 3y ago
I am so despondent at the lack of creativity in most of the (many, many) LLM powered projects that are popping up. I have seen hardly a single thing that goes beyond "it's a chat bot, but with a special prompt". Like, is this the best we can expect from this supposedly ground-breaking technology?
- pertymcpert 3y ago
  Same. You just know most of the paid apps are going to be abandoned in a few months.
- bugglebeetle 3y ago
  Most of the stuff it’s actually good at (like NLP tasks) are both super boring and require a secondary layer of processing to catch hallucinations. Not as cool of a sales pitch to everyone on the “it’s alive!” hype train.
  - msp26 3y ago
    Any advice for people building the second layer stuff?
    - bugglebeetle 3y ago
      Just basic sanity checking. You can use an LLM call for this or something procedural. Generally, it’s just of the order of like “does the output conform to this structure”, “is the returned text actually in the input” etc
  - fhd2 3y ago
    I've been working on NLP stuff using LLMs for a while, and it's not that the problems aren't somewhat interesting, but solving them is ridiculously tedious. Most of the time on that project I've just played around with different prompts to make it do what I'm hoping for, with almost no mental process for understanding problems and solutions, mostly just randomly coming up with experiments and reading the results very carefully, checking how consistent they are. I moved towards finding and isolating the parts LLMs are good (and reliable!) at and using deterministic approaches for everything I possibly can. That part is not too tedious, but all this black box trial and error (with all the waiting and errors)... Luckily the client doesn't expect LLMs to be what the hype says and mainly just wants some reasonably useful features so they can say they use AI - wouldn't enjoy dealing with someone thinking it's super easy and I just need to write a few scripts because they tried this in ChatGPT once.
mercurialsolo 3y ago
Retries, answer verification systems, formatting and parsing. Reasoning and validation of reasoning with natural language is a key challenge. Closed feedback loops to improve the quality of reasoning for LLM's requires human-in-the-loop tools. Prompting methods are often a hit and miss as the same prompts can lead to variable quality outputs.
sandGorgon 3y ago
One of the things that strongly resonate with me is the "text templating" part. I faced the same thing when i was building my AI application. Most SDK/frameworks for writing generative AI model this as libraries (or even React UI widgets). I think that is wrong - it is a config management problem. Think Prompts X Chains X LLMs. Your prompts wont work across everything and everything will break on model change. Coding this into ur classes is what everyone does. Instead we pull out the prompts X chains as jsonnet code. Call it trauma & learnings from the K8s/Borg world. We have formats that have evolved as a result of millions of lines of code wrangling clusters/terraform/etc - so we decided to build a SDK over it. that is what we did here - https://github.com/arakoodev/EdgeChains/releases/tag/0.2.0 https://github.com/arakoodev/EdgeChains/releases/tag/0.2.0 EdgeChains is basically Generative AI prompt engineering modeled as config management. Funnily people organically build this out over 6 months of engineering AI applications. That's what Martin did with templating this as text !!

171 points by nalgeon 3y ago

> Along the way, we’ve learned some useful lessons on how to build these kinds of applications, which we’ve formulated in terms of patterns. * Use a text template to enrich a prompt with context and structure * Tell the LLM to respond in a structured data format * Stream the response to the UI so users can monitor progress * Capture and add relevant context information to subsequent action * Allow direct conversation with the LLM within a context. * Tell LLM to generate intermediate results while answering * Provide affordances for the user to have a back-and-forth interaction with the co-pilot * Combine LLM with other information sources to access data beyond the LLM's training set

frankgrecojr 3y ago
> Stream the response to the UI so users can monitor progress This is a game changer to the UX
- senko 3y ago
  It's a crutch to minimize the user annoyance at having to wait up to a minute for the response. It sure beats the spinner but it's still a crutch.
  - thelittleone 3y ago
    Really? I feel like in most uses the stream is close to, if not slightly faster than my reading. I actually prefer that over an instant full-page response. It helps me keep my place in the text and feels like reduced cognitive load.
    - golergka 3y ago
      Also, the vibrations when LLM “types” in OpenAI iOS app are so satisfying for some reason. It may be a stupid subjective thing, but from my experience in game dev, that's exactly the kind of detail that creates the overall user experience feeling.
- behnamoh 3y ago
  Actually, it's annoying because as you start reading the first lines, the content keeps scrolling (often with jagged movements). I always have to scroll up immediately after the stream begins to disable this behavior.
  - tobr 3y ago
    That’s totally fixable, though. ReadyRunner handles it simply by scrolling all the way from the start, leaving space for the message to grow.
    - trafnar 3y ago
      Hey, that's my app! https://www.readyrunner.ai https://www.readyrunner.ai
      - darkteflon 3y ago
        That looks really nice. I do wish you would consider one-time pricing for those bringing their own API key on the “Dev” plan, though :) I’d pay ~$20-30 for a nice desktop app like this but won’t enter into another $48/year sub.
- huydotnet 3y ago
  Still not a reasonable way if you're expecting a structured data in the response, like JSON or something that you're required to parse before showing to the user.
  - jelled 3y ago
    On one of my apps I ask for a plain text response first and then make a second call to parse the original response to json.
- jamifsud 3y ago
  Anyone know of any good “tolerant” JSON parsers? I’d love to be able to stream a JSON response down to the client and have it be able to parse the JSON as it goes and handle the formatting errors that we sometimes see.
  - icyfox 3y ago
    There's no bulletproof solution to this. JSON5 (https://www.npmjs.com/package/json5 https://www.npmjs.com/package/json5) gets you slightly more leniency, as does plugging the currently streamed content into another smaller LLM. I also wrote a deterministic parser more tailored towards these partially-complete LM outputs. Not perfect certainly but handles the 99% of cases well: https://github.com/piercefreeman/gpt-json https://github.com/piercefreeman/gpt-json. In particular the "streaming" functionality here might be of interest to you.
    - darkteflon 3y ago
      This looks really cool, thanks for open-sourcing this. I’ve been similarly parsing and validating output from OpenAI’s new functions using a schema defined on a custom Pydantic class, but I can see that your code has a lot of niceties coming from proper battle-testing, including elegant error handling, transformations and good docs. I’d like to incorporate this in a production workflow for generating schema-compliant test data for use in few-shot promoting - would you mind saying a few words about your medium term plans for the library? The LangChain API is changing all the time at the moment so we’re trying to figure out where it’s safe to stand. No expectations, of course, just curious.
      - icyfox 3y ago
        Sure - I'm using it in a few different internal tools and know others are using it in production. The API should be relatively stable at this point since I intentionally kept the scope pretty limited. The main changes over time will be improved robustness and error correction as issues report different JSON schema breaks that we can fix automatically. Let me know if you see more cases that can be addressed here, would love to collaborate on it.
manojlds 3y ago
The short courses from dl.ai are better at driving these points - https://www.deeplearning.ai/short-courses/ https://www.deeplearning.ai/short-courses/

selalipop 3y ago

I worked on something very much in this vein (notionsmith.ai) and feel like I should do a write up after reading this! I think a lot of people are learning these lessons in isolation, I do wish there was a centralized place where people working on UX-focused LLM based apps were exchanging lessons

tinco 3y ago
I think a lot of us are working heads down in isolation because we don't have a shareworthy project yet. In a week or two I think my system will be fancy enough to write a blog post about and maybe make open source. HN has been a pretty good source of exchanging knowledge so far, every couple days or so there's a write up like this that has some new tidbits or confirmations of ideas. If everyone keeps doing that we're doing great in my opinion. Looking forward to seeing your write up on here!
ignoramous 3y ago
Things on the LLM front for utility apps are fairly nascent and by OpenAI's own admission, the current limitations are fleeting, as in, as a developer, you will soon not need the workarounds used today. Multi-modal models are going to change things even further.

sgt101 3y ago

I find the whole idea of adding text into text to drive a outcome pretty worrying if I have to rely on the output. If the probability of the model spitting out something bad is 0.01% will my testing find it? Probably not.. but my users certainly will.

phillipcarter 3y ago
Well, it's a tool for ideation, not a strategy emitter. You don't rely on the output, you rely on the people who finalize and commit to a strategy.
- sgt101 3y ago
  Yeah - for an application like this I get it. But no one is getting rich or shifting the dial on scientific progress with this sort of thing.
- mark_l_watson 3y ago
  Ideation is a prime application. At least once a week, I work out an idea for something with a long “conversation” chain staring with “I want to design a system that…” and asking for more detail, suggesting resources, etc. that said, being able to do this is not life changing for me. It just saves some time.

m3kw9 3y ago

LLM latency is a huge no go for most apps except for chat apps. I’ve try to build apps based on OpenAI and that itself creates a bad experience no matter how much elevator music/mirrors/spinners you place. Then you need proper error correction when dealing with structured responses/occasional hallucinations

PUSH_AX 3y ago
Latency is acceptable when the value of the result outweighs the time you believe to be excess. If AI can build me a useful marketing plan in 15 mins with multiple agents doing work this seems fine to me. It’s going to take much longer to get a human involved.
- m3kw9 3y ago
  You are assuming it gets you the result first time, it rarely does and you always need many back and forths. It’s still ok in your case if the results can kick start something, but you’d be nuts to assume you can use that all the way
  - PUSH_AX 3y ago
    Ultimately the same concept applies in my opinion, ROI for both effort and time to be clear. Even today it's in a good place for a lot of things, and will only get better.

daviding 3y ago

This is an interesting article, and a bit of a mish mash of UI conventions, application use ideas for GPT and actual patterns for LLMs. I really do miss Martin Fowler's actual take on these things, but using his name as some sort of gestalt brain for Thoughtworks works too. It still feels like a bit of a Wild West for patterns in this area as yet, with a lot of people trying lots of things and it might be too soon for defining terms. A useful resource is still things like the OpenAI Cookbook, that is a decent collection of a lot of the things in this article but with a more implementation bent.[1] The area that seems to get a lot of idea duplication currently is in providing either a 'session' or a longer term context for GPT, be it with embeddings or rolling prompts for these apps. The use of vector search and embedded chunks is something that seems to be missing so far from vendors like OpenAI, and you can't help but wonder that they'll move it behind their API eventually with a 'session id' in the end. I think that was mentioned as on their roadmap for this year too. The lack of GPT-4 fine tuning options just seems to push people more to look at the Pinecone, Weaviates etc stores and chaining up their own sequences to achieve some sort of memory. I've implemented features with GPT-4 and functions and so far it's feeling useful for 'data model' like use (where you're bringing json into the prompt about a domain noun, e.g. 'Tasks') but is pretty hairy when it comes to pure functions - the tuning they've done to get it to pick which function and which parameters to use is still hard going to get right, which means there doesn't feel like a lot of trust that it is going to be usable. It's like there needs to be a set of patterns or categories for 'business apps' that are heavily siloed into just a subset of available functions it can work with, making it more task-specific rather than as a general chat agent we see a lot of. The difference in approach between LangChain's Chain of Thought pattern and just using OpenAI functions is sort of up in the air as well. Like I said, it still all feels like we're in wild west times, at least as an app developer. [1] https://github.com/openai/openai-cookbook https://github.com/openai/openai-cookbook

ignoramous 3y ago
> A useful resource is still things like the OpenAI Cookbook, that is a decent collection of a lot of the things in this article By far, the best resource I've found is the Prompt Engineering Guide: https://www.promptingguide.ai/ https://www.promptingguide.ai/ > you can't help but wonder that they'll move it behind their API eventually with a 'session id' in the end For in-context learning, I think it is fair to expect 100k to 500k context windows sooner. OpenAI is already at 32k.

mvdtnz 3y ago

I am so despondent at the lack of creativity in most of the (many, many) LLM powered projects that are popping up. I have seen hardly a single thing that goes beyond "it's a chat bot, but with a special prompt". Like, is this the best we can expect from this supposedly ground-breaking technology?

pertymcpert 3y ago
Same. You just know most of the paid apps are going to be abandoned in a few months.
bugglebeetle 3y ago
Most of the stuff it’s actually good at (like NLP tasks) are both super boring and require a secondary layer of processing to catch hallucinations. Not as cool of a sales pitch to everyone on the “it’s alive!” hype train.
- msp26 3y ago
  Any advice for people building the second layer stuff?
  - bugglebeetle 3y ago
    Just basic sanity checking. You can use an LLM call for this or something procedural. Generally, it’s just of the order of like “does the output conform to this structure”, “is the returned text actually in the input” etc
- fhd2 3y ago
  I've been working on NLP stuff using LLMs for a while, and it's not that the problems aren't somewhat interesting, but solving them is ridiculously tedious. Most of the time on that project I've just played around with different prompts to make it do what I'm hoping for, with almost no mental process for understanding problems and solutions, mostly just randomly coming up with experiments and reading the results very carefully, checking how consistent they are. I moved towards finding and isolating the parts LLMs are good (and reliable!) at and using deterministic approaches for everything I possibly can. That part is not too tedious, but all this black box trial and error (with all the waiting and errors)... Luckily the client doesn't expect LLMs to be what the hype says and mainly just wants some reasonably useful features so they can say they use AI - wouldn't enjoy dealing with someone thinking it's super easy and I just need to write a few scripts because they tried this in ChatGPT once.

mercurialsolo 3y ago

Retries, answer verification systems, formatting and parsing. Reasoning and validation of reasoning with natural language is a key challenge. Closed feedback loops to improve the quality of reasoning for LLM's requires human-in-the-loop tools. Prompting methods are often a hit and miss as the same prompts can lead to variable quality outputs.

sandGorgon 3y ago

One of the things that strongly resonate with me is the "text templating" part. I faced the same thing when i was building my AI application. Most SDK/frameworks for writing generative AI model this as libraries (or even React UI widgets). I think that is wrong - it is a config management problem. Think Prompts X Chains X LLMs. Your prompts wont work across everything and everything will break on model change. Coding this into ur classes is what everyone does. Instead we pull out the prompts X chains as jsonnet code. Call it trauma & learnings from the K8s/Borg world. We have formats that have evolved as a result of millions of lines of code wrangling clusters/terraform/etc - so we decided to build a SDK over it. that is what we did here - https://github.com/arakoodev/EdgeChains/releases/tag/0.2.0 https://github.com/arakoodev/EdgeChains/releases/tag/0.2.0 EdgeChains is basically Generative AI prompt engineering modeled as config management. Funnily people organically build this out over 6 months of engineering AI applications. That's what Martin did with templating this as text !!