11 ms·
Hard stuff when building products with LLMs
- fnordpiglet 3y agoTL;DR new technologies are full of sharp edges and no blueprint for success. Engineering is still hard.
- hobs 3y agoEngineering will always be hard, but I think a lot of this current AI hype cycle doesn't even have a product - its just "well that's cool so I want that."
- dinvlad 3y agoWhen “it could be used for anything” really means only that they haven’t found a market fit and are just a solution in search of a problem, as with most venture-baked enterprises.
- quickthrower2 3y agoA search solution in search of a search problem?
- dinvlad 3y agoSearch was already solved - I could find anything I wanted using any number of search engines within seconds, and it provided much more exact information. LLMs are purely suited for search precisely because they can't guarantee to "find" information that actually exists in the real world, but they do add a lot of unnecessary noise, making it harder to weed out the truth, not easier.
- fnordpiglet 3y agoI know at every company I’m working with they’ve found pretty useful and interesting fits for this tech. It fills a space that’s been impossible to date - an abductive reasoning ability in an abstract semantic space. While they can’t actually reason - any more than our inductive or deductive reasoning systems of yore - the missing piece in a lot of stuff has been the ability to navigate an abstract space and find a likely “meaning” then produce a likely output that’s semantically “accurate.” The optimizing, constraining, informing via goal based agency, information retrieval, etc - those are simply integrations, as this article discusses. By looking at LLMs as a finished product you missed the magic. It’s not a product, it’s a capability in a larger system being displayed in demo ware. The larger systems are where the magic happens. Don’t take my word for it - while we will see more hype than we ever have before, we will also see systems that transcend what was possible by amazing leaps and bounds. The jaded are both right, and profoundly wrong - as are the wild eyed dreamers.
- dinvlad 3y agoOK, could you please give a concrete example of a problem that was solved by this that wasn't possible to do before?
- fnordpiglet 3y agoI don’t think that’s very generous - I think it’s “wow that’s amazing I’d like to find a way to integrate it” which I think is perfectly reasonable given it is amazing (even though I think it’s overestimated in its current form, and underestimated due to its current form)
- hobs 3y agoThat's the same thing - integrating Google Search in your product was hot at one point but brought no value, people would just use Google instead of your site search anyway. Working towards having a chat box on every website is not a useful outcome for users or businesses, just OpenAI.
- fnordpiglet 3y agoYes but that’s seeing it as a finished product rather than a demo of a limited technology. The effort here wasn’t using it as a chat box either. The effort here was using natural language generate non trivial queries that are semantically aware of the underlying trace system without the human learning the complex query language. I’m surprised if that’s hard see the value of and how it’s different than a chat box.
- laratied 3y agoIt is also the mindset. I imagine the first time going on the internet and writing it off because you can't see what "products" can be built.
- devjab 3y agoThis seems more like a list of everything everybody is talking about while skipping the “how to make a profit” hard stuff.
- JimtheCoder 3y ago"while skipping the “how to make a profit” hard stuff." Some things never change...
- beachy 3y agoThe article has done its job if you come away with "honeycomb query" ringing in your ears.
- phillipcarter 3y agoI elaborate on this a bit here: https://news.ycombinator.com/item?id=36098161 https://news.ycombinator.com/item?id=36098161
- softfalcon 3y agoWe had a hack-a-thon at my company around using AI tooling with respect to our products. The topics mentioned in this article are real and come up quickly when trying to make a real product that interfaces with an AI-API. This was so true that there was an obvious chunk of teams in the hack-a-thon who didn’t even bother doing anything more than a fancy version of asking ChatGPT “where should I go for dinner in Brooklyn?” or straight up failed to even deliver a concept of a product. Asking a clear question and harvesting accurate results from AI prompts is far more difficult than you might think it would be.
- commandlinefan 3y agoIt’s hard to build a product using humans - I don’t know why anybody would think using AI would make that part any easier.
- muglug 3y agoI predict there will be another six months of these sorts of articles, accompanied by a raft of LLM-powered features that aren't nearly as transformative as the people currently hyping AI are telling us to expect. The engineers I know whose job it is to implement LLM features are much more skeptical about the near future than the engineers who are interested in the topic but lack hands-on experience.
- alaskamiller 3y agoThe last chatbot wave (when FB opened up Messenger API and when Microsoft slung SKYPE as a plausible bot platform and when Slack rebranded their app store) fizzled out after 18 months. All to figure out the singular most important thing: chat interfaces are the worst.
- muspimerol 3y agoI agree that chat interfaces are not great, but we shouldn't reduce LLMs to implementations in chat interfaces. For example, the "autocompletion" I get with copilot is a very useful tool that I use daily, and I think that sort of UX could be built into plenty of other interfaces. Most applications where you input some form of text could benefit from LLM AI.
- phillipcarter 3y agoYes, this exactly. That's why we didn't go with chat for our UX here, and for future product areas we likely won't either. We already have good UX for our kind of product and haven't seen much feedback or convinced via some other means that adding chat would help more than it would hurt.
- JieJie 3y agoOne of the weirdest parts of using Bing Chat is that it has tab-to-autocomplete function that is almost always wrong about what I want to say. I wish there was an LLM that actually was an "autocorrect on steroids" because that's honestly one of my most-anticipated features of this technology. Having an LLM spell-checker that would autocorrect my spelling as I typed, based on the context of what I was typing? That would be magnificent.
- fswd 3y agoI could add a couple things from my own experiences. Storing prompts in a database seemed like a good idea, but in practice it ended up being a disater. Storing the prompt in a python/typescript file, up front at the top, works well. Using OpenAI playground with it's ability to export a prompt works well, or even better, something in gradio running in vscode with debugging mode, works even better. Few shots with refinements works really well. LangChain did not work well for any of my cases, I might go as boldly as saying that using langchain is bad practice.
- ukuina 3y agoSo much "Yes!" for LangChain being bad practice. An unnecessarily bloated abstraction over what should be simple API calls.
- yinser 3y agoDo you have a recommendation of how to easily connect a language model to a python repl, apify, bash shell, and composable chaining structures if not langchain? I find those structures invaluable but am curious where else I could build these programs.
- ntonozzi 3y agoIt’s great for prototyping and seeing what is possible but for running in production you’ll likely need to write it yourself, and it will just take a few minutes.
- jmccarthy 3y agoCould be we're in a (short?) interregnum analogous to pre-Rails Ruby: there are lots of nascent frameworks, but the dominant one hasn't been born yet. FWIW - DIY is working well for me.
- kordlessagain 3y agoCheck out https://www.featurebase.com/blog/doctorgpt-harnessing-the-power-of-semantic-knowledge-graphs-for-unstructured-data https://www.featurebase.com/blog/doctorgpt-harnessing-the-po... and the DoctorGPT repo for doing some things with documents and LLMs, without a framework. I use simple templating and vectors to assemble prompts.
- andy99 3y agoI'd call all of these things specific cases of some of the general problems we've faced with using neural networks for years. There's a big gap between demo and product. One one hand OpenAI has built a great product, on the other hand, it's not yet clear if downstream users will be able to do the same. http://marble.onl/posts/into-the-great-wide-open-ai.html http://marble.onl/posts/into-the-great-wide-open-ai.html
- ZephyrBlu 3y agoI would argue a more appropriate title would be something about integrating LLMs into complex products. A lot of the problems are much more easily solved when you're working on a new product from scratch. Also, I think LLMs work much better with structured data when you use them as a selector instead of a generator. Asking an LLM to generate a structured schema is a bad idea. Ask it to pick from a set of pre-defined schemas instead, for example. You're not using LLMs for their schema generating ability, you're using them for their intelligence and creativity. Don't make them do things they're not good at.
- selalipop 3y agoThat's how I built notionsmith.ai I don't go directly to an LLM asking for structured data, or even a final answer, so you can type literally anything into the entry field and get a useful result People are trying to treat them as conversational, but I'd say for most products it'll be rare to ever want more than one response for a given system prompt, and instead you'll want to build a final answer procedurally.
- HalcyonCowboy 3y agoJust wanted to say that I checked out your app, and it’s really impressive! When building it, did you bootstrap by asking it what developers like me would want out of a site like that?
- selalipop 3y agoIt actually came of my own use of the default ChatGPT interface: I was working on an indie game in my spare time and using it to spitball new mechanics with personas But it was really tedious to prompt ChatGPT into being properly critical about an idea that doesn't exist: A basic "make me a persona" prompt will give you an answer, but if you can really break down the generation of the persona (ie. instead of asking for the whole thing, ask who are the people likely to use X, what's the range of incomes they have, etc) you get a much better answer The site just automates that process and presents chats that are seeded with the result of that process so the LLM is more willing to imagine things. For example, if a persona complains about a feature, when can hit 'Chat with X' and interrogate them about it instead of running into 'As a LLM' you should get an actual answer
- hyperliner 3y ago[dead]
- joelm 3y agoLatency has been the biggest challenge for me. They cite "two to 15+ seconds" in this blog post for responses. Via the OpenAI API I've been seeing more like 45-60 seconds for responses (using GPT-3.5-turbo or GPT-4 in chat mode). Note, this is using ~3500 tokens total. I've had to extensively adapt to that latency in the UI of our product. Maybe I should start showing funny messages while the user is waiting (like I've seen porkbun do when you pay for domain names).
- phillipcarter 3y agoWas this in the past week? We had much worse latency this past week compared to the rest (in addition to model unavailability errors), which we attributed to the Microsoft Build conference. One of our customers that uses it a lot is always at the token limit and their average latency was ~5 seconds, but that was closer to 10 second last week. ...also why we can't wait for other vendors to get SOC I/II clearance, and I guess eventually fine-tuning our own model, so we're not stuck with situations like this.
- joelm 3y agoI've seen more errors lately I think, but no the latency has been an issue for months. I think it has grown some over the last few months, but not a dramatic change.
- phillipcarter 3y agoWell poop, hope that gets resolved fast. I guess OpenAI can't hire compute platform engineers fast enough!
- kristjansson 3y agoIf a user is waiting on the response, you basically have to stream the result instead of waiting on the entire completion.
- joelm 3y ago
- typpo 3y agoThis is a great summary of why productionizing LLMs is hard. I'm working on a couple LLM products, including one that's in production for >10 million users. The lack of formal tooling for prompt engineering drives me bonkers, and it compounds the problems outlined in the article around correctness and chaining. Then there are the hot takes on Twitter from people claiming prompt engineering will soon be obsolete, or people selling blind prompts without any quality metrics. It's surprisingly hard to get LLMs to do _exactly_ what you want. I'm building an open-source framework for systematically measuring prompt quality [0], inspired by best practices for traditional engineering systems. 0. https://github.com/typpo/promptfoo https://github.com/typpo/promptfoo
- darkteflon 3y agoThis looks excellent, thank you - really nails the UI. Going to use this this week.
- jmccarthy 3y agoVery nice, thank you! Will give it a try.
- anotherpaulg 3y agoThis looks really useful. Any thoughts on managing costs? I've been developing against gpt-4, and it runs up charges quickly. I've been thinking I will need to be careful about adding live api calls in any sort of testing situations. Wondering if your tool has any features to help avoid/minimize wasted api usage?
- typpo 3y agoThe tool maintains an LRU cache on disk by default - which means repeat identical requests will be fetched from cache instead of the live API.
- chaxor 3y agoIf you have a task that requires something suggested by "__exact__", then a full LLM is probably not the answer anyway. Try distilling step by step, especially if the goal to to generate a DSL or some restricted language. It can be helpful to have a different set of tokens available to the model for decoding, such that the only possible outcome is something like 'ATTCGGTCCCGGG' given some question to predict a DNA sequence.
- binarymax 3y agoThe first problem, context window size, is going to bite a lot of people. The gotcha is that it’s a search problem. The article mentions embeddings and dot product, but that’s the most basic and naive search you can do. Search is information retrieval, and it’s a huge problem space. You need a proper retriever that you tune for relevance. You should use a search engine for this, and have multiple features and do reranking. That’s the only way to crack the context window problem. But the good news, is that once you do, things get much much better! You can then apply your search/retriever skills on all kinds of other problems - because search is really the backbone of all AI.
- kristjansson 3y agoExactly. LLMs are incredible at information processing, and ok-to-terrible at information retrieval. All LLM applications that rely on accurate information are either infeasible or kick the entire can to the retrieval component.
- phillipcarter 3y agoYeah, we're definitely learning this. It's actually promising how well a very simple cosine similarity pass on data before sending it to an LLM can do [0]. But as we're learning, each further step towards accuracy is bigger and bigger, and there doesn't appear to be any turnkey solutions you can pay for right now. [0]: https://twitter.com/_cartermp/status/1657037648400117760 https://twitter.com/_cartermp/status/1657037648400117760
- darkteflon 3y agoSuch a great comment. The nice thing about this is that if search was a key part of your product pre-LLM, you likely already have something useful in place that requires very little adaptation.
- azinman2 3y agoContext window size will eventually be solved, likely with its own trade offs.
- 3y ago
- simonw 3y agoI think this may be the best thing I've read about real-world prompt engineering - there's SO MUCH hard earned knowledge in here. The description of how they're handling the threat of prompt injection was particularly smart.
- bluepoint 3y agoCan someone kindly explain, what does he mean by the use of embeddings? More specifically how are those embeddings calculated?
- potatoman22 3y agoThey give the LLM access to a search engine powered by embeddings so that it can pull in relevant info. These embeddings could be created numerous ways: TF-IDF, Word2Vec, BERT, GPT.
- phillipcarter 3y agoOpenAI has a pretty good article on embeddings: https://platform.openai.com/docs/guides/embeddings https://platform.openai.com/docs/guides/embeddings What we're looking at doing is: - Creating, storing, and updating an embedding of a schema that people query against - Creating an embedding of the user's input - Running a cosine similarity against the user input embedding and each column in a schema, then sorting by relevancy (it's a score from 1.0 to -1.0) - Using the top n most "relevant" columns instead of passing the full schema So far, there's some pros and cons. On the pros side, it's really fast and lets us generally be more accurate for schemas that are very large, since those can get truncated today. We've seen in some cases it can also help reduce LLM hallucinations. On the cons side, it's another layer of probabilistic behavior and still has the chance of "missing" a relevant column. We can't really say for sure if it's better overall in our test environment, so we're going to just test in production and flag it out if it's yielding worse results.
- lee101 3y ago[dead]
- alecco 3y agohttps://web.archive.org/web/20230528065531/https://www.honeycomb.io/blog/hard-stuff-nobody-talks-about-llm https://web.archive.org/web/20230528065531/https://www.honey...
- fogx 3y ago> especially if those inputs are extremely vague (how on earth should we interpret slow?!) isn't this exactly the (theoretical) strength of a chatbot - asking follow-up questions to remove uncertainty?