11 ms·

Lessons after a Half-billion GPT Tokens

512 points by lordofmoria 3y ago

Yacovlewis 3y ago
Interesting piece! My experience around Langchain/RAG differs, so wanted to dig deeper: Putting some logic around handling relevant results helps us produce useful output. Curious what differs on their end.
- mind-blight 3y ago
  I suspect the biggest difference is the input data. Embeddings are great over datasets that look like FAQs and QA docs, or data that conceptually fits into very small chunks (tweets, some product reviews, etc). It does very badly over diverse business docs, especially with naive chunking. B2B use cases usually have old PDFs and word docs that need to be searched, and they're often looking for specific keywords (e.g. a person's name, a product, an id, etc). Vectors terms to do badly in those kinds of searches, and just returning chunks misses a lot of important details
  - gdiamos 3y ago
    rare words are out of vocab errors in vectors Especially if they aren’t in the token vocab
    - mind-blight 3y ago
      Even worse, named entities vary from organization to organization. We have a client who uses a product called "Time". It's software time management. For that customer's documentation, time should be close to "product" and a bunch of other things that have nothing to do with the normal concept of time. I actually suspect that people would get a lot more bang for their buck fine tuning the embedding models on B2B datasets for their use case, rather than fine tuning an llm
      - Yacovlewis 2y ago
        Great example of how an entity like that could throw effective RAG out the window
trolan 3y ago
For a few uni/personal projects I noticed the same about Langchain: it's good at helping you use up tokens. The other use case, quickly switching between models, is a very valid reason still. However, I've recently started playing with OpenRouter which seems to abstract the model nicely.
- sroussey 3y ago
  If someone were to create something new, a blank slate approach, what would you find valuable and why?
  - lordofmoria 3y ago
    This is a great question! I think we now know, collectively, a lot more about what’s annoying/hard about building LLM features than we did when LangChain was being furiously developed. And some things we thought would be important and not-easy, turned out to be very easy: like getting GPT to give back well-formed JSON. So I think there’s lots of room. One thing LangChain is doing now that solves something that IS very hard/annoying is testing. I spent 30 minutes yesterday re-running a slow prompt because 1 in 5 runs would produce weird output. Each tweak to the prompt, I had to run at least 10 times to be reasonably sure it was an improvement.
    - codewithcheese 3y ago
      It can be faster and more effective to fallback to a smaller model (gpt3.5 or haiku), the weakness of the prompt will be more obvious on a smaller model and your iteration time will be faster
      - JeremyHerrman 3y ago
        great insight!
    - sroussey 3y ago
      How would testing work out ideally?
disqard 3y ago
> This worked sometimes (I’d estimate >98% of the time), but failed enough that we had to dig deeper. > While we were investigating, we noticed that another field, name, was consistently returning the full name of the state…the correct state – even though we hadn’t explicitly asked it to do that. > So we switched to a simple string search on the name to find the state, and it’s been working beautifully ever since. So, using ChatGPT helped uncover the correct schema, right?
WarOnPrivacy 3y ago
> We consistently found that not enumerating an exact list or instructions in the prompt produced better results Not sure if he means training here or using his product. I think the latter. My end-user exp of GPT3.5 is that I need to be - not just precise but the exact flavor of precise. It's usually after some trial and error. Then more error. Then more trial. Getting a useful result on the 1st or 3rd try happens maybe 1 in 10 sessions. A bit more common is having 3.5 include what I clearly asked it not to. It often complies eventually.
- xp84 3y ago
  OP uses GPT4 mostly. Another poster here observed that "the opposite is required for 3.5" -- so i think your experience makes sense.
KTibow 3y ago
I feel like for just extracting data into JSON, smaller LLMs could probably do fine, especially with constrained generation and training on extraction.
CuriouslyC 3y ago
If you used better prompts you could use a less expensive model. "return nothing if you find nothing" is the level 0 version of giving the LLM an out. Give it a softer out ("in the event that you do not have sufficient information to make conclusive statements, you may hypothesize as long as you state clearly that you are doing so, and note the evidence and logical basis for your hypothesis") then ask it to evaluate its own response at the end.
- codewithcheese 3y ago
  Yeah also prompts should not be developed in abstract. Goal of a prompt is to activate the models internal respentations for it to best achieve the task. Without automated methods, this requires iteratively testing the models reaction to different input and trying to understand how it's interpreting the request and where it's falling down and then patching up those holes. Need to verify if it even knows what you mean by nothing.
  - jsemrau 3y ago
    In the end, it comes down to a task similar to people management where giving clear and simple instructions is the best.
  - azinman2 3y ago
    Which automated method do you use?
    - CuriouslyC 3y ago
      The only public prompt optimizer that I'm aware of now is DSPy, but it doesn't optimize your main prompt request, just some of the problem solving strategies the LLM is instructed to use, and your few shot learning examples. I wouldn't be surprised if there's a public general prompt optimizing agent by this time next year though.
- deleted 3y ago
  [deleted]
thisgoesnowhere 3y ago
The team I work on processes 5B+ tokens a month (and growing) and I'm the EM overseeing that. Here are my take aways 1. There are way too many premature abstractions. Langchain, as one of may examples, might be useful in the future but at the end of the day prompts are just a API call and it's easier to write standard code that treats LLM calls as a flaky API call rather than as a special thing. 2. Hallucinations are definitely a big problem. Summarizing is pretty rock solid in my testing, but reasoning is really hard. Action models, where you ask the llm to take in a user input and try to get the llm to decide what to do next, is just really hard, specifically it's hard to get the llm to understand the context and get it to say when it's not sure. That said, it's still a gamechanger that I can do it at all. 3. I am a bit more hyped than the author that this is a game changer, but like them, I don't think it's going to be the end of the world. There are some jobs that are going to be heavily impacted and I think we are going to have a rough few years of bots astroturfing platforms. But all in all I think it's more of a force multiplier rather than a breakthrough like the internet. IMHO it's similar to what happened to DevOps in the 2000s, you just don't need a big special team to help you deploy anymore, you hire a few specialists and mostly buy off the shelf solutions. Similarly, certain ML tasks are now easy to implement even for dumb dumb web devs like me.
- tmpz22 3y ago
  > IMHO it's similar to what happened to DevOps in the 2000s, you just don't need a big special team to help you deploy anymore, you hire a few specialists and mostly buy off the shelf solutions. I advocate for these metaphors to help people better understand a reasonable expectation for LLMs in modern development workflows. Mostly because they show it as a trade-off versus a silver bullet. There were trade-offs to the evolution of devops, consider for example the loss of key skillsets like database administration as a direct result of "just use AWS RDS" and the explosion in cloud billing costs (especially the OpEx of startups who weren't even dealing with that much data or regional complexity!) - and how it indirectly led to Gitlabs big outage and many like it.
- gopher_space 3y ago
  > Summarizing is pretty rock solid in my testing, but reasoning is really hard. Asking for analogies has been interesting and surprisingly useful.
eigenvalue 3y ago
I agree with most of it, but definitely not the part about Claude3 being “meh.” Claude3 Opus is an amazing model and is extremely good at coding in Python. The ability to handle massive context has made it mostly replace GPT4 for me day to day. Sounds like everyone eventually concludes that Langchain is bloated and useless and creates way more problems than it solves. I don’t get the hype.
- CuriouslyC 3y ago
  Claude is indeed an amazing model, the fact that Sonnet and Haiku are so good is a game changer - GPT4 is too expensive and GPT3.5 is very mediocre. Getting 95% of GPT4 performance for GPT3.5 prices feels like cheating.
- Oras 3y ago
  +1 for Claude Opus, it had been my go to for the last 3 weeks compared to GPT4. The generated texts are much better than GPT4 when it comes to follow the prompt. I also tried the API for some financial analysis of large tables, the response time was around 2 minutes, still did it really well and timeout errors were around 1 to 2% only.
  - cpursley 3y ago
    How are you sending tabular data in a reliable way. And what is the source document type? I'm trying to solve this for complex financial-related tables in PDFs right now.
    - Oras 3y ago
      Amazon Textract, to get tables, format them with Python as csv then send to your preferred AI model.
      - cpursley 3y ago
        Thanks. How does Textract compare to come of the common cli utilities like pdftotext, tesseract, etc (if you made a comparison)?
mvkel 3y ago
I share a lot of this experience. My fix for "Lesson 4: GPT is really bad at producing the null hypothesis" is to have it return very specific text that I string-match on and treat as null. Like: "if there is no warm up for this workout, use the following text in the description: NOPE" then in code I just do a "if warm up contains NOPE, treat it as null"
- gregorymichael 3y ago
  For cases of “select an option from this set” I have it return an index of the correct option, or eg 999 if it can’t find one. This helped a lot.
  - mvkel 3y ago
    Smart
- gdiamos 3y ago
  We do this for the null hypothesis - is uses an LLM to bootstrap a binary classifier - which handles null easily https://github.com/lamini-ai/llm-classifier https://github.com/lamini-ai/llm-classifier
albert_e 3y ago
> Are we going to achieve Gen AI? > No. Not with this transformers + the data of the internet + $XB infrastructure approach. Errr ...did they really mean Gen AI .. or AGI?
- mdorazio 3y ago
  Gen as in “General” not generative.
_pdp_ 3y ago
The biggest realisation for me while making ChatBotKit has been that UX > Model alone. For me, the current state of AI is not about questions and answers. This is dumb. The presentation matters. This is why we are now investing in generative UI.
- codewithcheese 3y ago
  How are you using Generative UI?
  - _pdp_ 3y ago
    Sorry, not much to show at the moment. It is also pretty new so it is early days. You can find some open-source examples here https://github.com/chatbotkit https://github.com/chatbotkit. More coming next week.
- pbhjpbhj 3y ago
  Generative UI being creation of a specific UI dependent on an obedience from your model? What model is it? Google Gemini were showing something that I'd call 'adapted output UI' in their launch presentation. Is that close to what you're doing in any way?
swalsh 3y ago
The being too precise reduces accuracy example makes sense to me based on my crude understanding on how these things work. If you pass in a whole list of states, you're kind of making the vectors for every state light up. If you just say "state" and the text you passed in has an explicit state, than fewer vectors specific to what you're searching for light up. So when it performs the soft max, the correct state is more likely to be selected. Along the same lines I think his /n vs comma comparison probably comes down to tokenization differences.
legendofbrando 3y ago
The finding on simpler prompts, especially with GPT4 tracks (3.5 requires the opposite). The take on RAG feels application specific. For our use-case where having details of the past rendered up the ability to generate loose connections is actually a feature. Things like this are what I find excites me most about LLMs, having a way to proxy subjective similarities the way we do when we remember things is one of the benefits of the technology that didn’t really exist before that opens up a new kind of product opportunity.
AtNightWeCode 3y ago
The UX is an important part of the trick that cons peeps that these tools are better than they are. If you for instance instruct ChatGpt to only answer yes or no. It will feel like it is wrong much more often.
ilaksh 3y ago
I recently had a bug where I was sometimes sending the literal text "null " right in front of the most important part of my prompt. This caused Claude 3 Sonnet to give the 'ignore' command in cases where it should have used one of the other JSON commands I gave it. I have an ignore command so that it will wait when the user isn't finished speaking. Which it generally judges okay, unless it has 'null' in there. The nice thing is that I have found most of the problems with the LLM response were just indications that I hadn't finished debugging my program because I had something missing or weird in the prompt I gave it.
ein0p 3y ago
Same here: I’m subscribed to all three top dogs in LLM space, and routinely issue the same prompts to all three. It’s very one sided in favor of GPT4 which is stunning since it’s now a year old, although of course it received a couple of updates in that time. Also at least with my usage patterns hallucinations are rare, too. In comparison Claude will quite readily hallucinate plausible looking APIs that don’t exist when writing code, etc. GPT4 is also more stubborn / less agreeable when it knows it’s right. Very little of this is captured in metrics, so you can only see it from personal experience.
- CharlesW 3y ago
  This was with Claude Opus, vs. one of the lesser variants? I really like Opus for English copy generation.
  - ein0p 3y ago
    Opus, yes, the $20/mo version. I usually don’t generate copy. My use cases are code (both “serious” and “the nice to have code I wouldn’t bother writing otherwise”), learning how to do stuff in unfamiliar domains, and just learning unfamiliar things in general. It works well as a very patient teacher, especially if you already have some degree of familiarity with the problem domain. I do have to check it against primary sources, which is how I know the percentage of hallucinations is very low. For code, however I don’t even have to do that, since as a professional software engineer I am the “primary source”.
- Me1000 3y ago
  Interesting, Claude 3 Opus has been better than GPT4 for me. Mostly in that I find it does a better (and more importantly, more thorough) job of explaining things to me. For coding tasks (I'm not asking it to write code, but instead to explain topics/code/etc to me) I've found it tends to give much more nuanced answers. When I give it long text to converse about, I find Claude Opus tends to have a much deeper understanding of the content it's given, where GPT4 tends to just summarize the text at hand, whereas Claude tends to be able to extrapolate better.
msp26 3y ago
> But the problem is even worse – we often ask GPT to give us back a list of JSON objects. Nothing complicated mind you: think, an array list of json tasks, where each task has a name and a label. > GPT really cannot give back more than 10 items. Trying to have it give you back 15 items? Maybe it does it 15% of the time. This is just a prompt issue. I've had it reliably return up to 200 items in correct order. The trick is to not use lists at all but have JSON keys like "item1":{...} in the output. You can use lists as the values here if you have some input with 0-n outputs.
- 7thpower 3y ago
  Can you elaborate? I am currently beating my head against this. If I give GPT4 a list of existing items with a defined structure, and it is just having to convert schema or something like that to JSON, it can do that all day long. But if it has to do any sort of reasoning and basically create its own list, it only gives me a very limited subset. I have similar issues with other LLMs. Very interested in how you are approaching this.
  - thibaut_barrere 3y ago
    Not sure if that fits the bill, but here is an example with 200 sorted items based on a question (example with Elixir & InstructorEx): https://gist.github.com/thbar/a53123cbe7765219c1eca77e03e67577 https://gist.github.com/thbar/a53123cbe7765219c1eca77e03e675...
    - sebastiennight 3y ago
      There are a few improvements I'd suggest with that prompt if you want to maximise its performance. 1. You're really asking for hallucinations here. Asking for factual data is very unreliable, and not what these models are strong at. I'm curious how close/far the results are from ground truth. I would definitely bet that outside of the top 5, numbers would be wobbly and outside of top... 25?, even the ranking would be difficult to trust. Why not just get this from a more trustworthy source?[0] 2. Asking in French might, in my experience, give you results that are not as solid as asking in English. Unless you're asking for a creative task where the model might get confused with EN instructions requiring an FR result, it might be better to ask in EN. And you'll save tokens. 3. Providing the model with a rough example of your output JSON seems to perform better than describing the JSON in plan language. [0]: https://fr.wikipedia.org/wiki/Liste_des_communes_de_France_les_plus_peupl%C3%A9es https://fr.wikipedia.org/wiki/Liste_des_communes_de_France_l...
neals 3y ago
Do I need langchain if I want to analyze a large document of many pages?
- simonw 3y ago
  No. But it might help, because you'll probably have to roll some kind of recursive summarization - I think LangChain has mechanisms a for that which could save you some time.
larodi 3y ago
Agree largely with author, but this ‘wait for OpenAI to do it’ sentiment is not something valid. Opus for example is already much better (not only per my experience, but like… researchers evaluaiton). And even for the fun of it - try some local inference, boy. If u know how to prompt it you definitely would be able to run local for the same tasks. Like listening to my students all going to ‘call some API’ for their projects is really very sad to hear. Many startup fellows share this sentiment which a totally kills all the joy.
- jstummbillig 3y ago
  It sounds like you are a tech educator, which potentially sound like a lot of fun with llms right now. When you are integrating these things into your business, you are looking for different things. Most of our customers would for example not find it very cool to have a service outage because somebody wanted to not kill all the joy.
  - larodi 3y ago
    Sure, when availability and SLA kicks in…, but reselling APIs will only get you that far. Perhaps the whole pro/cons cloud argument can also kick in here, not going into it. We may well be on the same page, or we both perhaps have valid arguments. Your comment is appreciated indeed. But then is the author (and are we) talking experience in reselling APIs or experience in introducing NNs in the pipeline? Not the same thing IMHO. Agreed that OpenAI provides very good service, Gemini is not quite there yet, Groq (the LPUs) delivered a nice tech demo, Mixtral is cool but lacks in certain areas, and Claude can be lengthy. But precisely because I’m not sticking with OAI I can then restate my view that if someone is so good with prompts he can get the same results locally if he knows what he’s doing. Prompting OpenAI the right way can be similarly difficult. Perhaps the whole idea of local inference only matters for IoT scenarios or whenever data is super sensitive (or CTO super stubborn to let it embed and fly). But then if you start from day 1 with WordPress provisioned for you ready to go in Google Cloud, you’d never understand the underlying details of the technology. There sure also must be a good reason why Phind tuned their own thing to offer alongside GPT4 APIs. Disclaimer: tech education is a side thing I do, indeed, and been doing in person for very long time, more than dozen topics, to allow myself to have opinion. Of course business is different matter and strategic decisions arr not the same. Even though I’d not advise anyone to blindly use APIs unless they appreciate the need properly.
dougb5 3y ago
The lessons I wanted from this article weren't in there: Did all of that expenditure actually help their product in a measurable way? Did customers use and appreciate the new features based on LLM summarization compared to whatever they were using before? I presume it's a net win or they wouldn't continue to use it, but more specifics around the application would be helpful.
- lordofmoria 3y ago
  Hey, OP here! The answer is a bit boring: the expenditure definitely has helped customers - in that, they're using AI generated responses in all their work flows all the time in the app, and barely notice it. See what I did there? :) I'm mostly serious though - one weird thing about our app is that you might not even know we're using AI, unless we literally tell you in the app. And I think that's where we're at with AI and LLMs these days, at least for our use case. You might find this other post I just put up to have more details too, related to how/where I see the primary value: https://kenkantzer.com/gpt-is-the-heroku-of-ai/ https://kenkantzer.com/gpt-is-the-heroku-of-ai/
  - kristianp 3y ago
    Can you provide some more detail about the application? I'm not familiar with how llms are used in business, except as customer support bots returning documentation.
haolez 3y ago
That has been my experience too. The null hypothesis explains almost all of my hallucinations. I just don't agree with the Claude assessment. In my experience, Claude 3 Opus is vastly superior to GPT-4. Maybe the author was comparing with Claude 2? (And I've never tested Gemini)
satisfice 3y ago
I keep seeing this pattern in articles like this: 1. A recitation of terrible problems 2. A declaration of general satisfaction. Clearly and obviously, ChatGPT is an unreliable toy. The author seems pleased with it. As an engineer, I find that unacceptable.
- jstummbillig 3y ago
  ChatGPT is probably in the top 5 value/money subscriptions I have ever had (and that includes utilities). The relatively low price point certainly plays a role here, but it's certainly not a mainly recreational thing for me. These thing's are kinda hard to measure but roughly most + is engagement with hard stuff goes up, and rate of learning goes up, by a lot.
- simonw 3y ago
  Working with models like GPT-4 is frustrating from a traditional software engineering perspective because these systems are inherently unreliable and non-deterministic, which differs from most software tools that we use. That doesn't mean they can't be incredibly useful - but it does mean you have to approach them in a bit of a different way, and design software around them that takes their unreliability into account.
  - jbeninger 3y ago
    Unreliable? Non-deterministic? Hidden variables? Undocumented behaviour? C'mon fellow programmers who got their start in the Win-95 era! It's our time to shine!
- Kiro 3y ago
  That has nothing to do with you being an engineer. It's just you. I'm an engineer and LLMs are game changers for me.
- chx 3y ago
  https://hachyderm.io/@inthehands/112006855076082650 https://hachyderm.io/@inthehands/112006855076082650 > You might be surprised to learn that I actually think LLMs have the potential to be not only fun but genuinely useful. “Show me some bullshit that would be typical in this context” can be a genuinely helpful question to have answered, in code and in natural language — for brainstorming, for seeing common conventions in an unfamiliar context, for having something crappy to react to. == End of toot. The price you pay for this bullshit in energy when the sea temperature is literally off the charts and we do not know why makes it not worth it in my opinion.
FranklinMaillot 3y ago
In my limited experience, I came to the same conclusion regarding simple prompt being more efficient than very detailed list of instructions. But if you look at OpenAI's system prompt for GPT4, it's an endless set of instructions with DOs and DONTs so I'm confused. Surely they must know something about prompting their model.
- bongodongobob 3y ago
  That's for chatting and interfacing conversationally with a human. Using the API is a completely different ballgame because it's not meant to be a back and forth conversation with a human.
Civitello 3y ago
> Every use case we have is essentially “Here’s a block of text, extract something from it.” As a rule, if you ask GPT to give you the names of companies mentioned in a block of text, it will not give you a random company (unless there are no companies in the text – there’s that null hypothesis problem!). Make it two steps, first: > Does this block of text mention a company? If no, good you've got your null result. If yes: > Please list the names of companies in this block of text.
deleted 3y ago
[deleted]
sungho_ 3y ago
I'm curious if the OP has tried any of the libraries that control the output of LLM (LMQL, Outliner, Guadiance, ...), and for those who have: do you find them as unnecessary as LangChain? In particular, the OP's post mentions the problem of not being able to generate JSON with more than 15 items, which seems like a problem that can be solved by controlling the output of LLM. Is that correct?
- LASR 3y ago
  If you want x number of items every time, ask it to include a sequence number in each output, it will consistently return x number of items. Numbered bullets work well for this, if you don’t need JSON. With JSON, you can ask it to include an ‘id’ in each item.
orbatos 3y ago
Statements like this tell me your analysis is poisoned by misunderstandings: "Why is this crazy? Well, it’s crazy that GPT’s quality and generalization can improve when you’re more vague – this is a quintessential marker of higher-order delegation / thinking." No, there is no "higher-order thought" happening, or any at all actually. That's not how these models work.
Xenoamorphous 3y ago
> We always extract json. We don’t need JSON mode I wonder why? It seems to work pretty well for me. > Lesson 4: GPT is really bad at producing the null hypothesis Tell me about it! Just yesterday I was testing a prompt around text modification rules that ended with “If none of the rules apply to the text, return the original text without any changes”. Do you know ChatGPT’s response to a text where none of the rules applied? “The original text without any changes”. Yes, the literal string.
- Suppafly 2y ago
  If you look at any of the cake decorating fail sites, humans make that sort of mistake all the time.
- mechagodzilla 3y ago
  AmeliaBedeliaGPT
- phillipcarter 3y ago
  > I wonder why? It seems to work pretty well for me. I read this as "what we do works just fine to not need to use JSON mode". We're in the same boat at my company. Been live for a year now, no need to switch. Our prompt is effective at getting GPT-3.5 to always produce JSON.
  - Kiro 3y ago
    There's nothing to switch to. You just enable it. No need to change the prompt or anything else. All it requires is that you mention "JSON" in your prompt, which you obviously already do.
    - ShamelessC 3y ago
      I think that’s only true when using ChatGPT via the web/app, not when used via API as they likely are. Happy to be corrected however.
      - 3y ago
kromem 3y ago
Tip for your 'null' problem: LLMs are set up to output tokens. Not to not output tokens. So instead of "don't return anything" have the lack of results "return the default value of XYZ" and then just do a text search on the result for that default value (i.e. XYZ) the same way you do the text search for the state names. Also, system prompts can be very useful. It's basically your opportunity to have the LLM roleplay as X. I wish they'd let the system prompt be passed directly, but it's still better than nothing.
Brunocarl9 3y ago
[dead]
nprateem 3y ago
Anyone any good tips for stopping it sounding like it's writing essay answers, and flat out banning "in the realm of", delve, pivotal, multifaceted, etc? I don't want a crap intro or waffley summary but it just can't help itself.
- dudeinhawaii 3y ago
  My approach is to use words that indicate what I want like 'concise', 'brief', etc. If you know a word that precisely describes your desired type of content then use that. It's similar to art generation models, a single word brings so much contextual baggage with it. Finding the right words helps a lot. You can even ask the LLMs for assistance in finding the words to capture your intent. As an example of contextual baggage, I wrote a tool where I had to adjust the prompt between Claude and GPT-4 because using the word "website" in the prompt caused GPT-4 (API) to go into its 'I do not have access to the internet' tirade about 30% of the time. The tool was a summary of web pages experiment. By removing 'website' and replacing it with 'content' (e.g. 'summarize the following content') GPT-4 happily complied 100% of the time.
2099miles 3y ago
Great take, insightful. Highly recommend.
chromanoid 3y ago
GPT is very cool, but I strongly disagree with the interpretation in these two paragraphs: I think in summary, a better approach would’ve been “You obviously know the 50 states, GPT, so just give me the full name of the state this pertains to, or Federal if this pertains to the US government.” Why is this crazy? Well, it’s crazy that GPT’s quality and generalization can improve when you’re more vague – this is a quintessential marker of higher-order delegation / thinking. Natural language is the most probable output for GPT, because the text it was trained with is similar. In this case the developer simply leaned more into what GPT is good at than giving it more work. You can use simple tasks to make GPT fail. Letter replacements, intentional typos and so on are very hard tasks for GPT. This is also true for ID mappings and similar, especially when the ID mapping diverges significantly from other mappings it may have been trained with (e.g. Non-ISO country codes but similar three letter codes etc.). The fascinating thing is, that GPT "understands" mappings at all. Which is the actual hint at higher order pattern matching.
- fl0id 3y ago
  Well, or it is just memorizing mappings. Not like as in reproducing, but having vectors similar to mappings that it saw before.
  - chromanoid 3y ago
    Yeah, but isn't this higher order pattern matching? You can at least correct during a conversation and GPT will then use the correct mappings, probably most of the times (sloppy experiment): https://chat.openai.com/share/7574293a-6d08-4159-a988-4f0816f3aafc https://chat.openai.com/share/7574293a-6d08-4159-a988-4f0816...
konstantinua00 3y ago
> Have you tried Claude, Gemini, etc? > It’s the subtle things mostly, like intuiting intention. this makes me wonder - what if the author "trained" himself onto chatgpt's "dialect"? How do we even detect that in ourselves? and are we about to have "preferred_LLM wars" like we had "programming language wars" for the last 2 decades?
egonschiele 3y ago
I have a personal writing app that uses the OpenAI models and this post is bang on. One of my learnings related to "Lesson 1: When it comes to prompts, less is more": I was trying to build an intelligent search feature for my notes and asking ChatGPT to return structured JSON data. For example, I wanted to ask "give me all my notes that mention Haskell in the last 2 years that are marked as draft", and let Chat GPT figure out what to return. This only worked some of the time. Instead, I put my data in a SQLite database, sent ChatGPT the schema, and asked it to write a query to return what I wanted. That has worked much better.
- ukuina 3y ago
  Have you tried response_format=json_object? I had better luck with function-calling to get a structured response, but it is more limiting than just getting a JSON body.
  - egonschiele 3y ago
    I haven't tried response_format, I'll give that a shot. I've had issues with function calling. Sometimes it works, sometimes it just returns random Python code.
    - ukuina 2y ago
      Using the openai Python library?
- squigz 3y ago
  This seems like something that would be better suited by a database and good search filters rather than an LLM...
  - az226 3y ago
    Something something about everything looking like a nail when you’re holding a hammer
  - chasd00 3y ago
aubanel 3y ago
> I think in summary, a better approach would’ve been “You obviously know the 50 states, GPT, so just give me the full name of the state this pertains to, or Federal if this pertains to the US government.” Why not really compare the two options, author? I would love to see the results!
pamelafox 3y ago
Lol, nice truncation logic! If anyone’s looking for something slightly fancier, I made a micro-package for our tiktoken-based truncation here: https://github.com/pamelafox/llm-messages-token-helper https://github.com/pamelafox/llm-messages-token-helper
pamelafox 3y ago
I’ve also seen that GPTs struggle to admit when they dont know. I wrote up an approach for evaluating that here - http://blog.pamelafox.org/2024/03/evaluating-rag-chat-apps-can-your-app.html?m=1 http://blog.pamelafox.org/2024/03/evaluating-rag-chat-apps-c... Changing the prompt didn't help, but moving to GPT-4 did help a bit.
deleted 3y ago
[deleted]
Kiro 3y ago
> We always extract json. We don’t need JSON mode, Why? The null stuff would not be a problem if you did and if you're only dealing with JSON anyway I don't see why you wouldn't.
littlestymaar 3y ago
> One part of our pipeline reads some block of text and asks GPT to classify it as relating to one of the 50 US states, or the Federal government. Using a multi-billion tokens like GPT-4 for such a trivial classification task[1] is an insane overkill. And in an era where ChatGPT exists, and can in fact give you what you need to build a simpler classifier for the task, it shows how narrow minded most people are when AI is involved. [1] to clarify, it's either trivial or impossible to do reliably depending on how fucked-up your input is
gok 3y ago
So these guys are just dumping confidential tax documents onto OpenAI's servers huh.
- goatlover 3y ago
  Hopefully it won't end up as training data.
amelius 3y ago
This reads a bit like: I have a circus monkey. If I do such and such it will not do anything. But when I do this and that, then it will ride the bicycle. Most of the time.
- saaaaaam 3y ago
  I don’t really understand your comment. Personally I thought this was an interesting read - and more interesting because it didn’t contain any massive “WE DID THIS AND IT CHANGED PUR LIVES!!!” style revelations. It is discursive, thoughtful and not overwritten. I find this kind of content valuable and somewhat rare.
- deleted 3y ago
  [deleted]
peter_d_sherman 3y ago
>"Lesson 2: You don’t need langchain. You probably don’t even need anything else OpenAI has released in their API in the last year. Just chat API. That’s it. Langchain is the perfect example of premature abstraction. We started out thinking we had to use it because the internet said so. Instead, millions of tokens later, and probably 3-4 very diverse LLM features in production, and our openai_service file still has only one, 40-line function in it: def extract_json(prompt, variable_length_input, number_retries) The only API we use is chat. We always extract json. We don’t need JSON mode, or function calling, or assistants (though we do all that). Heck, we don’t even use system prompts (maybe we should…). When a gpt-4-turbo was released, we updated one string in the codebase. This is the beauty of a powerful generalized model – less is more." Well said!

512 points by lordofmoria 3y ago

Interesting piece! My experience around Langchain/RAG differs, so wanted to dig deeper: Putting some logic around handling relevant results helps us produce useful output. Curious what differs on their end.

mind-blight 3y ago
I suspect the biggest difference is the input data. Embeddings are great over datasets that look like FAQs and QA docs, or data that conceptually fits into very small chunks (tweets, some product reviews, etc). It does very badly over diverse business docs, especially with naive chunking. B2B use cases usually have old PDFs and word docs that need to be searched, and they're often looking for specific keywords (e.g. a person's name, a product, an id, etc). Vectors terms to do badly in those kinds of searches, and just returning chunks misses a lot of important details
- gdiamos 3y ago
  rare words are out of vocab errors in vectors Especially if they aren’t in the token vocab
  - mind-blight 3y ago
    Even worse, named entities vary from organization to organization. We have a client who uses a product called "Time". It's software time management. For that customer's documentation, time should be close to "product" and a bunch of other things that have nothing to do with the normal concept of time. I actually suspect that people would get a lot more bang for their buck fine tuning the embedding models on B2B datasets for their use case, rather than fine tuning an llm
    - Yacovlewis 2y ago
      Great example of how an entity like that could throw effective RAG out the window

trolan 3y ago

For a few uni/personal projects I noticed the same about Langchain: it's good at helping you use up tokens. The other use case, quickly switching between models, is a very valid reason still. However, I've recently started playing with OpenRouter which seems to abstract the model nicely.

sroussey 3y ago
If someone were to create something new, a blank slate approach, what would you find valuable and why?
- lordofmoria 3y ago
  This is a great question! I think we now know, collectively, a lot more about what’s annoying/hard about building LLM features than we did when LangChain was being furiously developed. And some things we thought would be important and not-easy, turned out to be very easy: like getting GPT to give back well-formed JSON. So I think there’s lots of room. One thing LangChain is doing now that solves something that IS very hard/annoying is testing. I spent 30 minutes yesterday re-running a slow prompt because 1 in 5 runs would produce weird output. Each tweak to the prompt, I had to run at least 10 times to be reasonably sure it was an improvement.
  - codewithcheese 3y ago
    It can be faster and more effective to fallback to a smaller model (gpt3.5 or haiku), the weakness of the prompt will be more obvious on a smaller model and your iteration time will be faster
    - JeremyHerrman 3y ago
      great insight!
  - sroussey 3y ago
    How would testing work out ideally?

disqard 3y ago

> This worked sometimes (I’d estimate >98% of the time), but failed enough that we had to dig deeper. > While we were investigating, we noticed that another field, name, was consistently returning the full name of the state…the correct state – even though we hadn’t explicitly asked it to do that. > So we switched to a simple string search on the name to find the state, and it’s been working beautifully ever since. So, using ChatGPT helped uncover the correct schema, right?

WarOnPrivacy 3y ago

> We consistently found that not enumerating an exact list or instructions in the prompt produced better results Not sure if he means training here or using his product. I think the latter. My end-user exp of GPT3.5 is that I need to be - not just precise but the exact flavor of precise. It's usually after some trial and error. Then more error. Then more trial. Getting a useful result on the 1st or 3rd try happens maybe 1 in 10 sessions. A bit more common is having 3.5 include what I clearly asked it not to. It often complies eventually.

xp84 3y ago
OP uses GPT4 mostly. Another poster here observed that "the opposite is required for 3.5" -- so i think your experience makes sense.

KTibow 3y ago

I feel like for just extracting data into JSON, smaller LLMs could probably do fine, especially with constrained generation and training on extraction.

CuriouslyC 3y ago

If you used better prompts you could use a less expensive model. "return nothing if you find nothing" is the level 0 version of giving the LLM an out. Give it a softer out ("in the event that you do not have sufficient information to make conclusive statements, you may hypothesize as long as you state clearly that you are doing so, and note the evidence and logical basis for your hypothesis") then ask it to evaluate its own response at the end.

codewithcheese 3y ago
Yeah also prompts should not be developed in abstract. Goal of a prompt is to activate the models internal respentations for it to best achieve the task. Without automated methods, this requires iteratively testing the models reaction to different input and trying to understand how it's interpreting the request and where it's falling down and then patching up those holes. Need to verify if it even knows what you mean by nothing.
- jsemrau 3y ago
  In the end, it comes down to a task similar to people management where giving clear and simple instructions is the best.
- azinman2 3y ago
  Which automated method do you use?
  - CuriouslyC 3y ago
    The only public prompt optimizer that I'm aware of now is DSPy, but it doesn't optimize your main prompt request, just some of the problem solving strategies the LLM is instructed to use, and your few shot learning examples. I wouldn't be surprised if there's a public general prompt optimizing agent by this time next year though.
deleted 3y ago
[deleted]

thisgoesnowhere 3y ago

The team I work on processes 5B+ tokens a month (and growing) and I'm the EM overseeing that. Here are my take aways 1. There are way too many premature abstractions. Langchain, as one of may examples, might be useful in the future but at the end of the day prompts are just a API call and it's easier to write standard code that treats LLM calls as a flaky API call rather than as a special thing. 2. Hallucinations are definitely a big problem. Summarizing is pretty rock solid in my testing, but reasoning is really hard. Action models, where you ask the llm to take in a user input and try to get the llm to decide what to do next, is just really hard, specifically it's hard to get the llm to understand the context and get it to say when it's not sure. That said, it's still a gamechanger that I can do it at all. 3. I am a bit more hyped than the author that this is a game changer, but like them, I don't think it's going to be the end of the world. There are some jobs that are going to be heavily impacted and I think we are going to have a rough few years of bots astroturfing platforms. But all in all I think it's more of a force multiplier rather than a breakthrough like the internet. IMHO it's similar to what happened to DevOps in the 2000s, you just don't need a big special team to help you deploy anymore, you hire a few specialists and mostly buy off the shelf solutions. Similarly, certain ML tasks are now easy to implement even for dumb dumb web devs like me.

tmpz22 3y ago
> IMHO it's similar to what happened to DevOps in the 2000s, you just don't need a big special team to help you deploy anymore, you hire a few specialists and mostly buy off the shelf solutions. I advocate for these metaphors to help people better understand a reasonable expectation for LLMs in modern development workflows. Mostly because they show it as a trade-off versus a silver bullet. There were trade-offs to the evolution of devops, consider for example the loss of key skillsets like database administration as a direct result of "just use AWS RDS" and the explosion in cloud billing costs (especially the OpEx of startups who weren't even dealing with that much data or regional complexity!) - and how it indirectly led to Gitlabs big outage and many like it.
gopher_space 3y ago
> Summarizing is pretty rock solid in my testing, but reasoning is really hard. Asking for analogies has been interesting and surprisingly useful.

eigenvalue 3y ago

I agree with most of it, but definitely not the part about Claude3 being “meh.” Claude3 Opus is an amazing model and is extremely good at coding in Python. The ability to handle massive context has made it mostly replace GPT4 for me day to day. Sounds like everyone eventually concludes that Langchain is bloated and useless and creates way more problems than it solves. I don’t get the hype.

CuriouslyC 3y ago
Claude is indeed an amazing model, the fact that Sonnet and Haiku are so good is a game changer - GPT4 is too expensive and GPT3.5 is very mediocre. Getting 95% of GPT4 performance for GPT3.5 prices feels like cheating.
Oras 3y ago
+1 for Claude Opus, it had been my go to for the last 3 weeks compared to GPT4. The generated texts are much better than GPT4 when it comes to follow the prompt. I also tried the API for some financial analysis of large tables, the response time was around 2 minutes, still did it really well and timeout errors were around 1 to 2% only.
- cpursley 3y ago
  How are you sending tabular data in a reliable way. And what is the source document type? I'm trying to solve this for complex financial-related tables in PDFs right now.
  - Oras 3y ago
    Amazon Textract, to get tables, format them with Python as csv then send to your preferred AI model.
    - cpursley 3y ago
      Thanks. How does Textract compare to come of the common cli utilities like pdftotext, tesseract, etc (if you made a comparison)?

mvkel 3y ago

I share a lot of this experience. My fix for "Lesson 4: GPT is really bad at producing the null hypothesis" is to have it return very specific text that I string-match on and treat as null. Like: "if there is no warm up for this workout, use the following text in the description: NOPE" then in code I just do a "if warm up contains NOPE, treat it as null"

gregorymichael 3y ago
For cases of “select an option from this set” I have it return an index of the correct option, or eg 999 if it can’t find one. This helped a lot.
- mvkel 3y ago
  Smart
gdiamos 3y ago
We do this for the null hypothesis - is uses an LLM to bootstrap a binary classifier - which handles null easily https://github.com/lamini-ai/llm-classifier https://github.com/lamini-ai/llm-classifier

albert_e 3y ago

> Are we going to achieve Gen AI? > No. Not with this transformers + the data of the internet + $XB infrastructure approach. Errr ...did they really mean Gen AI .. or AGI?

mdorazio 3y ago
Gen as in “General” not generative.

_pdp_ 3y ago

The biggest realisation for me while making ChatBotKit has been that UX > Model alone. For me, the current state of AI is not about questions and answers. This is dumb. The presentation matters. This is why we are now investing in generative UI.

codewithcheese 3y ago
How are you using Generative UI?
- _pdp_ 3y ago
  Sorry, not much to show at the moment. It is also pretty new so it is early days. You can find some open-source examples here https://github.com/chatbotkit https://github.com/chatbotkit. More coming next week.
pbhjpbhj 3y ago
Generative UI being creation of a specific UI dependent on an obedience from your model? What model is it? Google Gemini were showing something that I'd call 'adapted output UI' in their launch presentation. Is that close to what you're doing in any way?

swalsh 3y ago

The being too precise reduces accuracy example makes sense to me based on my crude understanding on how these things work. If you pass in a whole list of states, you're kind of making the vectors for every state light up. If you just say "state" and the text you passed in has an explicit state, than fewer vectors specific to what you're searching for light up. So when it performs the soft max, the correct state is more likely to be selected. Along the same lines I think his /n vs comma comparison probably comes down to tokenization differences.

legendofbrando 3y ago

The finding on simpler prompts, especially with GPT4 tracks (3.5 requires the opposite). The take on RAG feels application specific. For our use-case where having details of the past rendered up the ability to generate loose connections is actually a feature. Things like this are what I find excites me most about LLMs, having a way to proxy subjective similarities the way we do when we remember things is one of the benefits of the technology that didn’t really exist before that opens up a new kind of product opportunity.

AtNightWeCode 3y ago

The UX is an important part of the trick that cons peeps that these tools are better than they are. If you for instance instruct ChatGpt to only answer yes or no. It will feel like it is wrong much more often.

ilaksh 3y ago

I recently had a bug where I was sometimes sending the literal text "null " right in front of the most important part of my prompt. This caused Claude 3 Sonnet to give the 'ignore' command in cases where it should have used one of the other JSON commands I gave it. I have an ignore command so that it will wait when the user isn't finished speaking. Which it generally judges okay, unless it has 'null' in there. The nice thing is that I have found most of the problems with the LLM response were just indications that I hadn't finished debugging my program because I had something missing or weird in the prompt I gave it.

ein0p 3y ago

Same here: I’m subscribed to all three top dogs in LLM space, and routinely issue the same prompts to all three. It’s very one sided in favor of GPT4 which is stunning since it’s now a year old, although of course it received a couple of updates in that time. Also at least with my usage patterns hallucinations are rare, too. In comparison Claude will quite readily hallucinate plausible looking APIs that don’t exist when writing code, etc. GPT4 is also more stubborn / less agreeable when it knows it’s right. Very little of this is captured in metrics, so you can only see it from personal experience.

CharlesW 3y ago
This was with Claude Opus, vs. one of the lesser variants? I really like Opus for English copy generation.
- ein0p 3y ago
  Opus, yes, the $20/mo version. I usually don’t generate copy. My use cases are code (both “serious” and “the nice to have code I wouldn’t bother writing otherwise”), learning how to do stuff in unfamiliar domains, and just learning unfamiliar things in general. It works well as a very patient teacher, especially if you already have some degree of familiarity with the problem domain. I do have to check it against primary sources, which is how I know the percentage of hallucinations is very low. For code, however I don’t even have to do that, since as a professional software engineer I am the “primary source”.
Me1000 3y ago
Interesting, Claude 3 Opus has been better than GPT4 for me. Mostly in that I find it does a better (and more importantly, more thorough) job of explaining things to me. For coding tasks (I'm not asking it to write code, but instead to explain topics/code/etc to me) I've found it tends to give much more nuanced answers. When I give it long text to converse about, I find Claude Opus tends to have a much deeper understanding of the content it's given, where GPT4 tends to just summarize the text at hand, whereas Claude tends to be able to extrapolate better.

msp26 3y ago

> But the problem is even worse – we often ask GPT to give us back a list of JSON objects. Nothing complicated mind you: think, an array list of json tasks, where each task has a name and a label. > GPT really cannot give back more than 10 items. Trying to have it give you back 15 items? Maybe it does it 15% of the time. This is just a prompt issue. I've had it reliably return up to 200 items in correct order. The trick is to not use lists at all but have JSON keys like "item1":{...} in the output. You can use lists as the values here if you have some input with 0-n outputs.

7thpower 3y ago
Can you elaborate? I am currently beating my head against this. If I give GPT4 a list of existing items with a defined structure, and it is just having to convert schema or something like that to JSON, it can do that all day long. But if it has to do any sort of reasoning and basically create its own list, it only gives me a very limited subset. I have similar issues with other LLMs. Very interested in how you are approaching this.
- thibaut_barrere 3y ago
  Not sure if that fits the bill, but here is an example with 200 sorted items based on a question (example with Elixir & InstructorEx): https://gist.github.com/thbar/a53123cbe7765219c1eca77e03e67577 https://gist.github.com/thbar/a53123cbe7765219c1eca77e03e675...
  - sebastiennight 3y ago
    There are a few improvements I'd suggest with that prompt if you want to maximise its performance. 1. You're really asking for hallucinations here. Asking for factual data is very unreliable, and not what these models are strong at. I'm curious how close/far the results are from ground truth. I would definitely bet that outside of the top 5, numbers would be wobbly and outside of top... 25?, even the ranking would be difficult to trust. Why not just get this from a more trustworthy source?[0] 2. Asking in French might, in my experience, give you results that are not as solid as asking in English. Unless you're asking for a creative task where the model might get confused with EN instructions requiring an FR result, it might be better to ask in EN. And you'll save tokens. 3. Providing the model with a rough example of your output JSON seems to perform better than describing the JSON in plan language. [0]: https://fr.wikipedia.org/wiki/Liste_des_communes_de_France_les_plus_peupl%C3%A9es https://fr.wikipedia.org/wiki/Liste_des_communes_de_France_l...

neals 3y ago

Do I need langchain if I want to analyze a large document of many pages?

simonw 3y ago
No. But it might help, because you'll probably have to roll some kind of recursive summarization - I think LangChain has mechanisms a for that which could save you some time.

larodi 3y ago

Agree largely with author, but this ‘wait for OpenAI to do it’ sentiment is not something valid. Opus for example is already much better (not only per my experience, but like… researchers evaluaiton). And even for the fun of it - try some local inference, boy. If u know how to prompt it you definitely would be able to run local for the same tasks. Like listening to my students all going to ‘call some API’ for their projects is really very sad to hear. Many startup fellows share this sentiment which a totally kills all the joy.

jstummbillig 3y ago
It sounds like you are a tech educator, which potentially sound like a lot of fun with llms right now. When you are integrating these things into your business, you are looking for different things. Most of our customers would for example not find it very cool to have a service outage because somebody wanted to not kill all the joy.
- larodi 3y ago
  Sure, when availability and SLA kicks in…, but reselling APIs will only get you that far. Perhaps the whole pro/cons cloud argument can also kick in here, not going into it. We may well be on the same page, or we both perhaps have valid arguments. Your comment is appreciated indeed. But then is the author (and are we) talking experience in reselling APIs or experience in introducing NNs in the pipeline? Not the same thing IMHO. Agreed that OpenAI provides very good service, Gemini is not quite there yet, Groq (the LPUs) delivered a nice tech demo, Mixtral is cool but lacks in certain areas, and Claude can be lengthy. But precisely because I’m not sticking with OAI I can then restate my view that if someone is so good with prompts he can get the same results locally if he knows what he’s doing. Prompting OpenAI the right way can be similarly difficult. Perhaps the whole idea of local inference only matters for IoT scenarios or whenever data is super sensitive (or CTO super stubborn to let it embed and fly). But then if you start from day 1 with WordPress provisioned for you ready to go in Google Cloud, you’d never understand the underlying details of the technology. There sure also must be a good reason why Phind tuned their own thing to offer alongside GPT4 APIs. Disclaimer: tech education is a side thing I do, indeed, and been doing in person for very long time, more than dozen topics, to allow myself to have opinion. Of course business is different matter and strategic decisions arr not the same. Even though I’d not advise anyone to blindly use APIs unless they appreciate the need properly.

dougb5 3y ago

The lessons I wanted from this article weren't in there: Did all of that expenditure actually help their product in a measurable way? Did customers use and appreciate the new features based on LLM summarization compared to whatever they were using before? I presume it's a net win or they wouldn't continue to use it, but more specifics around the application would be helpful.

lordofmoria 3y ago
Hey, OP here! The answer is a bit boring: the expenditure definitely has helped customers - in that, they're using AI generated responses in all their work flows all the time in the app, and barely notice it. See what I did there? :) I'm mostly serious though - one weird thing about our app is that you might not even know we're using AI, unless we literally tell you in the app. And I think that's where we're at with AI and LLMs these days, at least for our use case. You might find this other post I just put up to have more details too, related to how/where I see the primary value: https://kenkantzer.com/gpt-is-the-heroku-of-ai/ https://kenkantzer.com/gpt-is-the-heroku-of-ai/
- kristianp 3y ago
  Can you provide some more detail about the application? I'm not familiar with how llms are used in business, except as customer support bots returning documentation.

haolez 3y ago

That has been my experience too. The null hypothesis explains almost all of my hallucinations. I just don't agree with the Claude assessment. In my experience, Claude 3 Opus is vastly superior to GPT-4. Maybe the author was comparing with Claude 2? (And I've never tested Gemini)

satisfice 3y ago

I keep seeing this pattern in articles like this: 1. A recitation of terrible problems 2. A declaration of general satisfaction. Clearly and obviously, ChatGPT is an unreliable toy. The author seems pleased with it. As an engineer, I find that unacceptable.

jstummbillig 3y ago
ChatGPT is probably in the top 5 value/money subscriptions I have ever had (and that includes utilities). The relatively low price point certainly plays a role here, but it's certainly not a mainly recreational thing for me. These thing's are kinda hard to measure but roughly most + is engagement with hard stuff goes up, and rate of learning goes up, by a lot.
simonw 3y ago
Working with models like GPT-4 is frustrating from a traditional software engineering perspective because these systems are inherently unreliable and non-deterministic, which differs from most software tools that we use. That doesn't mean they can't be incredibly useful - but it does mean you have to approach them in a bit of a different way, and design software around them that takes their unreliability into account.
- jbeninger 3y ago
  Unreliable? Non-deterministic? Hidden variables? Undocumented behaviour? C'mon fellow programmers who got their start in the Win-95 era! It's our time to shine!
Kiro 3y ago
That has nothing to do with you being an engineer. It's just you. I'm an engineer and LLMs are game changers for me.
chx 3y ago
https://hachyderm.io/@inthehands/112006855076082650 https://hachyderm.io/@inthehands/112006855076082650 > You might be surprised to learn that I actually think LLMs have the potential to be not only fun but genuinely useful. “Show me some bullshit that would be typical in this context” can be a genuinely helpful question to have answered, in code and in natural language — for brainstorming, for seeing common conventions in an unfamiliar context, for having something crappy to react to. == End of toot. The price you pay for this bullshit in energy when the sea temperature is literally off the charts and we do not know why makes it not worth it in my opinion.

FranklinMaillot 3y ago

In my limited experience, I came to the same conclusion regarding simple prompt being more efficient than very detailed list of instructions. But if you look at OpenAI's system prompt for GPT4, it's an endless set of instructions with DOs and DONTs so I'm confused. Surely they must know something about prompting their model.

bongodongobob 3y ago
That's for chatting and interfacing conversationally with a human. Using the API is a completely different ballgame because it's not meant to be a back and forth conversation with a human.

Civitello 3y ago

> Every use case we have is essentially “Here’s a block of text, extract something from it.” As a rule, if you ask GPT to give you the names of companies mentioned in a block of text, it will not give you a random company (unless there are no companies in the text – there’s that null hypothesis problem!). Make it two steps, first: > Does this block of text mention a company? If no, good you've got your null result. If yes: > Please list the names of companies in this block of text.

deleted 3y ago

[deleted]

sungho_ 3y ago

I'm curious if the OP has tried any of the libraries that control the output of LLM (LMQL, Outliner, Guadiance, ...), and for those who have: do you find them as unnecessary as LangChain? In particular, the OP's post mentions the problem of not being able to generate JSON with more than 15 items, which seems like a problem that can be solved by controlling the output of LLM. Is that correct?

LASR 3y ago
If you want x number of items every time, ask it to include a sequence number in each output, it will consistently return x number of items. Numbered bullets work well for this, if you don’t need JSON. With JSON, you can ask it to include an ‘id’ in each item.

orbatos 3y ago

Statements like this tell me your analysis is poisoned by misunderstandings: "Why is this crazy? Well, it’s crazy that GPT’s quality and generalization can improve when you’re more vague – this is a quintessential marker of higher-order delegation / thinking." No, there is no "higher-order thought" happening, or any at all actually. That's not how these models work.

Xenoamorphous 3y ago

> We always extract json. We don’t need JSON mode I wonder why? It seems to work pretty well for me. > Lesson 4: GPT is really bad at producing the null hypothesis Tell me about it! Just yesterday I was testing a prompt around text modification rules that ended with “If none of the rules apply to the text, return the original text without any changes”. Do you know ChatGPT’s response to a text where none of the rules applied? “The original text without any changes”. Yes, the literal string.

Suppafly 2y ago
If you look at any of the cake decorating fail sites, humans make that sort of mistake all the time.
mechagodzilla 3y ago
AmeliaBedeliaGPT
phillipcarter 3y ago
> I wonder why? It seems to work pretty well for me. I read this as "what we do works just fine to not need to use JSON mode". We're in the same boat at my company. Been live for a year now, no need to switch. Our prompt is effective at getting GPT-3.5 to always produce JSON.
- Kiro 3y ago
  There's nothing to switch to. You just enable it. No need to change the prompt or anything else. All it requires is that you mention "JSON" in your prompt, which you obviously already do.
  - ShamelessC 3y ago
    I think that’s only true when using ChatGPT via the web/app, not when used via API as they likely are. Happy to be corrected however.
    - 3y ago

kromem 3y ago

Tip for your 'null' problem: LLMs are set up to output tokens. Not to not output tokens. So instead of "don't return anything" have the lack of results "return the default value of XYZ" and then just do a text search on the result for that default value (i.e. XYZ) the same way you do the text search for the state names. Also, system prompts can be very useful. It's basically your opportunity to have the LLM roleplay as X. I wish they'd let the system prompt be passed directly, but it's still better than nothing.

Brunocarl9 3y ago

[dead]

nprateem 3y ago

Anyone any good tips for stopping it sounding like it's writing essay answers, and flat out banning "in the realm of", delve, pivotal, multifaceted, etc? I don't want a crap intro or waffley summary but it just can't help itself.

dudeinhawaii 3y ago
My approach is to use words that indicate what I want like 'concise', 'brief', etc. If you know a word that precisely describes your desired type of content then use that. It's similar to art generation models, a single word brings so much contextual baggage with it. Finding the right words helps a lot. You can even ask the LLMs for assistance in finding the words to capture your intent. As an example of contextual baggage, I wrote a tool where I had to adjust the prompt between Claude and GPT-4 because using the word "website" in the prompt caused GPT-4 (API) to go into its 'I do not have access to the internet' tirade about 30% of the time. The tool was a summary of web pages experiment. By removing 'website' and replacing it with 'content' (e.g. 'summarize the following content') GPT-4 happily complied 100% of the time.

2099miles 3y ago

Great take, insightful. Highly recommend.

chromanoid 3y ago

GPT is very cool, but I strongly disagree with the interpretation in these two paragraphs: I think in summary, a better approach would’ve been “You obviously know the 50 states, GPT, so just give me the full name of the state this pertains to, or Federal if this pertains to the US government.” Why is this crazy? Well, it’s crazy that GPT’s quality and generalization can improve when you’re more vague – this is a quintessential marker of higher-order delegation / thinking. Natural language is the most probable output for GPT, because the text it was trained with is similar. In this case the developer simply leaned more into what GPT is good at than giving it more work. You can use simple tasks to make GPT fail. Letter replacements, intentional typos and so on are very hard tasks for GPT. This is also true for ID mappings and similar, especially when the ID mapping diverges significantly from other mappings it may have been trained with (e.g. Non-ISO country codes but similar three letter codes etc.). The fascinating thing is, that GPT "understands" mappings at all. Which is the actual hint at higher order pattern matching.

fl0id 3y ago
Well, or it is just memorizing mappings. Not like as in reproducing, but having vectors similar to mappings that it saw before.
- chromanoid 3y ago
  Yeah, but isn't this higher order pattern matching? You can at least correct during a conversation and GPT will then use the correct mappings, probably most of the times (sloppy experiment): https://chat.openai.com/share/7574293a-6d08-4159-a988-4f0816f3aafc https://chat.openai.com/share/7574293a-6d08-4159-a988-4f0816...

konstantinua00 3y ago

> Have you tried Claude, Gemini, etc? > It’s the subtle things mostly, like intuiting intention. this makes me wonder - what if the author "trained" himself onto chatgpt's "dialect"? How do we even detect that in ourselves? and are we about to have "preferred_LLM wars" like we had "programming language wars" for the last 2 decades?

egonschiele 3y ago

I have a personal writing app that uses the OpenAI models and this post is bang on. One of my learnings related to "Lesson 1: When it comes to prompts, less is more": I was trying to build an intelligent search feature for my notes and asking ChatGPT to return structured JSON data. For example, I wanted to ask "give me all my notes that mention Haskell in the last 2 years that are marked as draft", and let Chat GPT figure out what to return. This only worked some of the time. Instead, I put my data in a SQLite database, sent ChatGPT the schema, and asked it to write a query to return what I wanted. That has worked much better.

ukuina 3y ago
Have you tried response_format=json_object? I had better luck with function-calling to get a structured response, but it is more limiting than just getting a JSON body.
- egonschiele 3y ago
  I haven't tried response_format, I'll give that a shot. I've had issues with function calling. Sometimes it works, sometimes it just returns random Python code.
  - ukuina 2y ago
    Using the openai Python library?
squigz 3y ago
This seems like something that would be better suited by a database and good search filters rather than an LLM...
- az226 3y ago
  Something something about everything looking like a nail when you’re holding a hammer
- chasd00 3y ago

aubanel 3y ago

> I think in summary, a better approach would’ve been “You obviously know the 50 states, GPT, so just give me the full name of the state this pertains to, or Federal if this pertains to the US government.” Why not really compare the two options, author? I would love to see the results!

pamelafox 3y ago

Lol, nice truncation logic! If anyone’s looking for something slightly fancier, I made a micro-package for our tiktoken-based truncation here: https://github.com/pamelafox/llm-messages-token-helper https://github.com/pamelafox/llm-messages-token-helper

pamelafox 3y ago

I’ve also seen that GPTs struggle to admit when they dont know. I wrote up an approach for evaluating that here - http://blog.pamelafox.org/2024/03/evaluating-rag-chat-apps-can-your-app.html?m=1 http://blog.pamelafox.org/2024/03/evaluating-rag-chat-apps-c... Changing the prompt didn't help, but moving to GPT-4 did help a bit.

deleted 3y ago

[deleted]

Kiro 3y ago

> We always extract json. We don’t need JSON mode, Why? The null stuff would not be a problem if you did and if you're only dealing with JSON anyway I don't see why you wouldn't.

littlestymaar 3y ago

> One part of our pipeline reads some block of text and asks GPT to classify it as relating to one of the 50 US states, or the Federal government. Using a multi-billion tokens like GPT-4 for such a trivial classification task[1] is an insane overkill. And in an era where ChatGPT exists, and can in fact give you what you need to build a simpler classifier for the task, it shows how narrow minded most people are when AI is involved. [1] to clarify, it's either trivial or impossible to do reliably depending on how fucked-up your input is

gok 3y ago

So these guys are just dumping confidential tax documents onto OpenAI's servers huh.

goatlover 3y ago
Hopefully it won't end up as training data.

amelius 3y ago

This reads a bit like: I have a circus monkey. If I do such and such it will not do anything. But when I do this and that, then it will ride the bicycle. Most of the time.

saaaaaam 3y ago
I don’t really understand your comment. Personally I thought this was an interesting read - and more interesting because it didn’t contain any massive “WE DID THIS AND IT CHANGED PUR LIVES!!!” style revelations. It is discursive, thoughtful and not overwritten. I find this kind of content valuable and somewhat rare.
deleted 3y ago
[deleted]

peter_d_sherman 3y ago

>"Lesson 2: You don’t need langchain. You probably don’t even need anything else OpenAI has released in their API in the last year. Just chat API. That’s it. Langchain is the perfect example of premature abstraction. We started out thinking we had to use it because the internet said so. Instead, millions of tokens later, and probably 3-4 very diverse LLM features in production, and our openai_service file still has only one, 40-line function in it: def extract_json(prompt, variable_length_input, number_retries) The only API we use is chat. We always extract json. We don’t need JSON mode, or function calling, or assistants (though we do all that). Heck, we don’t even use system prompts (maybe we should…). When a gpt-4-turbo was released, we updated one string in the codebase. This is the beauty of a powerful generalized model – less is more." Well said!