7 ms·

don't miss that OAI also published a prompting guide WITH RECEIPTS for GPT 4.1 specifically for those building agents... with a new recommendation for: - telli

• simonw 2y ago
  I'm surprised and a little disappointed by the result concerning instructions at the top, because it's incompatible with prompt caching: I would much rather cache the part of the prompt that includes the long document and then swap out the user question at the end.
  • swyx 2y ago
    yep. we address it in the podcast. presumably this is just a recent discovery and can be post-trained away.
    • aoeusnth1 2y ago
      If you're skimming a text to answer a specific question, you can go a lot faster than if you have to memorize the text well enough to answer an unknown question after the fact.
  • zaptrem 2y ago
    Prompt on bottom is also easier for humans to read as I can have my actual question and the model’s answer on screen at the same time instead of scrolling through 70k tokens of context between them.
  • mmoskal 2y ago
    The way I understand it: if the instruction are at the top, the KV entries computed for "content" can be influenced by the instructions - the model can "focus" on what you're asking it to do and perform some computation, while it's "reading" the content. Otherwise, you're completely relaying on attention to find the information in the content, leaving it much less token space to "think".
  • jeeeb 2y ago
    Wouldn’t it be the other way around? If the instructions are at the top the LV cache entries can be pre computed and cached. If they’re at the bottom the entries at the lower layers will have a dependency on the user input.
    • a2128 2y ago
      It's placing instructions AND user query at top and bottom. So if you have a prompt like this: [Long system instructions - 200 tokens] [Very long document for reference - 5000 tokens] [User query - 32 tokens] The key-values for first 5200 tokens can be cached and it's efficient to swap out the user query for a different one, you only need to prefill 32 tokens and generate output. But the recommendation is to use this, where in this case you can only cache the first 200 tokens and need to prefill 5264 tokens every time the user submits a new query. [Long system instructions - 200 tokens] [User query - 32 tokens] [Very long document for reference - 5000 tokens] [Long system instructions - 200 tokens] [User query - 32 tokens]
      • jeeeb 2y ago
        Ahh I see. Thank you for the explanation. I didn’t realise their was user input straight after the system prompt.
• swyx 2y ago
  references for all the above + added more notes here on pricing https://x.com/swyx/status/1911849229188022278 https://x.com/swyx/status/1911849229188022278 and we'll be publishing our 4.1 pod later today https://www.youtube.com/@latentspacepod https://www.youtube.com/@latentspacepod
• pton_xd 2y ago
  As an aside, one of the worst aspects of the rise of LLMs, for me, has been the wholesale replacement of engineering with trial-and-error hand-waving. Try this, or maybe that, and maybe you'll see a +5% improvement. Why? Who knows. It's just not how I like to work.
  • pclmulqdq 2y ago
    Software engineering has involved a lot of people doing trial-and-error hand-waving for at least a decade. We are now codifying the trend.
  • zoogeny 2y ago
    I think trial-and-error hand-waving isn't all that far from experimentation. As an aside, I was working in the games industry when multi-core was brand new. Maybe Xbox-360 and PS3? I'm hazy on the exact consoles but there was one generation where the major platforms all went multi-core. No one knew how to best use the multi-core systems for gaming. I attended numerous tech talks by teams that had tried different approaches and were give similar "maybe do this and maybe see x% improvement?". There was a lot of experimentation. It took a few years before things settled and best practices became even somewhat standardized. Some people found that era frustrating and didn't like to work in that way. Others loved the fact it was a wide open field of study where they could discover things.
    • jorvi 2y ago
      Yes, it was the generation of the X360 and PS3. X360 was 3 core and the PS3 was 1+7 core (sort of a big.little setup). Although it took many, many more years until games started to actually use multi-core properly. With rendering being on a 16.67ms / 8.33ms budget and rendering tied to world state, it was just really hard to not tie everything into eachother. Even today you'll usually only see 2-4 cores actually getting significant load.
    • hackernewds 2y ago
      there probably was still a structured way to test this through cross hatching but yeah like blind guessing might take longer and arrive at the same solution
• behnamoh 2y ago
  > - JSON BAD - use XML or arxiv 2406.13121 (GDM format) And yet, all function calling and MCP is done through JSON...
  • CSMastermind 2y ago
    Yeah anyone who has worked with these models knows how much they struggle with JSON inputs.
  • swyx 2y ago
    JSON is just MCP's transport layer. you can reformat to xml to pass into model
• Havoc 2y ago
  >- dont self-inject/parse toolcalls (+2%) What is meant by this?
  • intalentive 2y ago
    Use the OpenAI API/SDK for function calling instead of rolling your own inside the prompt.
• minimaxir 2y ago
  > no evidence that ALL CAPS or Bribes or Tips or threats to grandma work Challenge accepted. That said, the exact quote from the linked notebook is "It’s generally not necessary to use all-caps or other incentives like bribes or tips, but developers can experiment with this for extra emphasis if so desired.", but the demo examples OpenAI provides do like using ALL CAPS.
• kristianp 2y ago
  The size of that SWE-bench Verified prompt shows how much work has gone into the prompt to get the highest possible score for that model. A third party might go to a model from a different provider before going to that extent of fine-tuning of the prompt.
• cedws 2y ago
  Why XML over JSON? Are they just saying that because XML is more tokens so they can make more money?