14 ms·
> Building powerful and reliable AI Agents is becoming less about finding a magic prompt or model updates. Ok, I can buy this > It is about the engineering of
by JohnMakin 1y ago
> Building powerful and reliable AI Agents is becoming less about finding a magic prompt or model updates.
Ok, I can buy this
> It is about the engineering of context and providing the right information and tools, in the right format, at the right time.
when the "right" format and "right" time are essentially, and maybe even necessarily, undefined, then aren't you still reaching for a "magic" solution?
If the definition of "right" information is "information which results in a sufficiently accurate answer from a language model" then I fail to see how you are doing anything fundamentally differently than prompt engineering. Since these are non-deterministic machines, I fail to see any reliable heuristic that is fundamentally indistinguishable than "trying and seeing" with prompts.
- deleted 1y ago[deleted]
- edwardbernays 1y agoThe state of the art theoretical frameworks typically separates these into two distinct exploratory and discovery phases. The first phase, which is exploratory, is best conceptualized as utilizing an atmospheric dispersion device. An easily identifiable marker material, usually a variety of feces, is metaphorically introduced at high velocity. The discovery phase is then conceptualized as analyzing the dispersal patterns of the exploratory phase. These two phases are best summarized, respectively, as "Fuck Around" followed by "Find Out."
- mentalgear 1y agoIt's magical thinking all the way down. Whether they call it now "prompt" or "context" engineering because it's the same tinkering to find something that "sticks" in non-deterministic space.
- nonethewiser 1y ago>Whether they call it now "prompt" or "context" engineering because it's the same tinkering to find something that "sticks" in non-deterministic space. I dont quite follow. Prompts and contexts are different things. Sure, you can get thing into contexts with prompts but that doesn't mean they are entirely the same. You could have a long running conversation with a lot in the context. A given prompt may work poorly, whereas it would have worked quite well earlier. I don't think this difference is purely semantic. For whatever it's worth I've never liked the term "prompt engineering." It is perhaps the quintessential example of overusing the word engineering.
- ffsm8 1y agoYeah, if anything it should be called an art. The term engineering makes little sense in this context, but really... Did it make sense for eg "QA Engineer" and all the other jobs we tacked it on, too? I don't think so, so it's kinda arguing after we've been misusing the term for well over 10 yrs
- groestl 1y agoWell, to get the right thing into the context in a performant way when you dealing with a huge dataset is definitely engineering.
- shakna 1y agoEngineering tends to mean "the application of scientific and mathematical principles to practical ends". I'm not sure there's much scientific or mathematical about guessing how a non-deterministic system will behave.
- groestl 1y agoFunny how you use a scientific term to discredit applied statistics. I've built useful non-deterministic systems many times and they had nothing to do with AI. Also, particle physics would like to have a word with you.
- shakna 1y agoGuessing, how a non-deterministic system would behave. Statistics isn't guessing. But it is guessing when the confidence interval is unknowable and constantly shifting. We're not talking relativity, we're talking about throwing pancakes at a wall to tell if there's a person behind it.
- deleted 1y ago[deleted]
- surecoocoocoo 1y agoWe used to define a specification. In other words; context. But that was like old man programming. As the laws of physics changed between 1970 and 2009.
- prmph 1y agoIs this Haiku?
- ironmagma 1y agoWhat is all software but tinkering? I mean this not as an insult to software dev but to work generally. It’s all play in the end.
- 8n4vidtmkvmk 1y agoI don't buy this. With software engineering you can generally make incremental progress towards your goal. Yes, sometimes you have to scrap stuff, but usually not the entire thing because an LLM spout out pure nonsense.
- belter 1y agoGot it...updating CV to call myself a VibeOps Engineer in a team of Context Engineers...A few of us were let go last quarter, as they could only do Prompt Engineering.
- tootie 1y agoYou say "magic" I say "heuristic"
- dinvlad 1y ago> when the "right" format and "right" time are essentially, and maybe even necessarily, undefined, then aren't you still reaching for a "magic" solution? Exactly the problem with all "knowing how to use AI correctly" advice out there rn. Shamans with drums, at the end of the day :-)
- andy99 1y agoIt's called over-fitting, that's basically what prompt engineering is.
- evjan 1y agoThat doesn't sound like how I understand over-fitting, but I'm intrigued! How do you mean?
- FridgeSeal 1y agoIt’s just AI people moving the goalposts now that everyone has realised that “prompt engineering” isn’t a special skill.
- coliveira 1y agoIn other words, "if AI doesn't work for you the problem is not IA, it is the user", that's what AI companies want us to believe.
- shermantanktop 1y agoThat’s a good indicator of an ideology at work: no-true-Scotsman deployed at every turn.
- j45 1y agoEverything is new to someone and the tends of reference will evolve.
- csallen 1y agoThis is like telling a soccer player that no change in practice or technique is fundamentally different than another, because ultimately people are non-deterministic machines.
- felipeerias 1y agoIf someone asked you about the usages of a particular element in a codebase, you would probably give a more accurate answer if you were able to use a code search tool rather than reading every source file from top to bottom. For that kind of tasks (and there are many of those!), I don't see why you would expect something fundamentally different in the case of LLMs.
- skydhash 1y agoBut why not provide the search tool instead of being an imperfect interface between it and the person asking? The only reason for the latter is that you have more applied knowledge in the context and can use the tool better. For any other case, the answer should be “use this tool”.
- __loam 1y agoThe uninformed would rather have a natural language interface rather than learn how to actually use the tools.
- skydhash 1y agoThe reason for the expert in this case (an uninformed that wants to solve a problem) is that the expert can use metaphors as a bridge for understanding. Just like in most companies, there's the business world (which is heterogeneous) and the software engineering world. A huge part of software engineer's time is spent translating concepts across the two. And the most difficult part of that is asking questions and knowing which question to ask as natural language is so ambiguous.
- gpm 1y agoBecause the LLM is faster at typing the input, and faster at reading the output, than I am... the amount of input I have to give the LLM is less than what I have to give the search tool invocations, and the amount of output I have to read from the LLM is less than the amount of output from the search tool invocations. To be fair it's also more likely to mess up than I am, but for reading search results to get an idea of what the code base looks like the speed/accuracy tradeoff is often worth it. And if it was just a search tool this would be barely worth it, but the effects compound as you chain more tools together. For example: reading and running searches + reading and running compiler output is worth more than double just reading and running searches. It's definitely an art to figure out when it's better to use an LLM, and when it's just going to be an impediment, though. (Which isn't to agree that "context engineering" is anything other than "prompt engineering" rebranded, or has any staying power)
- Aeolun 1y agoThere is only so much you can do with prompts. To go from the 70% accuracy you can achieve with that to the 95% accuracy I see in Claude Code, the context is absolutely the most important, and it’s visible how much effort goes into making sure Claude retrieves exactly the right context, often at the expense of speed.
- majormajor 1y agoWhy are we drawing a difference between "prompt" and "context" exactly? The linked article is a bit of puffery that redefines a commonly-used term - "context" - to mean something different than what it's meant so far when we discuss "context windows." It seems to just be some puffery to generate new hype. When you play with the APIs the prompt/context all blurs together into just stuff that goes into the text fed to the model to produce text. Like when you build your own basic chatbot UI and realize you're sending the whole transcript along with every step. Using the terms from the article, that's "State/History." Then "RAG" and "Long term memory" are ways of working around the limits of context window size and the tendency of models to lose the plot after a huge number of tokens, to help make more effective prompts. "Available tools" info also falls squarely in the "prompt engineering" category. The reason prompt engineering is going the way of the dodo is because tools are doing more of the drudgery to make a good prompt themselves. E.g., finding relevant parts of a codebase. They do this with a combination of chaining multiple calls to a model together to progressively build up a "final" prompt plus various other less-LLM-native approaches (like plain old "find"). So yeah, if you want to build a useful LLM-based tool for users you have to write software to generate good prompts. But... it ain't really different than prompt engineering other than reducing the end user's need to do it manually. It's less that we've made the AI better and more that we've made better user interfaces than just-plain-chat. A chat interface on a tool that can read your code can do more, more quickly, than one that relies on you selecting all the relevant snippets. A visual diff inside of a code editor is easier to read than a markdown-based rendering of the same in a chat transcript. Etc.
- simonw 1y agoOne crucial difference between prompt and the context: the prompt is just content that is provided by a user. The context also includes text that was output by the bot - in conversational interfaces the context incorporates the system prompt, then the user's first prompt, the LLMs reply, the user's next prompt and so-on.
- ninetyninenine 1y agoYeah but do we have to make a new buzz word out of it? "Context engineer"
- autobodie 1y agoTha problem is that "right" is defined circularly
- PeterStuer 1y ago"these are non-deterministic machines" Only if you choose so by allowing some degree of randomness with the temperature setting.
- zelphirkalt 1y agoThis is what irks me so often when reading these comments. This is just software inside a ordinary computer, it always does the same with the same input, which includes hidden and global state. Stating that they are "non-deterministic machines" sounds like throwing the towel and thinking "it's magic!". I am not even sure what people want to actually express, when they make these false statements. If one wants to make something give the same answers every time, one needs to control all the variables of input. This is like any other software including other machine learning algorithms.
- edflsafoiewq 1y agoIn the strict sense, sure, but the point is they depend not only on the seed but on seemingly minor variations in the prompt.
- pegasus 1y agoThey are usually nondeterministic even at temperature 0 - due to things like parallelism and floating point rounding errors.
- Gracana 1y agoThis is dependent on configuration, you can get repeatable results if you need them. I know at least llama.cpp and vllm v0 are deterministic for a given version and backend, and vllm v1 is deterministic if you disable multiprocessing.
- PeterStuer 1y agofloating point rounding errors are still deterministic. Parallelism dynamics can impact results, but those are not specific to LLM's.
- pbreit 1y agoWhat's the difference?
- colordrops 1y ago> Since these are non-deterministic machines, I fail to see any reliable heuristic that is fundamentally indistinguishable than "trying and seeing" with prompts There are many sciences involving non-determinism that still have laws and patterns, e.g. biology and maybe psychology. It's not all or nothing. Also, LLMs are deterministic, just not predictable. The non-determinism is injected by providers. Anyway is there an essential difference between prompt engineering and context engineering? They seem like two names for the same thing.
- simonw 1y agoThey arguably are two names for the same thing. The difference is that "prompt engineering" as a term has failed, because to a lot of people the inferred definition is "a laughably pretentious term for typing text into a chatbot" - it's become indistinguishable from end-user prompting. My hope is that "context engineering" better captures the subtle art of building applications on top of LLMs through carefully engineering their context.
- v3ss0n 1y agoAt this point , due to non-deterministic nature and hallucination context engineering is pretty much magic. But here are our findings. 1 - LLM Tends to pick up and understand contexts that comes at top 7-12 lines.Mostly first 1k token is best understood by llms ( tested on Claude and several opensource models ) so - most important contexts like parsing rules need to be placed there. 2 - Need to keep context short . Whatever context limit they claim is not true . They may have long context window of 1 mil tokens but only up to avg 10k token have good accuracy and recall capabilities , the rest is just bunk , just ignore them. Write the prompt and try compressing/summerizing it without losing key information manually or use of LLM. 3 - If you build agent-to-agent orchestration , don't build agents with long context and multiple tools, break them down to several agents with different set of tools and then put a planning agent which solely does handover. 4 - If all else fails , write agent handover logic in code - as it always should. From building 5+ agent to agent orchestration project on different industries using autogen + Claude - that is the result.
- zvitiate 1y agoClaude’s system prompt is SO long though that the first 1k lines might not be as relevant for Gemini, GPT, or Grok.
- lblume 1y agoI have uploaded entire books to the latest Gemini and had the model reliably accurately answer specific questions requiring knowledge of multiple chapters.
- yahoozoo 1y agoMind if I ask how you’re doing this? I have uploaded short stories of <40,000 words in .txt format and when I ask questions like “How many chapters are there?” or “What is the last sentence in the story?” it gets it wrong. If I paste a chapter or two at a time then ask, it works better, but that’s tedious…
- deleted 1y ago[deleted]
- phyalow 1y ago“non-deterministic machines“ Not correct. They are deterministic as long as a static seed is used.
- kazga 1y agoThat's not true in practice. Floating point arithmetic is not commutative due to rounding errors, and the parallel operations introduce non-determinisn even at temperature 0.
- phyalow 1y agoWhat? You can get consistent output on local models. I can train large nets deterministically too (CUBLAS flags). What your saying isn't true in practice. Hell I can also go on the anthropic API right now and get verbatim static results.
- simonw 1y ago"Hell I can also go on the anthropic API right now and get verbatim static results." How? Setting temperature to 0 won't guarantee the exact same output for the exact same input, because - as the previous commenter said - floating point arithmetic is non-commutative, which becomes important when you are running parallel operations on GPUs.
- sva_ 1y agoShouldn't it be the fact that they're non-associative? Because the reduction kernels will combine partial results (like the dot‑products in a GEMM or the sum across attention heads) in a way that the order of operations may change (non-associative), which can lead to the individual floats to be round off differently.
- oxidi 1y agoI think lots of people misunderstand that the "non-deterministic" nature of LLMs come from sampling the token distribution, not from the model itself.
- thomastjeffery 1y agoModels are Biases. There is no objective truth. Everything is arbitrary. There is no such thing as "accurate" or "precise". Instead, we get to work with "consistent" and "exhaustive". Instead of "calculated", we get "decided". Instead of "defined" we get "inferred". Really, the whole narrative about "AI" needs to be rewritten from scratch. The current canonical narrative is so backwards that it's nearly impossible to have a productive conversation about it.