7 ms·
Context engineering
- elteto 11mo agoAre there any open source examples of good context engineering or agent systems?
- calebkaiser 11mo agoAny of the "design patterns" listed in the article will have a ton of popular open source implementations. For structured generation, I think outlines is a particularly cool library, especially if you want to poke around at how constrained decoding works under the hood: https://github.com/dottxt-ai/outlines https://github.com/dottxt-ai/outlines
- CjHuber 11mo agoI‘d consider DSPy to be one. While the prompts it is using are not the most elaborate, they are well tested and reliable
- voidhorse 11mo agoThere is nothing precise about crafting prompts and context—it's just that, a craft. Even if you do the right thing and check some fuzzy boundary conditions using autoscorers, the model can still change out from beneath you at any point and totally alter the behavior of your system. There is no formal language here. After all, mathematics exists because natural language is notoriously imprecise. The article has some good practical tips and it's not on the author but man I really wish we'd stop abusing the term "engineering" in a desperate attempt to stroke our own egos and or convince people to give us money. It's pathetic. Coming up with good inputs to LLMs is more art than science and it's a craft. Call a spade a spade.
- qrios 11mo agoI agree with you one hundred percent. But: Interestingly, the behavior of LLMs in different contexts is also the subject of scientific research.
- satisfice 11mo agoMy thoughts exactly. The author is saying we should think strategically about the use of context. Sure. Yes. But for that to qualify as engineering we need solid theory about how context works. We don’t have that, yet. For instance experiments show that not all parts of the context window are equally well attended. Imagine trying to engineer a bridge when no one really knows how strong steel is.
- skeeter2020 11mo agoor how wide the river is year round
- chrisweekly 11mo ago"Context crafting", ok, sure. I think a lot of expert researchers (like simonw) would agree.
- calebkaiser 11mo agoI think it's fair to question the use of the term "engineering" throughout a lot of the software industry. But to be fair to the author, his focus in the piece is on design patterns that require what we'd commonly call software engineering to implement. For example, his first listed design pattern is RAG. To implement such a system from scratch, you'd need to construct a data layer (commonly a vector database), retrieval logic, etc. In fact I think the author largely agrees with you re: crafting prompts. He has a whole section admonishing "prompt engineering" as magical incantations, which he differentiates from his focus here (software which needs to be built around an LLM). I understand the general uneasiness around using "engineering" when discussing a stochastic model, but I think it's worth pointing out that there is a lot of engineering work required to build the software systems around these models. Writing software to parse context-free grammars into masks to be applied at inference, for example, is as much "engineering" as any other common software engineering project.
- amonks 11mo agolong shot, apropos of nothing, just recognized your name: If you are the cincinnatian poet Caleb Kaiser, we went to college together and I’d love to catch up. Email in profile. If you aren’t, disregard this. Sorry to derail the thread.
- calebkaiser 11mo agoHello friend!
- grigio 11mo agoI'd like a RSS feed of this blog..
- vladsanchez 11mo agoIt's available, https://buttondown.com/chrisloy/rss https://buttondown.com/chrisloy/rss but it's not in sync with the blog, just a single 2024 entry found. :shrug:
- chrisloy 11mo agoThat's just a feed for my extremely occasional newsletter, this is the blog one: https://chrisloy.dev/rss.xml https://chrisloy.dev/rss.xml
- chrisloy 11mo agoSeems I broke this with a recent change! Reinstated: https://chrisloy.dev/rss.xml https://chrisloy.dev/rss.xml
- aeve890 11mo agoAre we still calling this things engineering?
- skeeter2020 11mo ago"professionally trained & legally responsible for the results" is definitely not the same thing as what we used to just call "good at googling".
- aeve890 11mo agoI'd say this shit is even worse that "good at googling". Literal incantation for stochastic machines is like just two notches above checking the horoscope.
- calebkaiser 11mo agoBased on the comments, I expected this to be slop listing a bunch of random prompt snippets from the author's personal collection. I'm honestly a bit confused at the negativity here. The article is incredibly benign and reasonable. Maybe a bit surface level and not incredibly in depth, but at a glance, it gives fair and generally accurate summaries of the actual mechanisms behind inference. The examples it gives for "context engineering patterns" are actual systems that you'd need to implement (RAG, structured output, tool calling, etc.), not just a random prompt, and they're all subject to pretty thorough investigation from the research community. The article even echoes your sentiments about "prompt engineering," down to the use of the word "incantation". From the piece: > This was the birth of so-called "prompt engineering", though in practice there was often far less "engineering" than trial-and-error guesswork. This could often feel closer to uttering mystical incantations and hoping for magic to happen, rather than the deliberate construction and rigorous application of systems thinking that epitomises true engineering.
- deleted 11mo ago[deleted]
- 11mo ago
- sgt101 11mo agoWhy would I believe that any of this works? This is just some blokes idea of what people should do. There is no evidence offered. No attempt to measure the benefits.
- calebkaiser 11mo agoMost of the inference techniques (what the author calls context engineering design patterns) listed here originally came from the research community, and there are tons of benchmarks measuring their effectiveness, as well as a great deal of research behind what is happening mechanistically with each. As the author points out, many of the patterns are fundamentally about in-context learning, and this in particular has been subject to a ton of research from the mechanistic interpretability crew. If you're curious, I think this line of research is fascinating: https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html https://transformer-circuits.pub/2022/in-context-learning-an...
- sgt101 11mo agoso why does the author not link to or reference this material so that other people can evaluate it?
- deleted 11mo ago[deleted]
- dwaltrip 11mo agoImagine the gall of someone who just goes on the internet and writes something.
- sgt101 11mo agoYeah - random baseless assertions are at the heart of progress.
- alecco 11mo agoThis looks AI generated slop.
- Balgair 11mo agoI know this is a bit of a non sequitur but, on my feed just below your comment, some asked for the RSS for this blog. The juxtaposition of the two comments here is just soooo HN
- OBELISK_ASI 11mo ago[dead]
- 8474_s 11mo agoThe only thing that passed the test of time,so far is specificity: if you ask for multiple things or vague things, you receive half-baked answers trying to cover all bases. If you ask for specific one thing and describe it, the answer quality goes up;e.g. LLMs creating multi-part content mix up the parts and qualities of them, so e.g. asking for Part 1*specific, will always get a better answer than "list all parts of X"(quality drops with length of list).