Improving AI Outcomes Through Context Engineering
Once upon a time, and that time wasn’t all that long ago, we heard that content was what mattered. Then AI entered the picture, and enterprises shifted from content to context. Content by itself did not lead to relevancy. AI, lacking context, could all too easily misinterpret what was needed. Is a “recent” financial report one that is 1 month, 6 months, or a year old? Is a “current customer” one who has made a purchase in the past quarter or one who maintains a subscription to the company blog? Acronyms pose their own brand of problematic interpretations. Is APR annual percentage rate, annual performance report, or April? What are the business rules that determine context, and how frequently are they maintained, updated, and re-examined?
Context in real life is often intuited by human beings. When the preschooler says, “I want an apple,” you know the request is for a snack and are delighted it’s fruit that is desired rather than candy. Fast-forward a few years. The teenager who whines, “I want an Apple,” is rejecting your choice of a Windows 11 machine and could care less about apple the fruit, preferring to snack on junk food while using his Mac. Intuition works well in human interactions. It doesn’t work that well for enterprise AI. As agentic AI moves into many enterprise operations, determining overall context along with context for individual agents becomes an important element for successful implementations. Plus, the topic of hallucinations that are delivered with utmost confidence even though they convey incorrect information and data has been widely reported and explored. One solution lies in context engineering, which many view as the key to a truly reliable enterprise AI.
I’m reminded of William Carlos Williams’ poem about the red wheelbarrow upon which so much depends (poetryfoundation.org/poems/45502/the-red-wheelbarrow). It was published in 1938. Williams never had to deal with enterprise AI, never asked a chatbot to search a large language model (LLM) for information about wheelbarrows, and never had to disambiguate a red wheelbarrow from a blue wagon. What is lacking from the poem is context—which literary scholars have lined up to supply. Analyses of the poem are vastly longer than the 16 words of the poem itself. And Williams doesn’t clarify what it is, exactly, that depends upon the wheelbarrow (or what role the chickens play).
Perhaps it was just an observation with no additional meanings. Sometimes, a cigar is just a cigar, according to Freud. The same could be said of wheelbarrows. Over-analysis may be an occupational hazard for literary critics, but it’s not entirely unfounded for ensuring that enterprise LLMs get the context right when training agents, particularly when you may have hundreds of agents dealing with multiple, specialized activities and responsibilities. Their ability to communicate with each other makes ensuring that all the agents understand the same context that is imperative for smooth functioning. Getting the analysis of context correct for every possibility within an enterprise leads people to trust the technology.
As agentic AI moves into many enterprise operations, determining overall context along with context for individual agents becomes an important element for successful implementations. Plus, the topic of hallucinations that are delivered with utmost confidence even though they convey incorrect information and data has been widely reported and explored. One solution lies in context engineering, which many view as the key to a truly reliable enterprise AI.
CONTEXT ENGINEERING VERSUS PROMPT ENGINEERING
Context engineering is not precisely the same as prompt engineering. And prompt engineering has often been miscast as simply entering a query into a search box. Advice about best practices for prompting LLMs is rampant and often contradictory. The fact is that much prompting is situational: it’s not a “one size fits all” approach. When someone tells you that they excel at prompt engineering, what they probably mean is that they are very good at translating an information research request into a prompt query that will match what a human wants with what resonates with an LLM.
According to Anthropic (“Effective Context Engineering for AI Agents”; anthropic.com/engineering/effective-context-engineering-for-ai-agents), “Context engineering represents a fundamental shift in how we build with LLMs. As models become more capable, the challenge isn’t just crafting the perfect prompt—it’s thoughtfully curating what information enters the model’s limited attention budget at each step.”
Seeing context engineering as a logical extension of prompt engineering, Anthropic distinguishes between context engineering and prompt engineering “Prompt engineering refers to methods for writing and organizing LLM instructions for optimal outcomes. … Context engineering refers to the set of strategies for curating and maintaining the optimal set of tokens (information) during LLM inference, including all the other information that may land there outside of the prompts.”
ESSENTIALS OF CONTEXT ENGINEERING
In its “Context Engineering: A Practical Guide for AI Agents” (sourcegraph.com/blog/context-engineering), Sourcegraph views the relationship between prompt engineering and context engineering somewhat differently from Anthropic: “Prompt engineering focuses on how to phrase and structure instructions for the LLM to generate the best results, while effective context engineering is about designing the entire system that feeds the model the right context at the right time, including all the context across multiple interactions.”
With context engineering, the scope is considerably more extensive than with prompt engineering, covering a “full set of tokens at inference time,” says Sourcegraph. While prompt engineering looks at optimizing the phrasing of queries and strives for fewer ambiguities, context engineering looks at lessening the amount of noise that results from a query to heighten the signal, essentially leading to better, more targeted responses. Ideally, there’s a “platform team building the agent pipeline,” rather than individuals entering prompts. Context engineering considers actual instructions, documents previously retrieved, and output schemas. History is thus an important element in the context puzzle.
This is not to say that context engineering makes prompt engineering obsolete. Each has its place. However, it’s important to distinguish between how the phrase “prompt engineering” is used in common parlance and how both it and the “context engineering” phrase appear in more technical environments. In Sourcegraph’s view, “If you’re swapping nouns and adjectives, you’re still doing prompt engineering. If you’re changing what data the agent retrieves, in what order, with what re-ranking, and what gets evicted when the context window fills, you’re doing context engineering. Prompt engineering is essential for one-off tasks, but context engineering is what matters for complex tasks and agent systems that maintain conversation history and pull in external data across many turns.”
In the early days of background prompt engineering, the tweaking, modification, and model training that happened was to ensure that responses to queries were relevant. That is still important, but perhaps more importantly, it is also where guardrails could and should be implemented, so that confidential information is not leaked to those not authorized to view it. It’s necessary to measure context quality against retrieval metrics and to think about the potential security risks, including data leakage and system prompt extraction defense.
Context engineering is responsible for making the language models reliable for agentic AI as well as other AI technologies while retaining the guardrails. Context here is important as well. Categories of employees, such as summer interns, retired, or top management should have different levels of access, or none at all, depending on the context of their category.
This isn’t as simple as intuiting what apple is wanted or knowing that it’s apples not oranges when it comes to choosing fruits; it’s understanding the broader picture. Take coding, for example. Advanced LLMs from companies such as Meta, Anthropic, OpenAI, and Google promise to dramatically shorten the time it takes to code a new project or an add-on feature to an existing product. To do this effectively, however, the code must sync with existing code. If it’s not compatible, the project will fail.
This is another context issue. The LLM must know the coding context to create workable new code. Humans who are coding experts must review the AI-created code to not only ensure compatibility but also to determine accuracy. Coding mistakes can sometimes be difficult to pinpoint, and simply asking AI to troubleshoot itself does not always work.