Tackling Explainability and Interpretability in Language Models

AGENT-BASED ARCHITECTURES

Employing agents fortified by language models is another means of reinforcing transparency for the results of the latter. As Sammons observed, such agents can “do our jobs for us. They can automatically gather customer sentiment, create a campaign against that sentiment, while also updating a product to fix bugs. They can do things we would normally pay lots of people to do.”  Organizations can employ agents to monitor the outputs of reasoning models, as well as whether or not models actually performed the steps they claim they did. With this paradigm, users “have a critique model where its job is to look at the inner monologue of a model and say this is good reasoning or not,” Stephenson explained. “Or, ‘Hey, you need to explore this area.’ This is kind of like a manager.”

Organizations can furnish transparency for the actions agents take by performing what Saurabh Mishra, SAS director of product management, termed “path analysis.” This mechanism typically provides historic information, complete with visual cues, about which steps individual and multiple agents took to complete some of the tasks that Sammons mentioned. Such analysis would reveal “the path the agent followed; it made a language model call here and had to call five tools to get this response, then it finished by posting this on a dashboard,” Mishra said. “It’s like looking under the hood instead of a black box when I ask an agent to do something.” Such auditability buttresses transparency. Additionally, path analysis can be employed prior to deployments as a means of testing the worthiness of agent-based systems.

CONTEXT ENGINEERING

The concept of context engineering emerged as a means of improving the prompt engineering conventionally associated with RAG. By providing more context to language models, the aim is to transition from “a black box to a glass box, where I can see what’s going on inside the model,” Sammons said. Although context engineering is a pivotal construct for contemporary RAG implementations, it’s also important for the employment of reasoning models. According to Alareqi, context for models stems ultimately from the data itself. The more context you give a model, the more it draws out of that context. Consequently, it’s imperative organizations have well-governed, quality data to provide as context for models. Mechanisms like “creating a data dictionary and a slang dictionary that will have internal definitions of what this means in the real world, which will help infer what the prompt might mean, can prompt language models in the right direction,” said Rob Lowe, associate director of digital and AI services at alliantDigital.

Context engineering is important for crafting accurate model responses—which considerably reduces the need for explainability—because it provides information on which models have not necessarily been trained or fine-tuned. Even a model that has been trained on a specific domain would still need contextualization when chatting with an organization’s customer, for example.

How is the voice agent supposed to know what the customer has already purchased?” Stephenson asked. “Or, what’s your name, how to say it, or that you had problems in the past with your returned item and that’s what you’re calling for?” Such details should be included in prompts as context. Furthermore, the input windows for model prompts have become drastically bigger, so that, for example, organizations can include the contents of an entire CRM system in a prompt to provide context. Such detailed, domain-specific information would boost the accuracy of questions in response to that system—making models more accountable and dependable in their outputs.