Powering AI Agents With RAG and Enterprise Knowledge

CONVERTING DATA TO AI-USABLE FORMATS

To enable agents to access legacy enterprise knowledge in all its multitudinous incarnations, incantations don’t work. Instead, a mechanism to convert unstructured data to structured, usable-for AI-data is needed. Several possibilities exist for this activity; many of them are tried-and-true older Enterprise service bus is another approach. This is “bus” in the computer science sense of the word, not the yellow school bus that goes by my front door at 7:30 a.m. Monday through Friday. According to AWS (aws.amazon.com/what-is/enterprise-service-bus), “The enterprise service bus (ESB) is a software architectural pattern that supports synchronous and asynchronous data exchange between separate applications. Large organizations have multiple applications that perform various functions using diverse data models, protocols, and security restrictions. An ESB makes application integration easier by performing operations such as data transformation, protocol conversion, message routing, and orchestration. Applications publish relevant data to the ESB, and it converts and forwards the data to other applications that need it.”

AWS goes on to say that ESBs are not used as much as they were, as the preference has switched to API gateways. Microsoft explains that an API gateway “provides a centralized entry point for managing interactions between clients and application services” (learn.microsoft.com/en-us/azure/architecture/microservices/design/gateway). “An API gateway simplifies communication, enhances client interactions, and centralizes the management of common service-level responsibilities. It acts as an intermediary, and it prevents direct exposure of application services to clients.”

RAG, LEGACY DATA, AND THE FUTURE

Whatever technology is used to make older information AI-ready—and it may well be more than one, based upon individual use cases—it can be an arduous process. RAG is powerful, but not omniscient, nor will it solve every content issue within enterprises. For one thing, RAG was designed with text in mind. Applying it to images and audio is problematic.

Depending on the format of the data, going back to the source code could be necessary. Strategic decisions about what legacy data is important for agents and what can be ignored need to be made. There’s no magic spell that can be invoked, no “presto chango” that instantaneously causes all that legacy data to become easily findable. Moreover, it’s possible that legacy systems have unacknowledged intrinsic value. It’s best to run some tests to ensure that AI can access older data. And not just access it, but also deliver—accurately, quickly, and without additional steps—needed information that is routinely requested by users.

Going forward, the key is having standard approaches to newly created content so that no backward compatible issues will need to be addressed. Consistent and up-to-date metadata, agreed-upon schemas, Model Context Protocol (MCP) connectors, and curated catalogs contribute to agentic AI-ready data. Agentic RAG will play a major part, as queries become evermore nuanced and require a more advanced series of steps and decision points. If all human workflows were as simple as “if this, then that,” we wouldn’t need AI agents.

But they’re not. Decisions about credit lines, supply chains, regulatory approvals, and exceptions to policies are far from being amenable to formulaic solutions. There is real science behind transforming legacy content into information usable for AI agents and to forestall problems going forward. Varieties of RAG technology hold great promise for reducing, if not totally eliminating, hallucinated data and providing the magical single source of truth.

Of course, it’s not really magic, but as Arthur C. Clarke propounded as one of his three futuristic laws, “Any sufficiently advanced technology is indistinguishable from magic.” The corollary is that, when those advanced technologies, in this case agentic AI, become just part of people’s everyday lives, they no longer seem magical. Meanwhile, we’ll look for how agentic and other AI technologies can bridge the gap between what is needed for real business outcomes and what is still in the realm of fantasy.