Powering AI Agents With RAG and Enterprise Knowledge
It should come as no surprise that AI has already infiltrated many parts of daily life. After all, AI has been around for decades. Still, some people are amazed to discover that things they take for granted, such as spellcheck, autocorrect, real-time traffic maps, spam filters, ride-hailing apps, and personalized music recommendations, are AI-enabled. The more that AI-enabled tools become embedded in daily routines, the less they seem to be advanced technology and more just a normal part of life. It’s no longer magic, it’s normal.
We’re not entirely at the same stage of magic becoming normality when it comes to agentic AI. Many aspects and implementations, however, are becoming standard operating procedures within enterprises. A key driver in powering AI agents is the connection of internal knowledge, wherever it might be stored, with retrieval-augmented generation (RAG).
What agentic AI brings to the enterprise, what sets it apart from the more standard and somewhat simplistic uses of AI, such as spellcheck, is the ability to reason. As agentic AI takes center stage in our enterprise AI world, RAG and enterprise knowledge systems help AI agents to deliver more accurate, contextual, and business-aware responses. Enterprises embracing agentic AI is a logical next step beyond the static chatbots and limited large language models (LLMs) that became commonplace for generative AI (GenAI) only a few years ago. They had no reasoning ability and too easily forgot what they had delivered earlier, even to the same inquiry from the same person.
In addition to a paucity of memory, they were one-trick ponies, unlike the dynamic, multistep, reasoning workflows of today’s agents. The agents employed within enterprises can make decisions and act autonomously. The early LLMs, available at no cost to pretty much everybody, were, by today’s standards, surprisingly limited, although they seemed extremely impressive at the time. Even the more advanced versions currently in existence are generic, not tailored to a particular enterprise or trained on language for a specific discipline. Basically, they are commodities, and their lack of proprietary context makes them less useful within organizations, even though they are perfectly adequate for less challenging, more general tasks. Human oversight should exist, but it is often minimal.
THE VALUE OF RAG
A huge drawback to the acceptance of GenAI into the enterprise was the LLMs’ annoying tendency to hallucinate. As enterprises sought to ensure that their models returned accurate, truthful information, not the plausible but false hallucinated information, they quickly turned to RAG.
RAG provides the very useful antidote to false information by consulting external data sources to validate what it’s found within an LLM. External, that is, to the LLM. Depending on how RAG is implemented, the validity checking might be to sources totally external to an organization. Within enterprises, it’s much more likely to be external to the model, but internal to the organization.
For a query about time-off policies, for example, an external source could be applicable labor laws, but policy manuals, documents, and databases held within the human resources department would probably be consulted first. Context is another area where RAG has value.
A common methodology for handling vast amounts of data is to chunk it, breaking it up into vectors to make finding nuggets within it easier for the model. This can have the unfortunate effect of combining chunks that do not actually relate to each other, thus creating answers that are inaccurate. Outside of the tech world, this can also happen when a statement is taken out of context.
An inaccurate statement is merged with an accurate one, resulting, overall, in misinformation. This can be as relatively innocuous as a statement that John Doe graduated from ABC University when, in fact, he lived for a time in the town where the university is located but did not attend that university. Perhaps he audited a class.
Other statements taken out of context can have more serious repercussions; just ask any public figure. On the positive side, RAG can help identify chunking errors. Scoring and reranking help with this. RAG also pulls from recent data so that newer, more accurate, information overwrites older, now inaccurate, data. In addition to promoting better accuracy and more targeted business-aware responses, this can reduce costs from both financial and computational standpoints.
There’s more to RAG than information retrieval. Agentic AI is a step well beyond GenAI, which is where RAG initially proved its value. Agents act; they adapt to changes in the environment in which they are to perform their tasks; they reason; and they scale. The agentic AI capabilities enabled by RAG include breaking down complex, multilayered, business-centric queries into a series of steps. Agents then follow those steps sequentially. Having done that, they can circle back, going through several iterations until they’ve adequately satisfied the request. According to an email from enterprise search company elastic, “The shift from retrieval to reasoning requires a different approach to building AI systems.”
AGENTIC AI ARCHITECTURE
The architecture behind agentic AI has a number of moving parts, given its autonomous, multi-agent nature. It reflects a complex system with many interdependent components. It’s not a question, however, of simply taking a turnkey approach. It’s not an easy “find and replace” exercise.
It’s not, “Oh, let’s take a chunk of technology, throw it at an agentic AI product, and hope it works.” To be successful, agents should understand what their tasks are, have the relevant training to act autonomously, and carry on with optimal performance. Training agents requires an understanding of all the legacy databases, unstructured data, and cloud resources that are part of the organization’s information infrastructure.
Previously created enterprise knowledge can critically impact the agent training process. This can seem overwhelming. Most companies produce massive amounts of documents in numerous formats. They store manuals, policy papers, press releases, customer correspondence, Slack and Teams messages, emails, spreadsheets, video clips, audio files, and a host of other types of information. Where and how that storage occurs vary greatly from one enterprise to another.
Commonly, however, it’s not nearly as systematically organized as one might wish. To ensure that agents work as intended, some foundational tools are necessary. Moving away from the generic LLMs is an obvious first step. Understanding how to train models for differing formats is also essential. What works for text files doesn’t necessarily work for images, audio, or numeric data.
Which model works best will depend heavily on what the use case is. In most instances, there will be multiple use cases, and one model will not suffice. Instead, enterprises often invest in several models. How many is the right number? That varies considerably from one company to another. approaches that can be adapted to the newer realm of agentic AI. Vector embedding is one.
An IBM explainer (ibm.com/think/topics/vector-embedding) puts it this way: “Vector embeddings are numerical representations of data points that express different types of data, including nonmathematical data such as words or images, as an array of numbers that machine learning (ML) models can process.” The similarity of the numbers reflects close matches back to the original material. The explainer adds, “Armed with such logical assumptions, vector embeddings can be used as inputs to models that perform useful real-world tasks through mathematical operations that compare, transform, combine, sort or otherwise manipulate those numerical representations.” Vector embeddings have long been popular, even before the ascendence of agentic AI, to make data semantically searchable, increase relevancy, and categorize input such as images or customer communications automatically. Vector embeddings are important to RAG by allowing LLMs access to the information stored in vector databases. They’ve acquired more potency as organizations struggle with implementing agentic AI as an operational part of their AI strategy, not as a pilot.