Architecting a Modern Data Stack for AI Agents

Even if you’ve got a brand-new business model based specifically on the capabilities of a language-centered AI agent, you still have to deal with time-honored challenges such as disparate and scattered data, slow response, finding necessary information, and trusting that information.

On top of these historical data management issues, you now have even more challenges, such as context caching, prompt template storage and reuse, and how to use reflection and other agentic techniques for better output.

I’ve already been hearing that as many as 70%–90% of AI projects don’t make it to production. Gartner expects more than 40% of AI projects at companies will be abandoned by 2027. That is exactly what happened in the early days of machine learning (ML). Making the jump from development and experimentation to dependable production deployment at scale offers many of the same difficulties, whether you’re trying to operationalize a decision tree algorithm or a technical support bot.

BUILD FOR PRODUCTION, NOT JUST DEMOING

It shouldn’t be shocking that if you want something operationalized, it has to scale and handle edge cases and equipment failures, yet still provide fast response. That’s been a requirement for production systems for years. But each time some hot new tech takes off, as ML did a few years back and generative AI (GenAI) is doing now, we go back to making toys, prototypes, and demos and wondering why they break when we hit them with real-life data; high-scale, fast data; plus unpredictable users. AI agent demos and proof of concepts are deceptively easy to make, fooling you into thinking that 90% of the work is done, when it’s really more like 10%.

If you want your agent to work in production, to provide real value to the business, start with the fundamentals and basic data management, and build it on top of that foundation.

For this article, I’ll focus on language models, but many of the same principles apply to image, video, audio, and other types of agents. They always need data.

BUILD FOR FAST RESPONSE, EVEN ON HUGE DATASETS

In the last decade, one of the most impactful changes to data architecture overall hasn’t been the new emphasis on language models and agentic AI workflows, it’s been the move from static datasets and batch data processing to event-driven streaming systems. This shift was driven not just by the increase in streaming data sources, but largely by consumers who now have the expectation of nearly instant, up-to-date responses.

In GenAI agents, this is even more true. Both automated workflows and human interaction require fast response, often when the underlying datasets are massive, moving fast, or both.

My next two points are likely to be controversial. When your goal is to build a data foundation for AI agents, you have to consider that they have slightly different requirements from other types of applications, but since you only want to build one data architecture, you need it to serve both types of requirements.

CENTRALIZE YOUR DATA

There’s been a pendulum swing over the years from centralized data stores to data meshes, data fabrics, and other forms of scattered data and virtualized querying. These have their advantages. The semantic layer that’s essential for a good data fabric tracks where everything is across the organization and how it’s related to other data, not just location relationships, but meaning relationships. The data product concept, in which subject matter experts focus on making the quality of their data excellent before it passes to the larger enterprise, is the most powerful aspect of a data mesh.

Now, it’s time to swing that pendulum back. If your goal is to support AI agents, you need fast response, even at high scale. Low-latency response is extremely difficult to achieve when the data is all over the place.

Pushing queries out to source systems and bringing the answers back is always going to be slower. The simple fact of network latency isn’t something you can handwave away. And streaming data is often an important component needed for AI functionality. Data fabric and mesh designs don’t work well with data in motion.