Written by Steve Crosson-Smith, Executive Manager, SDG Group UK

Part 2: Using AI to Augment Your Data Stack

In Part 1, we explored why the modern data stack is no longer sufficient for the demands of an AI-driven enterprise. Here, we discuss some of the practical ways in which AI can be used to augment the data stack, particularly data pipelines, and some of the benefits of doing so.

At the heart of the Modern Agentic Data Stack is the augmentation of pre-determined, rules-based logic with AI-driven, outcomes-based intelligence and continuous observation loops

Most data pipelines are "brittle"; that is, they do not respond to unexpected change well. Such changes arise within the data itself or as a result of environmental conditions, for example, a lost connection or a change made to a system that was not correctly propagated to downstream systems. In many cases, this is not a fault of the data platform itself, but rather in the technical and business processes surrounding it.

The Practical Role of AI in the Modern Agentic Data Stack

1- Automated Quality Assurance and Anomaly Detection

Monitoring the fact that data pipelines complete, with or without errors, is relatively common but generally only results in alerting.

Even if a pipeline does complete, data issues may go unnoticed, resulting in inaccurate analysis or failures downstream. Defining and applying quality checks to all relevant fields can be a huge undertaking and very difficult to do comprehensively.

By using AI agents to analyse a wide range of fields and their historical values, it is possible to automatically generate quality rules to ensure reliable data pipelines.

In addition to this, AI-driven anomaly detection is a technique borrowed from the manufacturing industry. By applying statistical techniques to data values as they flow through the pipelines it is possible, in many cases, to detect anomalies quickly and in far more flexible ways than specifying simple thresholds or rules.

2- Self-Healing Pipelines

Having detected that an issue exists, self-healing pipelines use agentic AI to determine the root cause and make system changes to correct or accommodate.

A simple example of this would be to support schema-drift i.e. the detection of a new field (or a dropped one) in the incoming data source and the reconfiguration of the pipeline to accommodate it. This is a common situation that can often cause pipelines to fail.

Of course, any situation where system behaviour can be altered automatically also requires robust guardrails to be in place to prevent exploitation and abuse.

3- Documentation and Governance

Even if a system has comprehensive and accurate documentation when it is first implemented, this is rarely the case after a significant period of operation, maintenance and modification. A very powerful feature of generative AI is its ability to create or replace detailed documentation automatically, in line with system changes.

This can also apply to governance mechanisms, such as creating metadata (data describing data and structure) and pushing it into a data catalogue or detecting and tagging PII or other sensitive data, for example.

4- Cost and Performance Management

Cost and performance management should be treated together as one significantly impacts the other.

For a given set of performance objectives, AI can help to optimise resource utilisation (for example, right-sizing compute, storage and the choice of LLM for a particular task).

Advantages of the Modern Agentic Data Stack

Done correctly, a Modern Agentic Data Stack will provide:

• A more trustworthy data foundation on which to build BI and AI-driven business outcomes

• Greater reliability and resilience in your data architecture

• Shorter time to insight and action

• Greater agility

• Reduced maintenance overhead

• Cost optimisation
  
  

In Part 3, we'll look at a technology stack that makes this possible and why the choices you make now will define your organisation's AI trajectory.