In Part 2, we discussed key areas where AI can augment your data stack to enable less brittle, resilient data pipelines and processes, reducing time to insight and maintenance overhead. In this part we explore some of the technologies that can bring the Modern Agentic Data Stack to life.
No longer is the question “is AI of value to my company?”, it is “how do I deploy AI at scale in a business safe, cost-effective and predictable manner?”.
Whether AI is being used to enhance and automate internal processes, the focus so far in this series of articles, or to provide value-added services on top of your data, similar requirements apply. Your platform needs to be:
The rest of this article focuses on a set of technologies which meet these criteria and which form a strong foundation for the Modern Agentic Data Stack.
Let’s explore it using the analogy of key components of a car:
This sets the foundations upon which everything else is built. Our platform of choice here is Snowflake.
Let me be clear: there are several highly competent platforms available and any attempt to make a feature-by-feature comparison is both exhausting and futile as they race to outdo each other with every release.
Similarly, discussions around comparative performance are ultimately floored because they are constrained by precise details of the workload and configuration being examined. All platforms seem to be able to produce evidence that theirs is faster than the rest.
Instead, I have chosen to focus on integrated capabilities, how that functionality is put into the hands of users, the deployment, configuration, governance and management experience. After all, getting workloads into production in an efficient, secure and governed way is where value is delivered.
Snowflake is a clear winner here. It offers a comprehensive set of features, from data ingestion, through analysis and ML, to the development and deployment of agentic AI solutions, all within its own highly governed, monitored and integrated environment. It adopts a strong “take your processing to the data” ethos rather than moving data between disparate environments.
Snowflake has recognised that the growth of AI depends as much on companies being able to ensure AI governance and cost control as it does on technical capability. As mentioned above, CxOs are switching focus from AI capability to controlled AI deployment.
To meet these demands, a number of cost monitoring and control features exist in the platform, including the ability to set daily and monthly budgets at the Account and User level, as well as automated, cost-based model selection based on task. These capabilities put control back into the hands of system administrators to prevent out of control AI costs.
Deploying Snowflake services, including AI agents, applications and integration mechanisms such as MCP, is extremely simple and can be done via a browser interface or using its extended variant of SQL. This lends itself well to automation and providing a low barrier to entry. Integrated security and governance across these services removes many of the headaches normally associated with assembling the components required to produce agentic AI systems.
Drive Train
This transfers the power available in the engine to the road in order to get the job of creating motion done.
dbt has emerged across industries as the most popular choice for creating robust, intelligent transformation pipelines for applying logic to data.
It applies software engineering principles and discipline to data transformation, incorporating important features such as high levels of reusability, automated testing, metadata and documentation generation. It lends itself very well to CI/CD approaches and has become a standard in our own AI-driven data migration and transformation solutions.
dbt’s semantic modelling capabilities help establish a shared analytical language across the enterprise which can then be surfaced within Snowflake as Semantic Views. These form a common business-focused data foundation for both AI and BI workloads, ensuring consistency and maintainability.
In effect, dbt becomes part of the governance fabric for enterprise AI and BI.
Most platforms have stopped trying to out-feature dbt and have, instead, opted for deep integration. Snowflake is no exception and, as mentioned above, now offers the option of running dbt as a service, entirely within its own environment and governance framework.
If Snowflake provides the scalable data and AI foundation, dbt provides the engineering discipline and semantic consistency required to operationalise intelligent analytics safely.
It is also worth noting that dbt is now owned by Fivetran, who has affirmed its commitment to continuing the OpenSource Core version.
I have not included a mention of data ingestion in this article (a whole topic of its own, not least because all of the major platforms do have capabilities in this area) but when it comes to standing up no-code, reliable, low maintenance ingestion pipelines, Fivetran is a SaaS solution takes a lot of beating.
Dashboard and Windscreen
For interacting with data in a business context, we propose Sigma.
Visual tools abound, and if providing interactive visual dashboards is your sole objective, then the conversation will inevitably turn into a comparison with Microsoft Power BI or building with OpenSource libraries.
Sigma has taken a different approach, offering deep integration with platforms such as Snowflake whose semantic layer it uses directly, leveraging the power of the platform to perform processing.
Sigma has also embraced the almost universal appeal of Excel, offering a spreadsheet-like interface to analyse cloud-scale data, coupled with rich visual dashboarding, to provide a truly enterprise alternative to the Excel sprawl seen in most companies.
Its headline feature, however, is the ability to write back to the source database, extending its appeal for a wide variety of workloads, including tasks such as budgeting and planning.
Taking these capabilities together, and the ability for Sigma to interact with Snowflake AI Agents, it enables the era of self-service analytics which has been long promised but under-delivered to date.
Conclusion
There are a number of viable combinations of technologies and platforms which can be used to realise the Modern Agentic Data Stack. What is proposed here offers, in my opinion, the best balance of capability coupled with governance and manageability available today.
Technical specifics can be debated, the requirement to deploy agentic solutions quickly, cost-effectively and safely cannot.