A new stack for a new era

Why the Modern Data Stack is being replaced by the Data & AI Stack

We live in a new era: The era of the Data & AI Stack

In the past era of the Modern Data Stack a global community of data researchers, practitioners, software vendors, and consultants gathered in a planetary movement to work on the perfection of data engineering for data science.

The goal was clear: data was the ultimate raw material for innovation, a source of enterprise growth.

And the ambition of the Modern Data Stack - in its heydays - was the detailed tooling of every single component in data engineering and also (to a lesser extent) data science. No other overview has reached the same level of influence than Matt Turcks MAD Landscape, updated every year with precise insights in technological evolution and always grouped in the most intuitive categories - the MAD landscape is still highly relevant.

Then, ChatGPT 3.5 was released.

That effectively put an end to the era of the modern data stack. VC funding changed from data technologies to AI technologies, and the modern data stack was from that point on experiencing extensive consolidation/rebundling into data intelligence platforms (it’s still ongoing). Data intelligence platforms are used in the context of even bigger ecosystems that unites user experience, data management and operations carried out through workflows - potentially redefining how software is created altogether.

We are now witnessing a technological configuration that can be characterized as the Data & AI Stack, where for example;

- data providers and data consumers are augmented with AI, eg. the Data Analyst as an AI Data Analyst benefitting from a text to SQL to text architecture.

- metadata management is augmented with AI, e.g. metadata data curation through automated description and tagging, natural language search and conversational search.

- the entire purpose of the the data & AI stack is shifting from purely data engineering/science/governance to facilitate generative architectures, and especially agentic architectures, that could potentially rewrite the entire codebase for how companies design and execute their value chains. This has the possibility to turn Software as a Service upside down to Service as Software, as clusters of code generated at a specific point in time for a specific task at hand. I suggest this will be monitored by humans.

The era of the Modern Data Stack is over.

This is the era of the Data & AI stack.