The AI Beatings Will Continue Until Data Improves

The struggle for AI success has been making news lately. The latest blow was the MIT Media Lab research that found a 95% failure rate for AI pilot projects. There are a lot of factors at play here, but one of the biggest is the lack of commitment and investment in data governance and data management. The good news is that could be changing.

There’s no shortage of research pointing to the underlying data as one of the key sources of AI project struggles. Data quality firm Ataccama today shared findings from its recent Data Trust Assessment, which found organizations scored an average of 42 out of 100 points on its data trust maturity index, with the lowest scores in areas like remediation workflows, policy enforcement, and reference/master data quality. That low score shows there’s a lot of work to do to prepare data for AI.

A Qlik survey from late 2024 identified data governance as the number one challenge (tied with AI skills) preventing organizations from achieving success with their AI projects. And a NetApp survey from the same period suggested that 80% of executives understood data unification to be a critical step to being prepared for AI, while two-thirds said to “optimize” their data for AI by making it “accessible, accurate, and well-documented.”

The importance of having good data management and governance practices isn’t news. It’s been something that companies have struggled with for decades. The big data revolution of the past 15 years was a good reminder of how far companies have to go, and also an appetizer of the types of benefits customers could get if they did it well. Now the main course is here with the mad rush to adopt AI, which is once again exposing how vitally important data management and governance is.

(Harsamadu/Shutterstock)

One company at the forefront of this battle is Indicium, a Brazilian-American provider of data consulting services. The company originally focused mostly on helping customers build big data apps, until it recognized how poorly managed and governed its customers data was. Now it’s focused largely on the “boring stuff” of helping customers build data stacks, said Indicium Co-founder and Chief Data Officer Daniel Avancini.

What “boring” stuff, exactly?

“All the traditional stuff,” he told BigDATAwire. “So you have to integrate your data from silos to a centralized or federated management system…Then most companies need to decide the rules and access controls. Who should have access to all this data? So there’s all the masking stuff, tagging, PII. Like I said, a lot of the boring things are still needed.”

Data meshes and data fabrics were topics of interest in the big data space before ChatGPT landed on us in late 2022. However, much of the talk was theoretical in nature, and companies largely didn’t see the point in investing money to build such a system in their organizations, Avancini said. Discussions around those core technologies for enabling data governance and data management strategies are now back on the table, he said.

“Many CDOs and CTOs are using the AI movement to actually pay for these kind of products, because they are definitely required to use AI or data,” he said. “They understand that AI is so transformative for companies, they’re starting to hear the IT departments and the CDOs say ‘We need to invest on our data management.’”

(Summit Art Creations/Shutterstock)

Indicium is a big backer of the data product approach to data management, which bundles good data management and governance processes and practices into shrink-wrapped data sets or applications that can be widely consumed. The idea behind data products is that it makes it easier for less sophisticated users to consume curated, trusted data without running into data quality issues or violating security or privacy rules.

Going back to basics and focusing on the blocking and tackling of data while generative and agentic AI is racking up the touchdowns may sound counterintuitive. But it’s a winning strategy so far for Indicium, which focuses mainly on financial services and pharmaceutical industries, with some retail and media clients. The company has about 500 employees, a figure that could double within 12 months, Avancini said.

“There’s a clear problem with AI,” he said. “Five years ago, they were saying the same things and the CEO is like, ‘I don’t care. I don’t see the value of investing that much money in data management.’ Now because they have this value with AI, they understand the value [of data management]. They see the middle layer as important.”

That doesn’t mean that there’s an easy button when it comes to data management and governance. Implementing data controls won’t magically transform your business into an AI-enabled business. But it is, in most cases, a necessary step toward that goal. If Indicium’s success is any indication, customers are learning that.

“I think AI is going to be the maybe the driver that leads all those data management initiatives,” Avancini said. “There were stuck because it was hard to solve. It’s really hard to tell your CEO that you need to have federated governance. Like, what is that?”

Related Items:

This Big Data Lesson Applies to AI

Data Management Will Be Key for AI Success in 2025, Studies Say

Focus on the Fundamentals for GenAI Success

The post The AI Beatings Will Continue Until Data Improves appeared first on BigDATAwire.