What Is The Difference Between Data-informed, Data-backed, And Data-driven?

The popularity of search terms like “data-driven,” “data-informed,” and “data-backed” are at an all-time high.

If your media consumption looks anything like mine, you’ve probably read countless articles about all the ways in which tech leaders leverage data to make informed decisions.

But unfortunately, as is the case with industry buzzwords, lots of organizations say they make “data-backed decisions” but simply don’t have the culture or toolkit to support them. Period.

They may run a UX research sprint or mine the data to understand their challenges, but if they fail to validate their ideas, they fall short of actually being data-backed. They are just data-informed.

With the advent of analytics platforms, tracking pixels, and attribution modeling, it seems like everyone is an analyst. Or at least they play one in meetings.

The Data Evolution

As Stanford’s Ramesh Johari notes, “100 years ago, data was expensive and slow and practitioners were trained statisticians. Today, data is cheap and real-time, and everyone is a practitioner.”

With the increased access to data, leaders are antsy to put that data to work. We regularly speak with leaders who want to transform their company’s relationship to their data: they run heat mapping tools on their site, have numerous analytics tools at their disposal, and their teams have dabbled in A/B testing. But many still struggle to make sense of the abundance of data and choose the right path forward. They struggle to move from data-informed to data-backed.

And being data-backed isn’t about just having a good data story to support your argument for why another treatment is better. You have to go beyond that and do the evaluative research to actually PROVE it to create a culture that is data-driven.

Though the language is similar, we make a crucial distinction between data-informed and data-backed.

Generative vs Evaluative Research

Many organizations aspire to be data-backed, but they often fall short by primarily relying on generative research methods. 

Generative methods are great for understanding what’s happening on a website and forming hypotheses about what would work better.

Generative research methods include things like: 

• Heatmap analysis
• Surveys
• Data analysis
• Observational analysis
• Open card sorting
• Reviews theming
• Social listening

Alternatively, evaluative research can be used to substantiate ideas with evidence. Evaluative research methods include things like: 

• A/B testing
• First-click testing
• Comparison testing 
• Tree testing

When we ask so-called “data-driven” companies what data they use, they typically list a number of generative research methods. Only about 25% of them list any evaluative research methods and rarely do they go beyond A/B testing.

Generative research, those methods on the left, are great for understanding what’s happening on a website and forming hypotheses about what would work better.

But in order to move forward with confidence that your solutions will actually work, you need the methods on the right. Evaluative research is how we move from data-informed to data-backed and eventually develop a company culture that is data-driven.

What to continue learning?

• Here’s a great example of a data-driven culture and efficiency-first mindset in action: How Andy Lightman’s Ecommerce Philosophy Created Millions in Revenue. “We could end up using half our development budget on building something that doesn’t perform… We didn’t want to overburden the development pipeline with projects that don’t have a financial impact.” 
• Dive into the evaluative research methods you can use to support your effort to be a data-driven organization: What is Rapid Validation and How is it Different from A/B Testing?