Why Business Experiments Fail–and How to Avoid It

Should startups bother running experiments? The short answer is a resounding yes — but only if you want the fast-track to a data-backed direction for products and services that people engage with, and that actually work too.

But the research process has to be more accessible to entrepreneurs. At the Center for Advanced Hindsight’s Startup Lab, our mission is to equip startups with the tools to make business decisions firmly grounded in research. Entrepreneurs may have the will to experiment, but often don’t have the time and resources to run studies like our colleagues in academia. Therefore, experimentation has to be especially approachable for entrepreneurs to leverage its benefits for business.

Adapting the Process: Making Experimental Results Actionable

What makes investing in experimentation worthwhile? Businesses need to take this question into account when wading into the world of research. The driving purpose of running experiments in business contexts should be to uncover results that are clearly actionable.

To produce actionable results, you should set up your process with this goal in mind from the start–not as an afterthought. The Startup Lab adapts Alan R. Andreasen’s concept of “backward market research” to bring the process of planning and executing a research project to entrepreneurs.

With the “backward market research” approach, you should first determine what decision you’ll make when you know the results of your research, and let that dictate what data you need to collect and what your results need to look like in order to make that decision.

This “backward” planning is not how businesses usually approach research projects. The typical approach is to start with a problem. In business, this often leads to identifying vague unknowns–a “broad area of ignorance” as Andreasen calls it–and leaves a loosely defined goal of simply reducing ignorance.

This approach–referred to as the the exploratory approach–can set you up for failure. An unclear question produces an unclear answer. You’ll end up with data that you can’t possibly base a concrete decision on.

The Exploratory Approach vs. The Backward Approach

Let’s look at an example of two companies with these opposing research approaches: the Exploratory Entrepreneurs and the Backward Market Research Group. Both are pursuing a venture that aims to help customers stick to an exercise routine.

The Exploratory Entrepreneurs approach product development by first exploring unknowns. They are tempted into asking their customers about their exercising challenges, or their ideal features (but this would be misguided if you don’t first determine what you will do with this information).

If you’re like the Exploratory Entrepreneurs, then you might implement product features based on your conversations, only to find they’re ineffective at accomplishing what your product sets out to do for the user. Will you trash your product and start over? Probably not. But if you set up your research in this way, then that may be the only logical conclusion.

By contrast, the Backward Market Research Group plans a research project to collect data on their users’ exercising behavior (instead of probing to simply reduce ignorance on users’ stated challenges and desires). They set up an experiment to test a potential new feature to increase adherence to an exercise routine (treatment group) against the current product version (control group).

If you’re like the Backward Market Research Group, you’re on the right track. You’ve set up an experiment, but don’t stop at designing your research. The “backward market research” approach forces you to specify which decisions you will make based on the outcome. If your treatment group exercises significantly more than the control group, will you build that new feature into your product? If your data indicates that the new feature contributes positively to your products effectiveness, then yes!

To orient your research toward validated decisions, start with the end in mind and follow through on the predetermined, data-backed decision.

You need to determine the following:

1. The decision you will make from every possible outcome of your research results, and
2. How each decision will be implemented.

Then your research is set up to lead to actionable insights that have the power to move your business forward. Actionable research begins with considering and committing to a decision you will make based on objective learning through a well-planned experiment.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.