Why Business Experiments Fail–and How to Avoid It

Should startups bother running experiments? The short answer is a resounding yes — but only if you want the fast-track to a data-backed direction for products and services that people engage with, and that actually work too.

But the research process has to be more accessible to entrepreneurs. At the Center for Advanced Hindsight’s Startup Lab, our mission is to equip startups with the tools to make business decisions firmly grounded in research. Entrepreneurs may have the will to experiment, but often don’t have the time and resources to run studies like our colleagues in academia. Therefore, experimentation has to be especially approachable for entrepreneurs to leverage its benefits for business.

Adapting the Process: Making Experimental Results Actionable

What makes investing in experimentation worthwhile? Businesses need to take this question into account when wading into the world of research. The driving purpose of running experiments in business contexts should be to uncover results that are clearly actionable.

To produce actionable results, you should set up your process with this goal in mind from the start–not as an afterthought. The Startup Lab adapts Alan R. Andreasen’s concept of “backward market research” to bring the process of planning and executing a research project to entrepreneurs.

With the “backward market research” approach, you should first determine what decision you’ll make when you know the results of your research, and let that dictate what data you need to collect and what your results need to look like in order to make that decision.

This “backward” planning is not how businesses usually approach research projects. The typical approach is to start with a problem. In business, this often leads to identifying vague unknowns–a “broad area of ignorance” as Andreasen calls it–and leaves a loosely defined goal of simply reducing ignorance.

This approach–referred to as the the exploratory approach–can set you up for failure. An unclear question produces an unclear answer. You’ll end up with data that you can’t possibly base a concrete decision on.

The Exploratory Approach vs. The Backward Approach

Let’s look at an example of two companies with these opposing research approaches: the Exploratory Entrepreneurs and the Backward Market Research Group. Both are pursuing a venture that aims to help customers stick to an exercise routine.

The Exploratory Entrepreneurs approach product development by first exploring unknowns. They are tempted into asking their customers about their exercising challenges, or their ideal features (but this would be misguided if you don’t first determine what you will do with this information).

If you’re like the Exploratory Entrepreneurs, then you might implement product features based on your conversations, only to find they’re ineffective at accomplishing what your product sets out to do for the user. Will you trash your product and start over? Probably not. But if you set up your research in this way, then that may be the only logical conclusion.

By contrast, the Backward Market Research Group plans a research project to collect data on their users’ exercising behavior (instead of probing to simply reduce ignorance on users’ stated challenges and desires). They set up an experiment to test a potential new feature to increase adherence to an exercise routine (treatment group) against the current product version (control group).

If you’re like the Backward Market Research Group, you’re on the right track. You’ve set up an experiment, but don’t stop at designing your research. The “backward market research” approach forces you to specify which decisions you will make based on the outcome. If your treatment group exercises significantly more than the control group, will you build that new feature into your product? If your data indicates that the new feature contributes positively to your products effectiveness, then yes!

To orient your research toward validated decisions, start with the end in mind and follow through on the predetermined, data-backed decision.

You need to determine the following:

1. The decision you will make from every possible outcome of your research results, and
2. How each decision will be implemented.

Then your research is set up to lead to actionable insights that have the power to move your business forward. Actionable research begins with considering and committing to a decision you will make based on objective learning through a well-planned experiment.

Why an NYU Neuroscientist Declared Exercise the Most Transformative Thing You Can Do for Your Brain

If you’re as busy as I am, finding the time to exercise can seem harder than bobsledding up a tree. But according to Dr. Wendy Suzuki, professor of neuroscience and psychology at New York University, getting off your couch or up from your desk through your workweek can have an insanely positive influence on your brain. In fact, in her November 2017 TEDWomen talk, Suzuki went so far as to say that physical exercise is the most transformative thing you can do for your mind.

Inspiration from experience

Suzuki explains that, like many a modern worker, her work left her pretty inactive physically. She was miserable, had gained weight and discovered the full extent of her weakness on a river rafting trip. Determined for change, she attacked her gym, trying just about every type of class they had. Her extra weight disappeared.

But her mood was better. Energy was better. Even writing her grants was easy, because she could focus.

Suzuki put two and two together and hypothesized that it likely was her exercise that was making a difference. But like any scientist, she wasn’t satisfied. She wanted details, so she started reviewing the literature and conducting new research in her own lab.

Your brain on workouts

Suzuki identified exercise as powerful for the brain for three reasons.

1. Physical activity gives you an immediate jolt.

As Suzuki explains, when you exercise, the brain releases a flood of neurotransmitters. These natural chemicals serve as messengers, helping cells in the body communicate and perform optimally.

Two of the chemicals released, serotonin and dopamine, help you feel happy, offering pleasure. Dopamine also plays a role in keeping you curious and motivated. A third chemical, noradrenaline (norepinephrine), is often connected to fight-or-flight. It ensures that you are physically able to respond, making sure your brain and heart both get plenty of blood and oxygen.

Taken together, these neurotransmitters translate to better mood, faster reaction times and an improved ability to shift and focus attention. What’s more, Suzuki’s research indicates the improvements can last up to two hours after your sweat fest.

2. Exercise literally changes your brain’s function, physiology and anatomy for the long-term.

If you engage in exercise consistently, improving cardiovascular function, exercise actually can produce new brain cells. More specifically, scientists see new development particularly in the prefrontal cortex and hippocampus.

You use your prefrontal cortex for critical thinking, including planning and decision making. But it also is important for your personality and control of social behavior. The hippocampus is associated with the formation and retention of long-term memories for facts and events.

As the size of your prefrontal cortex and hippocampus improves, all of the tasks they’re associated with can get better, too. The brain really is like a muscle in that the more you’re packing, the easier hardcore work gets.

3. Working out can protect you from neurodegenerative disease and decline.

Think of your brain like a tree for a second. If you were a puny little sapling, one good wind or blow of an axe might topple you. But if you were big and stout with many, many rings of growth, you’d still be standing after big gusts, and a lumberjack would have to work pretty dang hard before yelling timber.

Now review the second point above. What makes your brain big and stout? Exercise! It doesn’t guarantee you won’t have the wind or lumberjack beating on you. But it does ensure that your prefrontal cortex and hippocampus can take more damage before you start to show symptoms. Diseases like Alzheimer’s or normal age-related decline thus might take longer to have an effect. For this reason, Suzuki says exercise is like a “supercharged 401K for your brain”. The more you invest through physical activity, the more you’re prepared to maintain a high quality of life in your future.

How often to put on your sneakers

Suzuki says that the minimum amount you can exercise and see the above benefits is around three to four times a week, 30 minutes per workout. The exercise should include truly aerobic activity that elevates your heart rate. But she stresses there’s no need for fancy equipment or an expensive gym. You can do most cardio virtually anywhere, and when it comes to strength, there are plenty of body-weight exercises. Just modify as needed for your fitness level.

Going the extra mile

Suzuki is confident about the minimum exercise recommendation. But she doesn’t want to take a one-size-fits-all approach. Her goal, she says, is to go beyond the basic rule of thumb and find your optimum. What is ideal for you? How can you maximize your results and give yourself the best protection given factors like your age, fitness level and specific genetics? You might not have that answer for a while yet. But you can at least get a start and pencil in some sessions based on what we know so far.

If you need even more inspiration, remember that exercise has benefits far beyond mental performance. Self-acceptance, discipline, independence, being truly present–you learn all those kinds of things over and over again through physical challenges. If you really want to be successful, those lessons are pretty tough to pass up.

Why an NYU Neuroscientist Declared Exercise the Most Transformative Thing You Can Do for Your Brain

If you’re as busy as I am, finding the time to exercise can seem harder than bobsledding up a tree. But according to Dr. Wendy Suzuki, professor of neuroscience and psychology at New York University, getting off your couch or up from your desk through your workweek can have an insanely positive influence on your brain. In fact, in her November 2017 TEDWomen talk, Suzuki went so far as to say that physical exercise is the most transformative thing you can do for your mind.

Inspiration from experience

Suzuki explains that, like many a modern worker, her work left her pretty inactive physically. She was miserable, had gained weight and discovered the full extent of her weakness on a river rafting trip. Determined for change, she attacked her gym, trying just about every type of class they had. Her extra weight disappeared.

But her mood was better. Energy was better. Even writing her grants was easy, because she could focus.

Suzuki put two and two together and hypothesized that it likely was her exercise that was making a difference. But like any scientist, she wasn’t satisfied. She wanted details, so she started reviewing the literature and conducting new research in her own lab.

Your brain on workouts

Suzuki identified exercise as powerful for the brain for three reasons.

1. Physical activity gives you an immediate jolt.

As Suzuki explains, when you exercise, the brain releases a flood of neurotransmitters. These natural chemicals serve as messengers, helping cells in the body communicate and perform optimally.

Two of the chemicals released, serotonin and dopamine, help you feel happy, offering pleasure. Dopamine also plays a role in keeping you curious and motivated. A third chemical, noradrenaline (norepinephrine), is often connected to fight-or-flight. It ensures that you are physically able to respond, making sure your brain and heart both get plenty of blood and oxygen.

Taken together, these neurotransmitters translate to better mood, faster reaction times and an improved ability to shift and focus attention. What’s more, Suzuki’s research indicates the improvements can last up to two hours after your sweat fest.

2. Exercise literally changes your brain’s function, physiology and anatomy for the long-term.

If you engage in exercise consistently, improving cardiovascular function, exercise actually can produce new brain cells. More specifically, scientists see new development particularly in the prefrontal cortex and hippocampus.

You use your prefrontal cortex for critical thinking, including planning and decision making. But it also is important for your personality and control of social behavior. The hippocampus is associated with the formation and retention of long-term memories for facts and events.

As the size of your prefrontal cortex and hippocampus improves, all of the tasks they’re associated with can get better, too. The brain really is like a muscle in that the more you’re packing, the easier hardcore work gets.

3. Working out can protect you from neurodegenerative disease and decline.

Think of your brain like a tree for a second. If you were a puny little sapling, one good wind or blow of an axe might topple you. But if you were big and stout with many, many rings of growth, you’d still be standing after big gusts, and a lumberjack would have to work pretty dang hard before yelling timber.

Now review the second point above. What makes your brain big and stout? Exercise! It doesn’t guarantee you won’t have the wind or lumberjack beating on you. But it does ensure that your prefrontal cortex and hippocampus can take more damage before you start to show symptoms. Diseases like Alzheimer’s or normal age-related decline thus might take longer to have an effect. For this reason, Suzuki says exercise is like a “supercharged 401K for your brain”. The more you invest through physical activity, the more you’re prepared to maintain a high quality of life in your future.

How often to put on your sneakers

Suzuki says that the minimum amount you can exercise and see the above benefits is around three to four times a week, 30 minutes per workout. The exercise should include truly aerobic activity that elevates your heart rate. But she stresses there’s no need for fancy equipment or an expensive gym. You can do most cardio virtually anywhere, and when it comes to strength, there are plenty of body-weight exercises. Just modify as needed for your fitness level.

Going the extra mile

Suzuki is confident about the minimum exercise recommendation. But she doesn’t want to take a one-size-fits-all approach. Her goal, she says, is to go beyond the basic rule of thumb and find your optimum. What is ideal for you? How can you maximize your results and give yourself the best protection given factors like your age, fitness level and specific genetics? You might not have that answer for a while yet. But you can at least get a start and pencil in some sessions based on what we know so far.

If you need even more inspiration, remember that exercise has benefits far beyond mental performance. Self-acceptance, discipline, independence, being truly present–you learn all those kinds of things over and over again through physical challenges. If you really want to be successful, those lessons are pretty tough to pass up.

This Ostrich-Legged Robot May Be Your Next Pizza Delivery Guy

Agility Robotics, which was spun out of Oregon State University in early 2017, is building robots that can traverse any terrain–from rocky hillside to sidewalks. Its first product, Cassie, has found work at half a dozen research institutions. The Albany, Oregon-based company, which will close its Series A in early 2018, envisions the bot eventually being used for everything from deliveries to facility inspections to hazardous search-and-rescue operations.

The Brains Behind the Operation

Cassie has two internal com­puters. One operates a control loop that is responsible for the lower-order functions that keep Cassie operating safely–just as the cerebellum in humans and other animals controls posture and balance. The other, which responds to commands from its human controllers, is a sort of traffic cop, handling all active movement. This might seem complex, but the robot is designed to be user friendly: “You can walk it around your lab the day you buy it,” Hurst says. And you don’t have to take it to lunch.

CREDIT: Dan Saelinger

Under the Hood

Cassie’s motor must be on for it to stand upright; cut the power and it collapses in a heap. “That’s actually an advantage,” says Agility Robotics co-founder and CEO Damion Shelton. “It means there’s no built-in mechanical stability to the system, which makes its ability to react to disturbances and unexpected ground positioning really high.”

How to Balance a Bot

Objects that are top heavy are actually easier to balance, so Cassie is designed with more than 85 percent of its weight above the top joint. The bot’s lightweight legs provide a second benefit: Very little energy is required to move them, making the machine quicker to react and more efficient overall.

The Beauty of Ostrich Legs

Cassie’s avian aesthetic isn’t exactly by design. As part of his research at OSU, Agility co-founder and CTO Jonathan Hurst and his team closely studied the ways various animals perambulated, and then created a robot that replicated those mechanics. The resulting system, called Atrias, achieved the same basic output of a strolling animal–though it looked and moved nothing like one. To create Cassie, the Agility team used Atrias as a sort of first step, and then performed analyses to improve efficiency. “It turns out,” says Shelton, “if you start with a mechanism that represents the motion of animals but not the design of animals, and then you do a bunch of math to remove all the extra bits that you don’t need from the design, what you get back is something that looks a lot like an ostrich.”

Cats and Dogs and Dinosaurs

Like dogs, cats, rodents, and most birds, Cassie has a digitigrade walking style, meaning it walks on its toes. (Dinosaurs did this too. You, by contrast, plant your heel first when walking.) What look like inverted knees are actually the anatomical equivalent of Cassie’s ankles; the long rods below these joints are the feet, and the parts that touch the ground are the digits.

Yearning to Go Free

The company’s goal is for the bots to operate autonomously–say, in a dangerous industrial environment. For now, Cassie can be run from a laptop–although that’s actually technological overkill. “We found out,” Shelton says, “that you can’t engineer a solution that’s better than what people use for remote-control airplanes, so we just bought a bunch of those remotes.”

Cassie’s Next Task: Job Hunting

Agility’s current customers are research institutions, including the University of Michigan. The company is selling the bots with extensive warranties. “That’s to encourage people to push limits of what’s possible with the robot,” Shelton says. “We didn’t want them to feel like they have to baby it, because the goal is to have people figuring out what works well and what doesn’t work before we sell to corporate customers.” Agility’s plans include giving Cassie arms so it can do things like deliver pack­ages, carry tools, help around the house–and pick itself up when it falls down.

Amazon’s Jeff Bezos Just Proved That All Those Rich And Famous Do-What-I-Do Stories Are Nonsense

Sometimes, luck wins.

“How to be successful” returns 843 million results on Google. “I will teach you to be rich” is an actual website. And we can’t go a day without seeing how four habits will make us successful, or seven things the most successful people are always doing, or, from the astonishingly loquacious and for the above-average attention span, 37 secrets only successful people know.

All of it is nonsense.

Don’t get me wrong: there’s a lot of good advice, and there’s a lot that can help people. Go ahead: read one of these every day. Just don’t believe that they will make the entire difference between you living on $45,000/year and building the next Facebook.

Because … luck.

Don’t take my word for it. Listen to Amazon founder, new richest person on Earth, and currently a billionaire 130.5 times over. (Hold your breath for a moment, and if Amazon stock goes up, it’ll be 135 times.)

“The price of admission to space is very high,” Bezos recently said as he accepted a Buzz Aldrin exploration award. “I’m in the process of converting my Amazon lottery winnings into a much lower price of admission so we can go explore the solar system.”

There’s only two things you need to pay attention to in that statement.

One, it takes a lot of money to be a space explorer.

And two, Bezos acknowledges that his extraordinary success and stupendous wealth are, at least to some degree, the result of an incredible streak of luck.

There’s no doubt that Bezos is smart.

There’s no doubt that Bezos is visionary.

There’s no doubt that Bezos is a super-talented technologist and leader and business person and manager.

Even so, in his own words, his tremendous fortune is the result of “Amazon lottery winnings.” That might just be seen as simple politeness, downplaying his own role in amassing the modern world’s biggest fortune. But I don’t think so.

I think Bezos gets it.

Even super-smart and super-successful people rely on a lot of luck (and the help of friends) to get where they’ve gone. Where would Amazon be without the internet? Where would Amazon be if Bezos couldn’t find investors? Where would Amazon be if stock markets hadn’t been so forgiving of its lack of desire to show quarterly profits for almost two decades? And where would Amazon be today if a wonderful confluence of natural language processing, artificial intelligence, cloud technology, device manufacturing, and massive distribution power hadn’t made its Echo line of products one of the most successful in recent years?

Much of that required seizing opportunity by the horns and running with it.

But much of it required sheer, blind, luck. And thanks to Jeff Bezos’ comments, we can see that in operation from one of the most successful persons in the entire world.

Remember that next time you see a “5 steps to instant success” article on the internet.

How Your Past Success Can Lead to Future Failure

Many advertisements for investment opportunities warn that past success is no guarantee of future results. The reality is much worse than that. Past success may actually lead to future failure.

In the 1960s, Sears accounted for one percent of the U.S.’s gross national product. At that time, one out of every two American households had a Sears credit card. Sears wasn’t just the world’s largest retailer, it was a dominant force in the U.S. economy.

In 1983, Sears was still riding the success train and was over 10 times the size of Walmart. In six short years, things changed. October 1989 ushered in Walmart as the world’s largest retailer, and the rapid decline of Sears continued.

Today, Walmart has revenues over 20 times that of Sears Holdings (which also includes K-Mart). Although Amazon is banging on the door of the retail giant, Walmart still has revenues that are nearly three times that of everything Amazon-related, which goes way beyond just retail. 

Sears’s fall from grace is, of course, not unique. Pan American Airways, one of the largest airlines at one point, became extinct in 1991. The largest food provider in the 1960s was Howard Johnson’s restaurants (remember those orange roofs?), yet there is only one left in the country. Kodak invented the digital camera, yet it was that technology that pretty much killed the camera company. Although Circuit City was in the book Good to Great, the electronics company should have been in the book Good to Gone as it, too, is out of business. 

The list goes on and on. For each company, the specific circumstances of its demise may be different, but the underlying reason is the same: past success led to future failure. 

There are two reasons for this:
1.    Complacency/arrogance
2.    Expertise blindness

Complacency/Arrogance

Many great companies fail because their leaders believe they are too great to be dethroned. I’ve heard executives say, “Well, that may have happened to Sears, but it won’t happen to us. We are different.”

No, you aren’t. In fact, things are more challenging today than they were back when Sears was the top dog.

Success requires checking your ego at the door. Trust me, around the corner, there is someone who is confident he or she can make you irrelevant. And given past evidence, I would put my money on the new entrant, not the incumbent.

In 2008, Jim Keyes, Blockbuster’s CEO, said, “Neither RedBox nor Netflix are even on the radar screen in terms of competition.” Three years later, Blockbuster declared bankruptcy and two years after that, they were defunct. Of course, now Netflix is now the king of video.

Expertise Blindness

The second reason for this failure is something I call expertise blindness. When you know a topic well, it makes it difficult for you to think differently about it. If you spent the last 30 years in your industry, any ideas you develop for the future are most likely extrapolations of the past. That is, they are incremental changes. 

But the world around us is not changing at an incremental pace; it’s evolving at an exponential rate.

Expertise is the enemy of non-incremental innovation.

I find that the fastest way to overcome expertise blindness is by asking: “Who else has solved a similar problem in a different industry?” Anecdotal evidence and numerous studies prove that when people are cut from the same cloth, you only yield incremental improvements. The breakthroughs, in almost every case, are from either connecting to a different discipline, or from multi-disciplinary teams coming together to solve a problem.

Looking externally is critical to staying relevant. Assume that someone will eventually make you irrelevant. Who are the likely and unlikely suspects? Partner with external think tanks, universities, and trend watchers to better understand the shifts taking place from a technological and demographic perspective. Create a corporate radar system that senses and responds to these changes. And look outside your industry or function to see what’s really happening. 

For you to succeed in the future, the pace of change inside your organization needs to be faster than the pace outside.

How Your Past Success Can Lead to Future Failure

Many advertisements for investment opportunities warn that past success is no guarantee of future results. The reality is much worse than that. Past success may actually lead to future failure.

In the 1960s, Sears accounted for one percent of the U.S.’s gross national product. At that time, one out of every two American households had a Sears credit card. Sears wasn’t just the world’s largest retailer, it was a dominant force in the U.S. economy.

In 1983, Sears was still riding the success train and was over 10 times the size of Walmart. In six short years, things changed. October 1989 ushered in Walmart as the world’s largest retailer, and the rapid decline of Sears continued.

Today, Walmart has revenues over 20 times that of Sears Holdings (which also includes K-Mart). Although Amazon is banging on the door of the retail giant, Walmart still has revenues that are nearly three times that of everything Amazon-related, which goes way beyond just retail. 

Sears’s fall from grace is, of course, not unique. Pan American Airways, one of the largest airlines at one point, became extinct in 1991. The largest food provider in the 1960s was Howard Johnson’s restaurants (remember those orange roofs?), yet there is only one left in the country. Kodak invented the digital camera, yet it was that technology that pretty much killed the camera company. Although Circuit City was in the book Good to Great, the electronics company should have been in the book Good to Gone as it, too, is out of business. 

The list goes on and on. For each company, the specific circumstances of its demise may be different, but the underlying reason is the same: past success led to future failure. 

There are two reasons for this:
1.    Complacency/arrogance
2.    Expertise blindness

Complacency/Arrogance

Many great companies fail because their leaders believe they are too great to be dethroned. I’ve heard executives say, “Well, that may have happened to Sears, but it won’t happen to us. We are different.”

No, you aren’t. In fact, things are more challenging today than they were back when Sears was the top dog.

Success requires checking your ego at the door. Trust me, around the corner, there is someone who is confident he or she can make you irrelevant. And given past evidence, I would put my money on the new entrant, not the incumbent.

In 2008, Jim Keyes, Blockbuster’s CEO, said, “Neither RedBox nor Netflix are even on the radar screen in terms of competition.” Three years later, Blockbuster declared bankruptcy and two years after that, they were defunct. Of course, now Netflix is now the king of video.

Expertise Blindness

The second reason for this failure is something I call expertise blindness. When you know a topic well, it makes it difficult for you to think differently about it. If you spent the last 30 years in your industry, any ideas you develop for the future are most likely extrapolations of the past. That is, they are incremental changes. 

But the world around us is not changing at an incremental pace; it’s evolving at an exponential rate.

Expertise is the enemy of non-incremental innovation.

I find that the fastest way to overcome expertise blindness is by asking: “Who else has solved a similar problem in a different industry?” Anecdotal evidence and numerous studies prove that when people are cut from the same cloth, you only yield incremental improvements. The breakthroughs, in almost every case, are from either connecting to a different discipline, or from multi-disciplinary teams coming together to solve a problem.

Looking externally is critical to staying relevant. Assume that someone will eventually make you irrelevant. Who are the likely and unlikely suspects? Partner with external think tanks, universities, and trend watchers to better understand the shifts taking place from a technological and demographic perspective. Create a corporate radar system that senses and responds to these changes. And look outside your industry or function to see what’s really happening. 

For you to succeed in the future, the pace of change inside your organization needs to be faster than the pace outside.