Compared to What? The Question That Tells You Whether Your Decision Actually Worked

You change something at work: a sales process, a leadership program, or an incentive scheme. The metric you care about goes up. The instinct is to credit the change. 

But the harder question is: what would have happened if you had not made the change? Maybe the metric would have gone up anyway. Maybe it would have gone up even more. Without that comparison, you cannot know whether the change worked.

This is the focus of a paper I recently published with Johannes Stark and Niels Van Quaquebeke in the Journal of Management: “Controlling the Control Condition.” On the surface, it is about how management researchers design experiments. Underneath, it is about a question every leader faces: compared to what?

Pseudo-control vs. true control

In an experiment, you need a comparison to know whether a change had an effect. That comparison is the control condition. 

A true control condition asks the counterfactual question: what would have happened if A had not happened? It compares A to non-A: the same people, setting, and moment, but without the thing you changed. This is the only comparison that can tell you whether A had any effect at all.

A pseudo-control condition compares A with another option, B. This can answer useful questions such as “Is A better than B?” But it cannot answer the question most researchers and leaders actually want to ask: did A do anything?

A problem in management research 

In our review of 958 experiments published in top-tier management journals between 2021 and 2023, two-thirds used designs that did not allow researchers to estimate the actual treatment effect. In other words, most studies could not answer whether A or B had an effect.

Even more concerning, in 20 percent of those experiments, authors made causal claims their designs did not support. They wrote as if a baseline existed when only a relative contrast did. Our within-study meta-analysis also shows that the choice of control condition can shift the estimated effect size by a factor of two. The same treatment can look twice as powerful, or half as powerful, depending only on what it is compared against.

The same trap in management

Managers face the same problem when evaluating their own decisions. Suppose changing your distribution model coincides with higher sales. That looks like a win. But what if sales would have risen even more without the change because demand was already increasing? Your data cannot rule that out.

Three habits help:

1. Name your comparison before you celebrate. A forecast, last quarter, a sibling unit, or your gut each allows different conclusions.
2. Specify the counterfactual, not just the alternative. “What would have happened without the change?” comes before “Would another change have worked better?”
3. Treat success without a counterfactual as a hypothesis, not a finding. Test it through small internal experiments, staggered rollouts, or A/B tests. 

The discipline behind good experiments and clear strategic decisions is the same: both refuse the comfort of an unstated baseline. Both ask the uncomfortable question:

Compared to what?

Reference: Stark, J., Tröster, C., & Van Quaquebeke, N. (2026). Controlling the Control Condition: A Critical Methodological Review of Control Conditions in Experimental Management Research. Journal of Management. DOI: 10.1177/01492063261424849