The Quiet War for Truth

On the evening of January 27, 1986, engineers at Morton Thiokol were arguing against the launch of the space shuttle Challenger.

The forecast was unusually cold. The engineers were concerned about the rubber O-rings that sealed joints in the solid rocket boosters. They had seen damage to those seals after earlier flights. Cold could make the problem worse. They recommended delaying the launch.

Management reconsidered. The recommendation changed. The shuttle was cleared to fly.

The next morning, 73 seconds after liftoff, Challenger broke apart. Seven people died. The presidential commission later found serious failures in how technical warnings reached the people making the decision. The evidence had not simply been absent. It had been debated, reinterpreted, and in some cases kept from officials who needed to hear it.¹

That's what troubles me about this story. The engineers were not confronting a mystery that would only become visible afterward. They had seen warning signs. The organization had seen them before. Yet each flight that ended safely made it easier to treat the next warning as something that could also be managed.

Sociologist Diane Vaughan spent years examining the decisions behind the disaster. She described a process she called normalization of deviance: a departure from an accepted standard gradually becomes part of normal operations, especially when nothing terrible happens immediately.²

The question isn't why everyone failed to notice the warning. Some people noticed it quite clearly.

The question is what made the warning insufficient.

I keep returning to Bayes' theorem because it offers one of the cleanest ways to examine beliefs without pretending that certainty is available on demand.

We begin with an estimate of how plausible an explanation is. We encounter evidence. Then we consider how likely we would have been to see that evidence if the explanation were true, compared with how likely we'd have been to see it if the explanation were false. Our confidence changes accordingly. That's the movement from a prior belief to a posterior belief.³

It's not a command to defend what we already think. Nor is it a requirement to change our minds a little every time something new appears. Strong evidence may overturn a belief. Weak evidence may barely touch it. Sometimes the proper result is to admit that we still don't know.

I encounter failures of this reasoning constantly when working with startup founders.

A founder develops an advertising concept. There's been customer research, some deliberate thinking about the audience, perhaps a few earlier experiments. Then the founder shows it to a friend.

The friend says it doesn't work.

By the next morning the founder wants to change the campaign.

Maybe the friend is right. But why should this particular opinion outweigh everything that came before it? Does the friend buy this kind of product? Do they understand the intended customer? Are they reacting to the message, the design, or the fact that their friend made it?

One observation has arrived, and it feels enormous because it came from someone the founder knows. That doesn't make it representative. Amos Tversky and Daniel Kahneman described our tendency to expect too much from small samples and to overestimate what readily available examples tell us about the world.⁴⁵

The problem isn't listening to a friend. It's granting one friend's reaction an authority the evidence hasn't earned.

A single observation can also be decisive. If a user discovers that the checkout button doesn't work, we don't need a survey of a thousand customers before fixing it. Evidence matters according to what it can establish, not simply how many people supplied it.

That distinction asks for judgment. And sometimes restraint.

Advertising dashboards make uncertainty unusually difficult to tolerate. The numbers refresh constantly. Every change seems to ask for a response.

A campaign launches on Monday. By Tuesday afternoon it has thirty clicks and no purchases. Someone announces that the audience is wrong. Another person wants new creative. The founder has a different explanation: the platform must be broken.

All three explanations may be possible. None has been established by thirty clicks.

If the campaign's actual conversion rate were 2 percent, for example, the probability of getting zero purchases from thirty independent clicks would still be about 55 percent. Zero conversions in that small sample would be ordinary, not extraordinary. The exact calculation depends on the assumptions, but the lesson doesn't: a disappointing observation can be entirely compatible with a campaign that works.

Now imagine the same campaign after several thousand relevant visitors, with persistent losses and no sign of improvement. That's different evidence. Refusing to respond would no longer be patience. It would be avoidance.

Online experimentation researchers have documented how repeatedly checking results and stopping a conventional significance test the moment it looks favorable can increase false-positive findings. Sequential testing methods can address that problem, but they have to be designed for it.⁶

Bayesian reasoning does not require a ceremonial waiting period. New evidence can be evaluated as it arrives. The question is how much it tells us, and what acting before we know will cost.

That last part matters. The standard of evidence for changing a headline is not necessarily the standard for authorizing a shuttle launch. One decision is cheap to reverse. The other isn't.

I've watched founders mistake impatience for decisiveness and constant intervention for management. They aren't always ignoring the numbers. Sometimes they're staring at them so closely that every fluctuation becomes a verdict.

There is nothing indecisive about saying, “We don't know yet.” It may be the most accurate statement in the room.

Now consider the opposite failure.

Imagine a department that begins missing a safety standard. The first incident prompts an investigation. The second gets a warning. After the fifth, everyone has a reasonable explanation. By the twentieth, the deviation appears in the weekly report without attracting much attention.

The staff have learned something true. The department regularly misses that standard. Anyone still predicting perfect compliance would be misreading the evidence.

But another change has taken place.

What was once a failure has become an expectation. Soon the expectation begins to act like permission.

Those are different judgments. One describes what is happening. The other concerns whether it should continue.

We don't need an absolute theory of good and evil to notice when the two are being confused. We need to ask whether the standard was reconsidered deliberately or quietly displaced by habit.

A frequent event is not necessarily a harmless event. A familiar explanation is not necessarily a good explanation. And a practice that has survived twenty trials may still fail on the twenty-first, particularly when the cost of failure is severe.

This was part of what made the Challenger decision so dangerous. Earlier flights had returned. Damage was real, but catastrophe had not followed. Successful outcomes were allowed to compete with evidence of a failure mechanism that remained unresolved.

The organization became more accustomed to the anomaly. It did not necessarily become better at judging it.

There's another way our understanding changes without our noticing. We lose the reference point.

In 1995, fisheries scientist Daniel Pauly described shifting baseline syndrome. Scientists and fishers could take the state of the ocean they first encountered as their picture of a normal ocean. As fish populations declined, each new generation began from a poorer baseline. Loss accumulated even as it became less visible to those living through it.⁷

A later study interviewed 108 fishers from three generations in Mexico's Gulf of California. Older fishers identified far more species that had once been abundant and were now depleted. They remembered productive fishing grounds that younger fishers had never known.⁸

The younger fishers weren't necessarily ignoring the evidence in front of them. They had inherited a different starting point.

We can make sensible adjustments to each new observation and still end up with a diminished picture of what's possible, because the earlier evidence has disappeared.

It happens in organizations, markets, and communities. We compare this quarter with last quarter. We compare the newest policy with the one it replaced. Each comparison may be accurate. But if the reference point keeps moving, we can lose track of the cumulative change.

Our ability to adjust can outpace our ability to judge what those adjustments mean.

Now make the problem harder.

Suppose the reports reaching us have been selected. Not falsified, necessarily. Selected.

A company celebrates its best customer testimonials while excluding people who stopped using the product. An institution reports improvement against a new benchmark without showing the old one. A leader receives a dozen reassuring summaries, all based on the same underlying source.

Every fact in those reports might be accurate. Together, they can still produce a false impression.

Bayesian reasoning depends on assumptions about how evidence is generated. If selected observations are treated as representative, or repeated versions of one report are mistaken for independent confirmation, confidence can grow for the wrong reasons. Statisticians examine related problems under model misspecification: the calculations may be consistent even when the model connecting observations to reality is wrong.⁹

This doesn't mean every institution is lying. Assuming that would simply replace one prejudice with another.

It means we must investigate the route by which information reaches us. What was counted? What was excluded? Who chose the comparison? What would the evidence look like if our preferred explanation were wrong?

Sometimes the source of the distortion isn't a dishonest report. It's a truthful report nobody was willing to deliver.

This is where the mathematics runs out.

I've long used a simple three-part check when assessing character and decisions: honesty, courage, and kindness. Not a device for sorting people into good and bad. A way of noticing what might be missing when someone's behavior doesn't hold up to examination.

Honesty means more than not lying. It means acknowledging what the evidence says, including when it contradicts something we've defended, a decision we've made, or a person we admire. It means knowing the difference between what we have established and what we'd prefer to be true.

Courage is different. Someone can understand the evidence and still refuse to admit its implications in front of other people.

What if an honest assessment costs you a promotion? What if it means telling the founder the product isn't working? What if you've spent ten years defending an idea and the evidence now tells you to abandon it?

You can understand the answer and refuse to give it.

Kindness brings another person into the calculation. Who bears the cost of the decision? Who isn't represented in the data? Whose experience was dismissed because it complicated an otherwise tidy finding?

Kindness doesn't mean adjusting facts to spare feelings. It can require telling someone something they desperately don't want to hear. But it also requires taking seriously the consequences for people whose interests aren't reflected in the outcome we happen to be measuring.

The three qualities don't always point in the same direction. Telling the truth may cause pain. Defending someone may cost you standing. Concern for a colleague may tempt you to conceal a mistake. That's why I call this a check, not a formula.

Good character doesn't guarantee correct reasoning. A decent person can be disastrously wrong. But character affects which evidence we're willing to face, which questions we permit, and what we do after reaching an uncomfortable conclusion.

So what happens when a person wants to be honest, but the organization has taught them that honesty is dangerous?

Picture a marketing analyst presenting the results of a campaign the founder personally designed.

The founder wants to declare a winner. The analyst has a problem: the results are inconclusive. They've checked the numbers, and there isn't enough evidence to support the claim.

The analyst has seen what happens when people contradict this founder. Meetings turn unpleasant. Assignments disappear. People learn to keep their objections brief.

So the analyst says the results are “promising.” The founder hears confirmation. The campaign receives more money.

No one needed to fabricate a number. The distortion occurred between the evidence and the person making the decision.

This is why psychological safety matters so much to clear thinking. Amy Edmondson defined it as a shared belief that a team is safe for interpersonal risk-taking. Her research with 51 work teams found an association between psychological safety and learning behavior. People were better able to ask questions, acknowledge mistakes, and speak up about problems when doing so wasn't treated as a personal threat.¹⁰

Psychological safety is not an agreement to be pleasant. Nor is it protection from scrutiny or accountability. It is the ability to challenge, admit uncertainty, and report a mistake without being humiliated or punished for bringing the information forward.

Honesty becomes possible to practice openly. Courage becomes less costly. Kindness shows up in how people respond to the colleague who has just made themselves vulnerable by saying something unwelcome.

Simon Sinek describes a useful distinction in The Infinite Game. Drawing on conversations with Navy SEALs, he presents a Performance versus Trust Matrix. A high-performing person who cannot be trusted may be a worse teammate than someone with more modest performance who is consistently reliable. Sinek uses the example to challenge organizations that reward results while overlooking how those results are produced.¹¹

It's a leadership heuristic, not a standardized scientific instrument. But the question it raises is worth asking: what does the highest performer cost everyone else?

A manager may hit every revenue target while teaching the team to conceal bad news. A founder may make fast decisions and still train everyone nearby to agree before entering the room. The scorecard looks excellent. The organization gets worse at learning.

I've used honesty, courage, and kindness as a practical check on this sort of behavior. Performance tells me what someone can accomplish. Those qualities help me ask what happens to the people and the information around them while they accomplish it.

Trust and psychological safety aren't identical. Trust concerns what we expect of other people. Psychological safety concerns whether we believe we can take interpersonal risks within the group. But both affect whether an organization receives the information its decisions depend on.

A team that punishes disagreement doesn't merely have a morale problem. It has an evidence problem.

And the more successful the person at the top appears to be, the harder that problem may become to detect. People report what gets rewarded. They withhold what gets punished. Eventually the leader's confidence is fed by the very silence their behavior has produced.

Months before the Challenger launch, Morton Thiokol engineer Roger Boisjoly documented his concerns about O-ring erosion in a July 31, 1985 memorandum. The warning is preserved in the National Archives.¹²

On the night before the launch, engineers objected again. The objection did not prevail.

The problem wasn't simply a lack of observations. There were observations, disagreements over what they meant, and organizational pressures shaping which interpretation became the decision.

Compare that with the founder who wants to tear up an advertising campaign because a friend disliked it. The stakes could hardly be more different. But both cases ask whether a person can give evidence the weight it deserves rather than the weight assigned by fear, habit, impatience, or authority.

Sometimes that means waiting. Sometimes it means acting immediately. A faulty checkout button and a potentially catastrophic rocket-seal failure do not call for the same tolerance of uncertainty. Nor does a campaign with thirty clicks call for the certainty of a verdict.

A better idea isn't always pleasant. It may ask us to give up a theory, reverse a decision, confront a superior, or acknowledge harm we have become accustomed to seeing.

What if the greatest threat to clear thinking isn't our inability to recognize better ideas, but our unwillingness to accept what recognizing them may demand of us?

Bayes can help us determine how our beliefs should move.

It can't make us move with them.