Ask the Opposite Question

A few weeks ago, I asked an AI model whether well-structured SOPs were a prerequisite for developing AI use cases. It gave a thorough, well-sourced answer. Convincing. Then I posed the opposite: are there cases where developing a use case without an SOP is actually the right call? The model delivered an equally compelling argument — different research, different framing, different conclusions.

Source: Source

That gap told me more than either response alone — including a finding I hadn’t expected: in rapidly evolving use cases, building the SOP collaboratively with AI often makes more sense than requiring one upfront.

This is the sycophancy problem. Stanford found that AI affirms users 49% more often than humans do. A study in Nature reported AI agreeing with up to 100% of illogical medical requests. LLMs are trained to be helpful, and “helpful” has drifted toward “agreeable.”

The fix is straightforward: for any meaningful assertion, ask the model to argue both sides, then synthesize a balanced view. The technique is ancient — dialectical reasoning traces to Socrates, devil’s advocacy is a staple of good decision-making. The difference is that our primary research tool is now structurally inclined to skip the counterpoint.

Four reasons this compounds in value:

1. It sharpens the question. Formulating the counter-assertion forces you to clarify what you’re actually claiming.
2. It surfaces broader evidence. The model pulls research for both sides rather than cherry-picking support for one.
3. It works with the model’s design. Sycophancy cuts both ways — the LLM will argue the counterpoint just as earnestly.
4. It checks personal bias. Individual experiences with AI are powerful but limited data points. Structured counterarguments reveal whether a view is widely supported or merely familiar.

For settled facts, skip it. For emerging, contested questions in a fast-moving space — where nobody sees everything and opinions shift weekly — posing the counter-assertion is a two-minute investment that dramatically improves output quality.

Our anecdotal experiences are powerful but limited data points. The counter-question is how you keep them honest.

Stanford: AI advice sycophancy research · Nature: AI in medical decision-making