Software

Debugging like a doctor

Six questions to ask before settling on a cause.

Gene Kranz wearing a headset at the flight director console during a mission simulation.
Flight director Gene Kranz during a Gemini IV simulation, April 1965. Credit: NASA.

A user reports that the application is broken. You open the logs, spot an error, and start forming a theory. It’s tempting to begin fixing things before you’ve established what happened.

Doctors face a similar problem, and their training gives them a framework for investigating it. They gather a history, examine the patient, consider possible causes, and use evidence to narrow those possibilities. That structure helps them move from a reported symptom toward an explanation of what’s causing it.

We can learn to apply that same discipline to our own systems. One useful starting point is OPQRST, a mnemonic for gathering a symptom’s history. It’s one part of the clinical investigation, and it gives us six things to investigate before we become too attached to an explanation.

OPQRST

OPQRST stands for onset, provocation or palliation, quality, region or radiation, severity, and time. It helps organize questions about symptoms. The answers contribute to an assessment alongside examination and, when needed, further testing.

Here is the basic shape of the questions:

  • Onset: When did it begin? What was happening at the time? Did it start suddenly or gradually?
  • Provocation or palliation: What makes it worse, and what makes it better?
  • Quality: What does it feel like? How would you describe it?
  • Region or radiation: Where is it, and does it spread elsewhere?
  • Severity: How intense is it, and how much does it interfere with what you need to do?
  • Time: Is it constant or intermittent? How has it changed since it began?

Onset and time overlap, but I find the distinction useful: onset asks about the beginning; time asks about the pattern since then.

How the questions work

Imagine a clinician taking a history from someone with a cough. The questions might look like this:

Onset: “When did your cough start? What was happening around that time?”

Provocation: “Does activity make it worse? Have you noticed anything that helps?”

Quality: “Is it dry, or are you coughing anything up?”

Region: “Do you have discomfort associated with the cough? Where do you feel it?”

Severity: “Is it interrupting your sleep or keeping you from your usual activities?”

Time: “Does it come and go? Is it getting better or worse? Is there a time of day when it’s most noticeable?”

The questions give the clinician a more useful account than “I have a cough.” Each answer helps direct the investigation: which possible causes fit, what still needs explaining, and what evidence to gather next. The framework connects the initial report to the work of finding its underlying cause. We can use it to make that same connection between a user’s report and the behavior of our software.

From symptoms to a debugging history

For software, I would translate the six questions this way:

Question What to investigate
Onset The earliest known failure, the last known success, and what changed between them.
Provocation or palliation Inputs, load, configuration, or actions that trigger the problem or make it disappear.
Quality The exact behavior: an error, a timeout, incorrect output, or something else.
Region or related events Which users, components, and dependencies are affected, and what else happens at the same time.
Severity The impact on users: inconvenience, blocked work, lost data, or a wider outage.
Time Frequency, duration, recurrence, and whether the behavior is changing.

“Related events” is my adaptation of the R. It reminds me to look beyond the component where the symptom first appeared. Correlated events are leads to investigate; they don’t establish which event caused the other.

An example: intermittent server errors

Let’s imagine users are reporting HTTP 500 errors. That tells us a request failed on the server, but it gives us little to work with. We’ll build a history using the six questions.

Onset: A user first reported the problem six days ago. We examine the available logs and find errors beginning around 2 p.m. that day. There was no application deployment that day, but the load-balancer settings changed. We note that as a lead.

Provocation: The errors appear more frequently during busy periods. We compare failure rates with traffic levels to check whether the problem becomes more likely under load, rather than merely producing more errors because there are more requests.

Quality: Users see intermittent 500 responses. The failures happen in clusters, with quiet periods between them. We collect the corresponding application errors so we can distinguish requests that fail for different reasons.

Region and related events: Database alerts roughly coincide with the failures. We compare timestamps and, where available, trace individual failed requests through the application to the database.

Severity: Users report that refreshing often lets them continue. The interruption still matters: we need to establish whether failed requests lose work or leave an operation partly completed. A successful retry doesn’t tell us what happened to the first attempt.

Time: The clusters recur at peak times and subside when traffic falls. We check how long they last and whether requests recover on their own or only after an intervention.

We now have a plausible direction for the investigation: something about database access may be failing under load. The load-balancer change gives us another thing to examine. We haven’t ruled out application code; existing code can fail when traffic, data, or configuration changes.

Suppose further investigation finds requests waiting for database connections until they time out. We can test a connection-pool exhaustion hypothesis by examining pool usage, connection wait times, and the errors from those requests. We still need to explain why connections are unavailable—slow queries, connections held too long, or increased concurrency could each send us down a different path.

OPQRST helped us decide what evidence to gather. The evidence still has to support the diagnosis.

Practice asking the questions

Developers get plenty of practice troubleshooting, but that doesn’t mean we have a consistent framework for investigation. We know how to inspect logs, trace execution, and reproduce failures. The harder question is often what to ask next. I think OPQRST gives us a useful structure for gathering evidence before choosing where to dig.

Doctors are trained to work through that uncertainty systematically. We can practice the same habit: gather the history, identify possible causes, and choose checks that help distinguish between them. Applied to our own systems, that framework gives us a path from “it’s broken” toward understanding why.

As an exercise, read a few answers in the Stack Overflow thread “What’s the toughest bug you ever found and fixed?”. Try sorting the evidence in each account under these six headings. Which questions were answered early? Which missing detail kept the investigation going in the wrong direction?

The next time someone says “it’s broken,” try gathering that history before committing to your first theory.

Return to top