Stephen T’s Blog Spot

A blog aimed at issues only data scientists, data analysts, statisticians, evaluators, and researchers care about.

recent posts

about

I spend my time thinking about how the government can use research and evaluation methods to make evidence informed decisions and policy. It can be a struggle! That is why I need your thoughts. Join me, have a read, and make a comment. Its how we add to the evidence base,
  • You Are Not as Sure as You Think

    You Are Not as Sure as You Think

    Try a quick test. Pick ten quantities you cannot know off the top of your head, the length of the Nile, the year a certain invention appeared, the population of a country, and for each give a low and a high value wide enough that you are 90 percent sure the truth falls between them. Not certain, just 90 percent. If you are well calibrated, you should miss about one of the ten. Almost no one manages that: across hundreds of studies, the true value lands inside people’s 90 percent ranges less than half the time, and often far less. We are not as sure as we think. We are much surer.

    This effect has a name, overprecision, and it is the most stubborn member of a family of overconfidence biases. Researchers separate three: overestimating your own performance, overrating yourself relative to others, and overprecision, holding your beliefs with too much certainty. The first two come and go with the situation; overprecision is remarkably robust. A striking version of the classic demonstration shows that the range people give when asked to be 98 percent sure is often barely wider than the range they give when asked to be 50 percent sure, as though the extra certainty cost nothing to claim.

    This matters because our work is built on ranges and probabilities. A cost estimate, a schedule, a risk rating, a projected effect size, a win probability on a bid, each is really a statement about how sure we are. Draw the range too narrow and you plan as if the future is more pinned down than it is, and then you are surprised, repeatedly and expensively, by outcomes you had quietly ruled out. The overrun that supposedly could not happen was usually just outside a range someone drew too tightly.

    The comforting thought is that expertise cures this. It does not. When specialists put ranges on uncertain quantities in their own field, a standard step in formal risk analysis, their ranges are still too narrow, and the tendency is described as pervasive and hard to correct. Knowing more can even make it worse, since confidence tends to rise faster than accuracy, and on genuinely hard questions overprecision is at its strongest.

    Part of the reason awareness alone does not fix it is how we build a range in the first place. We start from a best single guess and add a little to each side, and that adjustment is always too small, the anchoring problem this series has covered. We end up answering an easier question, how confident do I feel, in place of the hard one, how wide is the real spread of possibilities. Being told to widen your intervals barely moves the needle.

    What does work is structure that forces you to face the ways you could be wrong. Rather than drawing a range around your best guess, set the two ends separately: first a low value you are 90 percent sure the truth sits above, then a high value you are 90 percent sure it sits below. That small change reliably widens the interval. Before you commit, run a premortem: assume the estimate turned out badly wrong and explain how, which drags the ignored scenarios into view. And keep score. Forecasters who become genuinely well calibrated, weather forecasters being the standard example, are the ones who make many predictions and get prompt, unambiguous feedback on what happened. Track your ranges against outcomes and, over time, your ninety percent starts to mean ninety percent.

    For a contractor this is not an academic nicety. An overprecise cost range loses money; an overprecise risk assessment misses the event that sinks the project; a confident single number is a promise you cannot keep. An honestly wide range is more useful than a narrow one that is wrong, and in front of a client who has been burned before, it is more credible too.

    So here is my question. When you hand over an estimate or a forecast, is the range wide enough to be right ninety percent of the time, or only wide enough to feel confident?

  • Beyond Averages: Designing for Diverse Outcomes

    Beyond Averages: Designing for Diverse Outcomes

    In the late 1940s, United States Air Force jets were crashing with alarming frequency, and no mechanical fault could be found. Attention turned to the cockpit, which had been designed decades earlier to fit the average pilot. A young researcher named Gilbert Daniels was asked to help update that average; however, he arrived with a different question: how many pilots are actually average? He measured 4,063 of them on ten dimensions and, defining average generously as the middle 30 percent of the range on each, counted how many fell in that band on all ten. The answer was zero. Not one pilot out of more than four thousand was average across the board. A cockpit built for the average pilot fit no one, and the fix was not a better average but adjustable seats and controls that fit the whole range.

    The average is a real number and a treacherous summary. It compresses an entire distribution into a single point, and the moment that distribution is anything other than a tidy symmetric hump, the point can describe nobody and mislead everybody. Daniels found the extreme version; the ordinary versions fill our reports.

    Consider skew. When a distribution has a long tail, the mean and the median separate, and the mean is dragged toward the tail. Report the average income, the average wait time, or the average cost, and a handful of large values pull the number well above what most people actually experience. The word average then overstates the typical case, and a reader who pictures a person in the middle is picturing someone who does not exist. For a skewed quantity the median is usually the more honest one-number summary.

    Even the median cannot rescue a distribution with two humps. If a program helps a younger group substantially and an older group not at all, the average sits in the empty valley between them and describes neither. The average of scalding and freezing is not comfortable. Whenever a single summary lands where few of the actual observations are, it is concealing the very structure that matters.

    There is a subtler trap in acting on averages rather than merely reporting them. Feeding average inputs into a plan does not produce the average outcome, a fact sometimes called the flaw of averages. Sam Savage put it that plans based on average assumptions are wrong on average, and the old joke is the statistician who drowned wading across a river that was, on average, three feet deep. When demand, staffing, or arrival times vary, a plan built on their averages will miss, and usually in the costly direction.

    The same collapse hides inside evaluation. An average treatment effect near zero can mean a program did nothing, or it can mean it helped some people and harmed others who cancel out in the mean. Working on average is not the same as working, and it is a long way from working for everyone, since the average can hide the very subgroup a program is failing. That is a reason to design for heterogeneity in advance; however, it is not license to fish for flattering subgroups after the results are in, which manufactures false patterns of its own.

    The discipline is to refuse to let one number stand in for a distribution. Show the shape, a histogram, the spread, a few key percentiles, not the mean alone. Choose the summary that fits the decision: the median for a typical skewed value, the tail for a risk, the range for a plan. Put variation next to the center every time. And when you act, remember you are acting on a distribution, so stress-test the plan against the spread rather than the midpoint. Daniels’s Air Force did not find a better average; it built a cockpit that fit the range.

    So here is my question. When you report or plan on an average, do you know the shape of the distribution underneath it, and whether the number you are quoting describes anyone at all?

  • Understanding the Surrogate Paradox in Health and Education

    Understanding the Surrogate Paradox in Health and Education

    We rarely measure the thing we actually care about. Durable employment, real learning, long-term health, and safety are slow to arrive, costly to observe, and hard to pin on any one program. So we measure a proxy that is faster and cheaper, a job placement, a test score, a lab value, and we treat movement in the proxy as progress toward the goal. The proxy is usually chosen for a good reason: it tracks the outcome we want. The danger is that a program can move the proxy without moving the outcome, and sometimes move the proxy while pushing the outcome the wrong way.

    The clearest evidence comes from medicine, where the stakes made the lesson unmissable. In the 1980s it was well established that irregular heartbeats after a heart attack predicted sudden cardiac death. Drugs that suppressed those arrhythmias were expected to save lives, and the expectation was so strong that many researchers considered withholding them in a trial unethical. The drugs were approved on the strength of the surrogate and taken by hundreds of thousands of people. When a placebo-controlled trial was finally run, they suppressed the arrhythmias exactly as designed and roughly doubled the death rate. The number improved; the patients did not.

    The reason is subtle and general. A surrogate earns its place because it predicts the outcome; an arrhythmia predicts death. Predicting an outcome, however, is not the same as carrying a treatment’s effect on it. A drug can move the surrogate through one pathway and harm the outcome through another the surrogate never registers. Torcetrapib raised HDL, the so-called good cholesterol, and increased deaths, a real signal of risk and a false guide to treatment. This is a point an earlier post made about prediction and explanation: a variable that forecasts a result is not therefore a lever you can pull to change it.

    The formal requirement makes the difficulty plain. For a surrogate to be trustworthy, the treatment’s effect on it must capture the treatment’s effect on the real outcome, a demanding condition that is rarely checked. And a sharper trap sits beneath it, the surrogate paradox: even when the surrogate and the outcome are strongly and positively correlated, a treatment that improves the surrogate can still worsen the outcome. A tight correlation between two measures does not license acting on one to move the other.

    This is related to the familiar warning that a measure gamed into a target stops being a good measure, but it is not the same problem. A surrogate can fail when no one games anything, because the causal chain from proxy to outcome does not carry the whole of the intervention’s effect. Gaming is one way the link breaks; a treatment acting through unintended channels is another, and harder to see coming.

    None of this is confined to medicine; it is the daily condition of program evaluation. The outcomes that justify our work are slow and costly, so we lean on proxies: placements, test scores, certificates, attendance, short-term output counts. Each is chosen because it correlates with the goal, and each can move without the goal following. A training program can lift placement rates by steering people into jobs that do not last; a school can raise scores by narrowing teaching to the test. The proxy improves, the report looks strong, and the mission is no better off.

    The discipline is to keep the real outcome in view even when you cannot measure it directly. Name it explicitly, and treat the surrogate as a stated hypothesis about how you will reach it rather than proof that you have. Ask whether moving this proxy has actually moved the outcome before, for this kind of intervention. Look hard for the pathways by which your program could improve the proxy without helping, or while harming, the goal. And when the stakes are high, measure at least some true outcomes directly, on a subset or with longer follow-up, rather than trusting the proxy on faith.

    A better number is easy to produce and easy to celebrate. Whether it means a better outcome is a separate question, and it is the one that finally matters.

    So here is my question. For the proxies your program reports, can you point to evidence that moving them actually moves the outcome you care about, or are you trusting that the link will hold?

  • The Pitfalls of Retrospective Data Collection

    The Pitfalls of Retrospective Data Collection

    A large share of the data we collect asks people to remember. How many times did you see a doctor last year? How much did you drink last month? When did the symptoms start? Since the program ended, have you found work? We treat the answers as records of what happened. They are not records. They are reconstructions, and memory reconstructs with a systematic tilt.

    The mind does not store events like a tape and play them back on request. It rebuilds them from fragments, cues, and present beliefs, and that rebuilding introduces error. The error is not random noise that averages out; it follows predictable patterns that push estimates in particular directions. Knowing the patterns separates a usable retrospective measure from a misleading one.

    Several patterns do most of the damage. Recall decay is the simplest: the further back you ask, the more is forgotten, so long recall periods undercount events, especially minor ones. Telescoping is sneakier: people misdate events, usually pulling them forward so they feel more recent, which inflates the count inside a bounded window like the past twelve months, because older events get dragged into it. Rounding and heaping pile responses onto salient numbers, zero, five, ten, about twice a week, distorting the distribution and its tails. And salience skews what survives at all, since vivid or emotional events are remembered while routine ones vanish, so rare dramatic behaviors are overcounted and frequent mundane ones undercounted.

    Then there is the pattern that turns error into bias. Current state colors memory: people who feel unwell now recall more past symptoms, and people who believe something helped recall their earlier situation as worse than it was. This is harmless as long as it operates equally across the groups you compare; it becomes dangerous the moment it does not. In a case-control study, people who have the disease search their memory harder for causes than healthy controls do, so the cases recall more exposure and an association appears that may not be real. This is differential recall, and it does not merely add noise; it manufactures or erases the very difference you are trying to measure.

    Evaluation has its own popular version. Asking people at the end of a program to rate where they were before it, thinking back, how confident were you at the start, is cheap and sidesteps some real problems with baseline surveys. However, it hands the respondent’s current beliefs about the program a direct channel into the baseline measure. If they think the program worked, they will tend to remember a lower starting point, and the design will produce an effect whether or not one occurred. A retrospective baseline is the easiest way to measure a program’s reputation and call it impact.

    The repair follows the diagnosis. Prefer contemporaneous measurement whenever you can afford it: capture data at baseline, use diaries or real-time prompts, or draw on records instead of memory. When you must ask retrospectively, shorten and bound the recall window, and anchor dating with calendars or landmark events to blunt telescoping. Offer specific options to recognize rather than forcing people to generate answers from nothing, and validate self-report against records where any exist. Above all, match recall conditions across your groups, so that any recall error is similar on both sides rather than concentrated in the group with the most reason to reconstruct the past.

    None of this is academic for program evaluation, which runs on retrospective self-report: exit surveys, follow-up interviews, questions that begin with since the program. The convenience is genuine; however, the recall structure quietly favors finding an effect, because the people most invested in a program are often the ones whose memory of before has been most reshaped by after. Treating memory as a reconstruction, whose errors are patterned and can differ by group, is the difference between measuring what a program did and measuring what people now believe about it.

    So here is my question. When your data depend on what people remember, do you account for the fact that memory is reconstructed and skewed, and that the skew may be larger in exactly the group you expect to benefit?

  • Understanding the Limitations of Member Checking

    Understanding the Limitations of Member Checking

    A common and well-meant step in qualitative research is to take your findings back to the people who gave you the data and ask, does this ring true? It feels like the ultimate check. Who better to confirm an interpretation than the people it is about? This is member checking, also called respondent or participant validation, and it is often treated as the gold standard of qualitative rigor. Return your account to participants, and if they endorse it, it must be accurate, and if they reject it, it must be wrong. The instinct is good; however, the logic in its strong form does not hold. Agreement and accuracy are not the same thing, and neither are disagreement and error.

    Consider why a participant might agree with an account that is not faithful to the data. They might agree because it is easier than arguing, because the researcher is an authority, because they do not want to contradict a polished document, or simply because they are being polite. Agreement can be social rather than epistemic. And a study that only checks back with the participants likely to nod along is not validating anything; it is cherry-picking its own confirmation.

    Now consider why a participant might reject an account that is accurate. Good qualitative analysis usually produces interpretations that go beyond what any single person would say about themselves. It connects themes across many accounts, names a pattern no one participant lived as a whole, and situates individual stories in a larger frame. A participant may not recognize their own words in that higher-order reading, and non-recognition is not the same as the interpretation being wrong; the analyst’s job was never to simply transcribe each person’s self-understanding. Participants also sometimes reject a finding precisely because it is accurate and unflattering, and people change, so by the time you check back they may reinterpret the experience they first described.

    Underneath all of this is a single confusion. Member checking quietly merges two different questions: is this account faithful to the evidence, and does this participant endorse it? Those can come apart in both directions, which is why endorsement cannot serve as a truth test. It is the same lesson an earlier post drew about triangulation, that agreement across sources is not automatically a stamp of validity: a thing can feel confirmed without being correct.

    None of this means you should stop. It means you should stop treating it as a verdict and treat it as what it is: a source of data and a relationship, not a vote. A participant’s reaction to your interpretation is itself rich material worth analyzing, and disagreement especially is informative. When someone pushes back on a theme supported by many others, the theme does not fall, but the pushback is a case worth understanding, the qualitative equivalent of hunting for the exception that tests your account. Member checking also catches plain factual errors, and it honors participants by giving them a say in how they are represented, which matters on its own ethical terms.

    Doing it well means being clear about what you are asking. Are you checking facts, or seeking reactions? Treat the responses as data to be interpreted, not ballots to be counted. Expect disagreement and welcome it. And never let a participant’s endorsement or objection quietly override the weight of the evidence. What you should not do is write that member checking confirmed the findings, as though that sentence settled the matter.

    In evaluation this lands close to home. We routinely take findings back to program staff, participants, and stakeholders, and the pressure is real to read agreement as validation and disagreement as a problem to manage. Both are errors. Those reactions are invaluable, as data, as ethics, and as a route to use, but a program’s staff endorsing a flattering finding does not make it true, and their rejecting a critical one does not make it false. Keep the reaction and the evidence in separate columns.

    So here is my question. When you take your findings back to the people they are about, are you testing whether the account is true, or learning how they respond to it, and do you keep those two firmly apart?

  • Identifying Bias in Linked Data Sets

    Identifying Bias in Linked Data Sets

    A great deal of modern research and evaluation runs on linked data. We connect a program’s enrollment file to earnings records, a survey to health claims, a benefits roster to death records, and suddenly we can follow people across systems we could never afford to track ourselves. The catch is in the joining. When two datasets do not share a clean, reliable identifier, and usually they do not, we match on names, dates of birth, addresses, and partial identifiers. That matching is never perfect, and its imperfections are not random.

    Linkage produces two kinds of error, pulling in opposite directions. A false match joins two records that belong to different people, when common names, shared birthdates, or transcription errors make strangers look identical. A missed match fails to join two records that belong to the same person, when a name changed at marriage, an address moved, a field was blank, or a digit was mistyped. One invents a person who is not there; the other loses a person who is.

    The heart of the problem is that these errors are not spread evenly. Whether a record links correctly depends on the quality and stability of its identifiers, and those depend on who the person is. People who move frequently, change their names, hold common names, or appear in lower-quality records are systematically harder to link, and those traits travel with age, income, housing stability, immigration status, and race. This is not hypothetical: in large administrative systems, missed matches have been shown to fall hardest on ethnic minorities, people in deprived areas, foreign nationals, and those with no fixed address. The people who fail to link are not a random slice; they are disproportionately the mobile and the marginalized, often exactly the people an equity-focused evaluation most needs to see.

    That turns the linked dataset into a biased sample, tilted toward the easy to link, and the bias enters your results in two ways. If linkage is related to the outcome, whether someone is employed, or has died, or was readmitted, then the linked sample misrepresents that outcome, and you can manufacture or erase an effect through nothing more than who matched. And false matches attach the wrong record to a person, misclassifying an outcome in ways a clean-looking analysis never reveals. Because match rates differ by group, comparisons across groups are contaminated by differential linkage rather than real differences, which is how a study of disparities can end up measuring its own linkage.

    What makes this easy to miss is that the linked file arrives looking authoritative, one tidy row per person, often assembled by someone other than the analyst. The match rate may be reported as a single reassuring number, but a high overall rate can hide large gaps by subgroup, and the errors are baked in before analysis begins. Standard errors computed on linked data usually assume the linkage was perfect, so they come out falsely small. It is a fitness-for-use problem sitting upstream of everything you do.

    The discipline is to treat linkage as part of how the data were made, not a preprocessing footnote. Ask how records were matched, deterministic or probabilistic, on which fields, at what threshold. Examine match rates by subgroup, not just overall, and treat a differential rate as a bias to investigate rather than a nuisance to note. Where you can, check the linkage against a gold-standard sample and estimate which way the error pushes. And report honestly that your analytic sample is the linked sample, which may differ from the population in patterned ways.

    This is the nonresponse question from earlier in this series, moved to a new step. There the issue was who failed to answer, and whether it related to what you measured; here it is who failed to link, and whether it relates to what you measure. Same logic, same danger, further upstream. An impact estimate from linked data inherits every bias of the join, whether or not anyone looked.

    So here is my question. When you analyze linked data, do you know who did not link, and whether the reason they are missing is related to the very thing you are trying to estimate?

  • Predictive vs Explanatory Modeling: Key Differences

    Predictive vs Explanatory Modeling: Key Differences

    Two questions sound almost the same and are not. One is: what will happen? The other is: why does it happen, and what should we change? A model can be excellent at the first and useless at the second, and confusing them is one of the most common and costly mistakes in applied analysis. It is getting more common as predictive tools spread into places that used to rely on explanation.

    Galit Shmueli drew the line cleanly in a 2010 paper whose title is the whole question: to explain, or to predict? Explanatory modeling tests a theory about why something happens, so it cares about getting the underlying structure and the coefficients right. Predictive modeling forecasts new cases, so it cares about one thing, accuracy on data the model has not seen. These are different goals, and they lead to different decisions at every step, from variable selection to model complexity to how you judge success. Start from the same data with different goals and you end up with different models.

    The reasons they diverge are worth seeing concretely. A predictive model will happily use a variable that is a downstream consequence of the outcome, or a proxy, or a common effect of two other things, as long as it sharpens the forecast. An explanatory model has to exclude exactly those, because putting them in corrupts the estimate of the cause you care about, a lesson this series has drawn before about controlling for the wrong variable. A superb predictor can be a terrible thing to adjust for.

    Bias behaves differently too. Prediction will accept a little bias in exchange for lower variance, since a slightly biased model often forecasts new data more accurately, which is why methods that deliberately shrink estimates toward zero are standard predictive tools. Explanation wants an unbiased estimate of the parameter even at the cost of precision, because a biased coefficient is a wrong answer to how much something matters. The best predictive model is often intentionally biased in ways an explanatory one could not tolerate.

    And success is judged differently. A predictive model is judged out of sample, on how well it forecasts cases it never saw, the discipline this series covered under overfitting. An explanatory model cannot be judged that way, because no accuracy score can tell you whether the assumptions that license a causal reading hold. A model can predict beautifully and carry a completely wrong story about why.

    That is where the real damage happens. The most common mistake is to build a model for prediction and then read its coefficients as if they explained. A variable that ranks high in predictive importance is not therefore a cause, and changing it will not necessarily change the outcome. Predictive importance answers whether something helps forecast, not whether it drives the result. Acting on a strong predictor as though it were a lever is how you try to change the temperature by adjusting the thermometer.

    The repair is unglamorous: decide the goal before you model, and say it out loud. If you need a forecast, optimize and validate for out-of-sample accuracy, and do not interpret the coefficients as causes. If you need to explain, or to decide what to change, build for identification, choose variables by their causal role rather than their predictive punch, and judge the model by whether its assumptions hold. Only the explanatory reading licenses the sentence change this to move that. Prediction tells you where to look; explanation tells you what to do.

    This is not abstract for anyone in government. Agencies increasingly run predictive models, risk scores, early-warning flags, targeting tools, and then face pressure to explain them and act on their inputs. A model that forecasts who is likely to drop out or be readmitted is answering a prediction question, and its most important features are not a menu of policy levers. Treating them as levers, or as an explanation of why a group scores high, is a category error with real consequences for who gets helped.

    So here is my question for the group. Before you build or trust a model, do you decide whether you are trying to predict or to explain, and do you resist reading a forecast as if it told you what to change?

  • Aligning Time to Avoid Immortal Time Bias

    Aligning Time to Avoid Immortal Time Bias

    Observational studies keep discovering that people who did a certain thing live longer. Patients who filled their prescriptions outlive those who did not. Heart transplant recipients outlive those on the waiting list. Oscar winners outlive the nominees who lost. Some of these gaps are real. Many are an illusion produced by a single, subtle flaw in how the groups were defined, and the flaw has a name: immortal time bias.

    The idea is easier to feel than to state. To end up in the group that filled a prescription, you had to live long enough to fill it. To count as a transplant recipient, you had to survive on the list until an organ arrived. To win an award, you had to be alive on the night it was given. The stretch of time between the start of follow-up and the moment you qualified is time during which, by the very definition of the group, you could not have had the event: if you had died, you would not be a filler, a recipient, or a winner, you would be in the other group. That guaranteed, event-free stretch is the immortal time.

    The bias appears when that immortal time gets handed to the treated group. In some studies it is counted as time under treatment, even though treatment had not started. In others, people who die during it are quietly moved into the comparison group or dropped. Either way, the treated group is credited with a block of survival no treatment produced, because that survival was a precondition for being in the group at all. The comparison is tilted before the treatment can do anything, and it tilts in a predictable direction: toward making the treatment look protective.

    The reason it is so easy to commit is that nothing about it looks like an error. The groups are real, the people genuinely received the treatment, the data are accurate, and the analysis is a standard survival model. The flaw is not in the measurement or the sample; it is in the alignment of time. The clock for the treated group effectively starts late, or their early, guaranteed survival is counted as if the treatment had earned it. That is why it slips past researchers careful about everything else.

    The most famous illustration is the finding that Oscar winners live years longer than other nominees, once offered as evidence that status and esteem lengthen life. But a winner had to survive to the ceremony, and to every later ceremony that made them a winner rather than a hopeful. When the advantage was reanalyzed with the time properly aligned, so that no one was credited for years they simply had to be alive to accumulate, most of the effect dissolved. The prize had not added the years; the years had been a requirement for the prize.

    The repair is to align the clock. Everyone’s follow-up must start at the same well-defined moment, and treatment must be handled as something that happens at a point in time rather than a label applied to a whole history in hindsight. In practice that means letting a person contribute untreated time until the treatment begins and treated time only afterward, or choosing a landmark moment and classifying everyone by their status then, discarding the immortal stretch. Most generally, it means specifying the study as the randomized trial you wish you could run, with one clear moment of eligibility and assignment, the target trial emulation an earlier post described, which was built precisely to make immortal time impossible.

    This is not only a clinical problem. Any evaluation built on administrative data where participation takes time to accrue is exposed. Compare people who completed a training program with those who did not, and remember that completers had to stay enrolled long enough to finish. The time they spent getting there can be silently credited to the program, so the apparent effect of completing is partly the effect of not having left early. Whenever the exposure is defined by something that takes time, ask whether the treated group was quietly granted immortal time.

    So here is my question. When your treated group is defined by something that took time to happen, have you checked that you are not crediting the treatment for the survival it required in the first place?

  • Transforming Appraisal: From Scores to Structured Judgments

    Transforming Appraisal: From Scores to Structured Judgments

    Its companion post argued that a single quality score is unreliable and often invalid, and that bias is not one property of a study but something specific to a mechanism and an outcome. That is the diagnosis. The repair is not a better scale. It is a different way of working, one that treats appraisal as transparent, structured judgment rather than a number.

    Start by refusing to sum. The modern tools are built this way on purpose: the Cochrane risk-of-bias tool for randomized trials, and ROBINS-I for non-randomized studies, ask for a separate judgment in each bias domain, how participants were assigned, whether groups stayed comparable, how missing data were handled, how outcomes were measured, whether results were selectively reported, and keep those judgments apart instead of collapsing them into one figure. A study becomes a profile across domains, not a rank, which removes the arbitrary weighting that made summary scores indefensible.

    Not every domain carries equal evidence, so weight your attention accordingly. A few items have been shown empirically to predict distorted effects, concealment of allocation and blinding chief among them, and the meta-epidemiological studies tell you not just that they matter but when. Take those seriously. Be honest that other items on the checklist rest on expert consensus rather than demonstrated bias, and do not give a consensus item the same evidentiary weight as one with a track record.

    Because the same flaw bites differently depending on what is measured, appraise risk of bias for each outcome, not once for the whole study. A trial can be at low risk for an objective outcome like mortality and higher risk for a subjective one rated by unblinded assessors. Rating the study once, and stamping that rating on every result it reports, throws away the very distinction that matters most.

    Since two reviewers will disagree, manage it the way you would manage any unreliable measurement. Use two independent appraisers, write down decision rules before you start, pilot the tool on a few studies to calibrate, and record the reasoning behind each judgment so a reader can check it. The goal is not to eliminate judgment, which is impossible, but to make it transparent and reproducible.

    Here is the move that matters most, and it is one this series has made before in another setting. Do not use risk of bias as a gate that admits or excludes studies, and do not fold it into a weight. Use it as a sensitivity analysis: pool the evidence, then ask what happens when you restrict to the studies at lowest risk of bias. If the effect holds, the conclusion is robust. If it shrinks toward nothing once the most biased studies are removed, the bias was doing the work, and that is itself the finding. Risk of bias earns its keep as a question you ask of the result, not a filter you apply to the inputs.

    At the level of the whole body of evidence, this is what GRADE formalizes, rating certainty for each outcome by combining risk of bias with consistency, directness, precision, and the threat of publication bias. Used well, it communicates how much to trust a result. Used carelessly, its tidy structure can lend an unearned air of objectivity to a chain of judgment calls, so the discipline is to show the reasoning, not just the rating.

    The through-line of both posts is the same. The aim of appraisal is not a number that ranks studies but a transparent, domain-by-domain, outcome-specific judgment you can defend and stress-test. Grade the bias, for the outcome, out loud, and let the conclusion prove it can survive the least biased evidence.

    So here is my question. In your reviews, does risk of bias end up as a score that ranks studies, or as a question you put to the result to see whether it holds?

  • The Flaws of Quality Scores in Research Synthesis

    The Flaws of Quality Scores in Research Synthesis

    Every systematic review rests on a step that rarely gets questioned: appraising the quality, or the risk of bias, of each study it includes. To do that, reviewers reach for an appraisal tool, a checklist or a scale. What almost no one asks is whether the tool itself is any good. An appraisal tool is a measurement instrument, and when you hold it to the same standards it imposes on the studies it judges, reliability and validity, it often falls short.

    The oldest and most seductive form of appraisal is the summary quality score: run down a checklist, add up the points, and rank the studies by the total. That number feels objective and is anything but. It is a composite index, and it carries every problem a composite index carries, which items you include, how you weight them, how you combine them. The demonstration is now a classic: Jüni and colleagues scored the same trials with twenty-five different published quality scales and found that the choice of scale could change the conclusion of the meta-analysis, a trial ranked high by one scale ranked low by another. Sander Greenland had put it bluntly years earlier, that quality scores are useless and potentially misleading. The score launders a pile of judgment calls into one authoritative-looking figure, which is why the best modern tools refuse to produce one.

    Even setting the score aside, appraisal is less reliable than its users like to think. Give the same study to two capable reviewers and they often disagree. The Newcastle-Ottawa Scale, the most widely used tool for observational studies, is the cautionary example: it has repeatedly shown poor agreement between reviewers, and it was never formally validated. That means a risk-of-bias rating is not a fixed fact about a study. It carries measurement error of its own, and a rating from a single reviewer is a shaky foundation to build a conclusion on.

    There is a construct problem underneath the reliability problem. The idea of quality was always fuzzy, blending methodological rigor, completeness of reporting, and relevance, three different things. And an appraisal tool can only assess what the authors actually wrote down, so it partly measures how well a study was reported, not how well it was conducted. A careful study described carelessly gets marked down; a weak study written up smoothly can pass. The tool may be grading the prose as much as the science.

    The deepest issue is that bias is not a single property of a study at all. It is specific to a mechanism and specific to an outcome. The clearest evidence comes from trials: inadequate concealment of who was assigned to which group, and lack of blinding, exaggerate treatment effects substantially for subjective outcomes but barely at all for objective ones like death. The same design feature is a serious flaw for one outcome and almost irrelevant for another in the very same study. This should sound familiar: just as a survey does not have one bias but a different bias for each estimate, a study does not have one quality. So the statement that a study is high quality is not well formed. The honest unit of appraisal is a specific bias, for a specific result.

    The tool that grades the evidence deserves the same skepticism we bring to the evidence itself, and the single quality number is the least trustworthy thing the whole process produces. None of this means appraisal is hopeless. It means it has to be done differently, by domain, by outcome, and with its judgments made transparent rather than buried in a score. Its companion post takes up exactly that: if a single quality score fails, how should we appraise and use risk of bias instead?

    So here is my question. When a review tells you a study is high or low quality, do you ask which bias, for which outcome, and how reliably that judgment was reached?

  • Beyond Response Rate: Measuring Survey Bias Correctly

    Beyond Response Rate: Measuring Survey Bias Correctly

    The first question people ask about a survey is almost always the response rate. Sponsors set targets for it, reviewers judge studies by it, and a low one is often treated as a fatal flaw. The instinct feels unimpeachable: surely the more people who answer, the closer you are to the truth. But the response rate is a surprisingly weak guide to whether a survey is biased, and treating it as the grade leads people to trust the wrong surveys and dismiss the right ones.

    Start with what actually produces nonresponse bias. It is not the number of nonrespondents on its own. It is the combination of two things: how large the nonresponse is, and how different the people who did not answer are from the people who did, on the specific quantity you are estimating. Roughly, the bias is the nonresponse rate multiplied by that difference. So if the people you failed to reach are, on the variable in question, just like the people you reached, a high nonresponse rate costs you almost nothing; and if the missing people are systematically different, even a modest nonresponse rate can produce a large bias.

    Because the response rate captures only one of those two factors, and misses the one that often matters more, it cannot by itself tell you how biased a survey is. Robert Groves and Emilia Peytcheva assembled fifty-nine studies in which the true values were known, so the real nonresponse bias could be measured, and they found very little correlation between the response rate and the bias. Surveys with low response rates were sometimes nearly unbiased; surveys with high response rates were sometimes badly biased. The number everyone treats as the quality grade barely tracked the quality, and pushing a response rate higher did not reliably shrink the bias.

    There is a subtler point underneath. Nonresponse bias is not a property of a survey; it is a property of each estimate within the survey. The same survey can be nearly unbiased for one question and badly biased for another, because who is missing matters differently depending on what you measure. A health survey that quietly loses the sickest people can be badly biased on health status while remaining almost unbiased on commute times. So a claim that the survey as a whole has low bias is not even well formed; bias lives at the level of the number, not the instrument.

    If the response rate is not the answer, what is? The question that matters is whether nonresponse is related to what you are measuring, which is the missing-data logic from earlier in this series, the difference between data missing at random and data missing for reasons tied to the outcome. In practice that means comparing respondents to known population benchmarks, using whatever frame or administrative information you have to see how respondents and nonrespondents differ, and where you can, following up a subsample of nonrespondents to measure the difference directly rather than assume it away. And it means reporting the risk of bias estimate by estimate, not as one response rate stamped on the whole survey.

    None of this makes the response rate worthless. A higher rate shrinks the multiplier, so it buys some protection against the worst case, and a very low rate leaves more room for trouble if the difference turns out to be large. The honest framing is that the response rate is a bound on the risk, not a measurement of the bias: it tells you how much room there is for a problem, not whether you have one. So clearing a federal response-rate threshold is worth doing, but it is not the same as solving the bias problem, and missing it does not automatically make a survey worthless.

    So here is my question for the group. When you judge a survey, do you stop at the response rate, or do you ask whether the people who are missing differ on the very thing you are trying to measure?

  • The Impact of Anchoring on Estimates and Decisions

    The Impact of Anchoring on Estimates and Decisions

    Someone says a number out loud, a budget figure, a timeline, a rough guess, and from that instant your own estimate is quietly bent toward it. Not because the number was correct. Often it was arbitrary, sometimes obviously so, and you knew it. Yet it moved you anyway. This is anchoring, and it is one of the most reliable and least escapable biases in all of human judgment.

    The classic demonstration is almost unfair in how well it works. In 1974, Amos Tversky and Daniel Kahneman sat people in front of a wheel of fortune, numbered 0 to 100 but secretly rigged to stop on either 10 or 65. After the wheel landed, they asked each person whether the percentage of African countries in the United Nations was higher or lower than that number, and then asked for their best estimate. The people who saw 10 gave a median estimate of 25 percent. The people who saw 65 gave a median estimate of 45 percent. A number pulled from a spinning wheel, visibly random and plainly irrelevant to African geography, dragged the answers twenty points apart.

    The mechanism is a shortcut we all use. Faced with an uncertain quantity, we do not build an answer from nothing. We grab whatever value is nearby and adjust away from it, and the adjustment is almost always too small. We stop before we have moved far enough, and we come to rest closer to the anchor than we should. Any number in the vicinity becomes the starting line, whether or not it has any right to be.

    Here is the part that should worry us. Anchoring does not go away when you know about it. It survives explicit warnings, it survives knowing the anchor is random, and it survives paying people to be accurate; Tversky and Kahneman found that offering rewards for correct answers did not shrink the effect. Later work showed that even absurd anchors work, that asking whether Einstein first visited the United States before the year 1215 anchors people about as well as a plausible date. Expertise is thin protection, because experts anchor on irrelevant numbers much like everyone else. You cannot simply resolve not to be anchored.

    In our line of work the anchors are everywhere, and they arrive early. A client mentions an off-hand budget, and every cost estimate that follows orbits it. A previous contract’s price becomes the gravitational center of the new one. In a negotiation, the opening offer sets the terms of the whole exchange, which is why a first number, even an aggressive one, is so powerful. And when we lean on expert judgment, asking a panel to estimate an effect size, a prevalence, a risk, showing them a preliminary figure quietly contaminates the judgment we came to collect.

    Because willpower does not work, the defenses have to be structural. Collect estimates independently before anyone says a number aloud, so no single figure anchors the group. When an anchor is unavoidable, make it a defensible one: replace the arbitrary starting point with real reference data, the distribution of similar past projects, which is the outside view this series has praised before. Force yourself to consider the opposite, generating reasons the true value could be far higher or far lower before you settle. And ask for a range built from evidence before anyone commits to a point.

    The through-line is simple. The first number in the room is doing work, whether or not it has earned the right to. Half the discipline is noticing that it is happening at all. The other half is arranging the process so the anchor that shapes the answer is a piece of evidence you chose on purpose, not a figure that happened to be said first.

    So here is my question. When your team lands on an estimate, do you know what number it started from, and whether that starting point was evidence or an accident?

  • The Ranking Is a Choice

    The Ranking Is a Choice

    A single number that ranks things carries enormous rhetorical power. States ordered by vulnerability, hospitals by quality, countries by competitiveness, programs by performance: the ranking looks like a measurement, objective and settled. But a composite index is not a measurement in the way a thermometer reading is. It is a construction, assembled from many separate pieces through a chain of choices, and those choices, as much as the underlying reality, decide who ends up on top.

    Building an index means making at least four consequential decisions, each defensible and each arbitrary at the margin. First, which indicators to include, and which to leave out; the index measures only what you chose to put in it, so an omitted dimension simply vanishes from the ranking. Second, how to normalize them, since they arrive in different units and must be made comparable, and rescaling by z-scores, by a minimum-to-maximum stretch, or by ranks can give different results, especially with outliers. Third, how to weight them; equal weights feel neutral but are a strong assumption, and any set of weights is a statement about what matters more, usually chosen rather than derived. Fourth, how to aggregate; adding the pieces lets a high score compensate for a low one, while multiplying does not, so a weak spot cannot be bought back, and that choice alone can change the order.

    Because every one of these choices moves the ranking, the order is partly an artifact of the recipe. Two competent analysts, each making entirely defensible choices, can produce different rankings from the very same data. Formal studies of index construction show exactly this: as you vary the weights and methods within reasonable bounds, a unit’s rank can swing, sometimes dramatically. Andrea Saltelli, one of the field’s authorities, warned that an index built without a sensitivity analysis can be made to tell almost any story. A rank presented as a fact is often a choice presented as a fact.

    The real hazard is what this does to accountability. An index launders subjective judgments into an objective-looking number. The weighting decision, the most value-laden step of all, disappears into a formula, and the result emerges wearing the authority of arithmetic. People then treat the order as discovered rather than built, and they make real decisions, funding, targeting, oversight, on differences between adjacent ranks that fall well within the noise of the method. Position fourteen beats position fifteen, and a resource follows, when a slightly different but equally reasonable recipe would have swapped them.

    None of this means indices are useless; it means they should be treated as models, not measurements. Make every choice explicit: what went in, how it was scaled, weighted, and combined, and why. Then run a sensitivity analysis, varying those choices to see how stable each position is, and report the instability instead of hiding it. Distinguish the gaps that survive reasonable variation from the ranks that reshuffle at a touch. And resist false precision: an index may be serviceable for sorting units into broad tiers, high, medium, and low, while being close to meaningless for ranking one position against the next.

    For those of us working with federal programs, this is not abstract. Vulnerability indices, deprivation measures, risk scores, and performance rankings are used to direct money and to hold programs to account. When a funding formula or an oversight decision rests on an index rank, the construction choices behind that rank are not technical footnotes. They are policy choices in disguise, and they deserve the same scrutiny we would give any other policy choice.

    So here is my question for the group. When you use a ranking, do you ask how it was built and how much the order would move under different reasonable choices, or do you treat the number as the thing itself?

  • Understanding Uncertainty in Data Visualization

    Understanding Uncertainty in Data Visualization

    Picture a simple bar chart. Two bars, one taller than the other. Your eye settles the matter in an instant: this group is higher than that one. But the chart has withheld the single fact you need in order to trust that reading, which is how much each bar could have come out differently by chance. Strip the uncertainty out of a picture and you have not made the result simpler. You have made it look more certain than it is.

    This is the quiet failure of most default charts. A bar is a single number. A line connects single numbers. A dashboard tile shows one figure in bold. Nothing on the page signals that each of those numbers is one draw from a distribution, that a rerun of the same study would have landed somewhere nearby but not identical. The crisp, confident geometry of a clean chart is a claim, and often an overclaim, made by omission.

    The consequences are concrete. Two bars that differ by a sliver can be statistically indistinguishable, their intervals heavily overlapping, and yet the picture shouts that one is bigger. A line that drifts upward can be indistinguishable from flat once you allow for noise, and yet a trend line reads as momentum. The viewer walks away with a strong conclusion the data does not support, and no one had to lie with a number to produce it. The chart did the overstating on its own, by leaving the uncertainty out.

    The remedy is to put the uncertainty back on the page. Add error bars or confidence intervals to the bars and points. Draw a shaded band around a fitted line or a forecast. Where you can, show the spread of the underlying data and not only the summary, because a mean conceals the distribution it came from. The aim is for the eye to take in the estimate and its uncertainty at the same time, so that the strength of the visual impression matches the strength of the evidence, no more and no less.

    Showing uncertainty well has its own traps, and two are worth naming. First, an error bar is meaningless until you say what it is. A standard deviation, a standard error, and a confidence interval describe three different things with three different widths, and readers routinely confuse them, so label them every time. Second, you cannot judge a difference by whether two intervals overlap. Overlapping confidence intervals do not prove there is no real difference, and separated ones do not prove there is one; the eyeball test is unreliable. When the difference between two groups is the point, the honest move is to show the uncertainty of the difference itself, not to leave readers comparing two bars by sight.

    Step back and this is simply the communication end of everything the series has said about uncertainty. A p-value, a confidence interval, an effect size with its range, these are all efforts to be honest about how much we do not know. A chart that shows only the point estimate throws that honesty away at the last step, exactly when the result reaches the people who will act on it. The picture is often the only part of a study a decision-maker ever sees, which makes it the most important place to tell the truth about what is uncertain.

    For those of us who build dashboards, briefings, and one-page summaries, the pressure to present a clean and confident picture is real, and it runs the wrong way. A bar chart of outcomes by site, with nothing to signal uncertainty, quietly invites a manager to rank the sites and act on gaps that may be noise. Adding the intervals is not hedging or clutter. It is the difference between informing a decision and inviting a mistake.

    So here is my question. When you put a result in front of someone who will act on it, does your chart show how much you do not know, or does it quietly promise a certainty you cannot back up?

  • Understanding the Table 2 Fallacy in Regression Analysis

    Understanding the Table 2 Fallacy in Regression Analysis

    Open almost any study built on regression and you will find a table, often the second one, that lists the outcome regressed on a main variable of interest plus a row of controls. Each row carries a coefficient, an interval, and a p-value, lined up in a neat column. The natural thing to do is read down that column and take each number as the effect of that variable. It is one of the most common mistakes in applied statistics, and it has a name.

    The trouble starts with what the model was actually for. A multivariable regression is built to estimate the effect of one thing, the exposure or treatment you care about, and the other variables are included to remove confounding of that particular relationship. They are there to clean up the estimate for your main variable, not to have their own effects measured. Their coefficients are adjusted associations, conditional on everything else in the model, and that adjustment was chosen for your question, not for theirs.

    Here is why that matters more than it sounds. Confounding is not a property of a variable in isolation; it is a property of a specific pairing of a cause and an effect. A variable that is a proper confounder for your exposure can play an entirely different role for some other variable in the same model. It might sit on the causal path from that other variable to the outcome, a mediator, so adjusting for it strips out part of that variable’s real effect. Or it might be a common effect of two things, a collider, so adjusting for it opens a spurious association that was not there before. The very same regression that gives you a clean estimate for your exposure can hand you a badly biased estimate for a covariate, because the set of controls that is right for one is wrong for the other.

    Daniel Westreich and Sander Greenland named this the Table 2 fallacy in 2013, after the table where it usually happens. Their point was not that the model is broken. The model does its job for the exposure it was designed around. The error is in the reading: treating a table built to answer one causal question as if it answered a dozen at once.

    An example makes it concrete. Suppose you estimate the effect of a training program on earnings, adjusting for age, prior schooling, and motivation. For the training question, that may be a sensible set of controls. But if your eye drifts to the coefficient on prior schooling and you read it as the effect of schooling on earnings, you should not. Motivation may be a mediator of schooling, part of how schooling raises earnings, so adjusting for it removes some of schooling’s real effect. And the confounders you would need to estimate schooling’s effect properly, family background and the like, were never in the model, because the training question did not require them. The schooling coefficient is not the effect of schooling; it is a number shaped by a model built for a different purpose.

    The discipline is simple to state. Decide, before you model, which single effect you are estimating, and choose the controls for that effect. If you truly need the causal effects of several variables, that is several distinct questions, each deserving its own adjustment set and often its own model, not one table read greedily. When you must show the full output, label the secondary coefficients honestly as adjusted associations rather than effects, and resist interpreting them. And as a reader, do the same: find the one effect the model was built to estimate, and treat the rest as machinery.

    For those of us reading and writing evaluation reports, the temptation to mine a regression table for a dozen findings is strong, and it quietly misleads decision-makers about which factors matter. Estimating an effect is a question you answer one at a time.

    So here is my question. When you read a regression table, do you interpret only the effect the model was designed to estimate, or do you read down the column as though every coefficient were a finding?

  • Representativeness Is Not Always the Goal

    Representativeness Is Not Always the Goal

    Ask most people how to choose a sample and they will describe some version of representativeness: draw at random so the sample mirrors the population, and the larger the better. That instinct is right for one job, estimating a quantity in a population. It is the wrong instinct for much of qualitative work, where the goal is not to mirror a population at all. There, the point of sampling is to learn as much as possible about a question, and the best sample is the one that teaches you the most, not the one that looks most like the whole.

    This is purposeful sampling, and it is one of the defining features of qualitative inquiry. As Michael Quinn Patton puts it, the logic and power of the approach lie in selecting information-rich cases, the ones whose study will illuminate the question in depth. You are not trying to average across a crowd. You are trying to understand a phenomenon, and some cases reveal far more about it than others. Choosing those cases on purpose is not a compromise forced by a small budget. It is the method.

    The strategies differ because different questions call for different cases. Maximum variation sampling deliberately spans the range, so that whatever patterns hold across very different circumstances are likely robust, and the differences themselves are informative. Extreme or deviant case sampling goes to the outliers, the notable success or the striking failure, because the unusual case often exposes what stays hidden in the typical one. Critical case sampling picks the case that settles the matter: if it cannot work here, it will not work anywhere. And disconfirming case sampling seeks the examples that might break an emerging pattern, marking the boundary of a claim, the qualitative cousin of hunting for the case that breaks your theory.

    There is an even more dynamic version. In theoretical sampling, associated with grounded theory, you do not fix the sample in advance at all. You collect and analyze together, and the developing analysis tells you who or what to sample next: a concept is emerging, so you go looking for the case that will sharpen or challenge it. Sampling becomes a series of analytic decisions rather than a plan set before the first interview, and it continues until new cases stop changing the picture.

    This reframes the question people love to ask, which is how many. In qualitative work the honest answer is not a number from a power calculation but a judgment about information. Malterud and colleagues gave this a useful name, information power: the more relevant information your sample already holds, the fewer participants you need. A narrow, well-specified question studied through rich dialogue can be answered with a handful of well-chosen cases; a broad, loosely defined one may need many more. Sufficiency, not size, is the standard.

    Two honest cautions keep this from becoming an excuse. First, purposeful is not the same as convenient. Choosing the cases that are easiest to reach is the weakest form of sampling, information-poor and low in credibility, and it is not what any of this endorses. The selection logic has to be explicit and defensible, because purposeful sampling gives the researcher the power to shape the sample, and that power quietly becomes cherry-picking if the reasoning is not on the table. Second, this kind of sample does not generalize by statistics. It generalizes, when it does, by transferability: you give enough about the cases and their context that a reader can judge whether the findings extend to their own setting.

    It is worth noticing that this is the mirror image of an earlier post. A nonprobability sample is dangerous when you want to estimate a population, because its bias does not shrink as it grows. The very same nonprobability logic is the right choice when the goal is understanding rather than estimation. The tool did not change; the question did.

    So here is my question. When you choose whom to study, do you reach reflexively for a representative sample, or do you first ask what you are trying to learn and which cases would teach you the most?

  • Two Ways to Be Uncertain

    Two Ways to Be Uncertain

    You calculate a 95 percent confidence interval and describe it the natural way: there is a 95 percent chance the true value lies inside it. Almost everyone reads it like this, and it is wrong. Not because the arithmetic failed, but because you asked one framework of statistics for an answer only the other framework can give. Behind a surprising share of misread numbers is a quiet collision between two different ways of thinking about uncertainty.

    Start with the older and more dominant one. Frequentist statistics treats the unknown quantity you care about, the true effect, the real population value, as a fixed number. What is random is your data, drawn as one sample out of many you could have collected. So probability, in this world, describes how your procedure behaves over the long run. A 95 percent confidence interval means that if you repeated the whole study over and over, about 95 percent of the intervals you built that way would contain the true value. It says nothing about this one interval. The truth is either inside it or not; the confidence lives in the method, not in the particular answer sitting in front of you.

    The other framework turns the picture around. Bayesian statistics treats the unknown quantity as itself uncertain, something you hold a degree of belief about, and treats the data as the fixed thing you actually observed. You begin with a prior, an explicit statement of what you believed before seeing the data, you update it with the evidence through Bayes’ rule, and you end with a posterior, a full probability distribution over the unknown. From that you can build a credible interval, and it means exactly what people want a confidence interval to mean: given your data and your model, there is a 95 percent probability the value lies inside. It is the direct answer to the direct question.

    This is why the confusion is so universal. People are natural Bayesians. When we see a result, what we want to know is how probable the truth is, given what we saw, and that is a Bayesian question. But the standard tools of the last century, p-values and confidence intervals, are frequentist, and they answer a different question about the long-run behavior of a procedure. So we take a frequentist number and read a Bayesian meaning into it. The p-value becomes the probability the null is true; the confidence interval becomes the probability the truth is inside. These misreadings are not sloppy so much as wishful: the answer we actually wanted, projected onto a tool that cannot supply it.

    Neither framework escapes without a cost, and it is worth being fair about both. The Bayesian approach gives you the direct probability, but only by requiring a prior, and where that prior comes from is a real question. State it well and it encodes genuine knowledge; state it carelessly and a strong prior can swamp weak data and smuggle your assumptions into the answer. The frequentist approach refuses the prior and the subjectivity that comes with it, and pays by being unable to make any probability statement about the parameter itself, only about the procedure. With a neutral prior the two often produce nearly identical intervals: the numbers can coincide while the meaning does not.

    The practical point is not to pick a side. It is to know which question your number answers, and to resist reading one framework’s result as though it were the other’s. If a decision truly needs a probability about the thing itself, how likely it is that this program helps, and by how much, that is a Bayesian question and deserves a Bayesian method with a stated, defensible prior. If you want guaranteed long-run error properties and no reliance on a prior, frequentist tools are right, as long as you read them as statements about the procedure. The error is never choosing one; it is using one and interpreting it as the other.

    So here is my question. When you report a confidence interval or a p-value, are you clear, to yourself and your audience, about which question it actually answers, and which one everyone secretly wants it to?

  • Not Every Program Holds Still – The Need for Developmental Evaluation

    The standard picture of evaluation assumes a program that stays put. You specify the model, define the goals, let it run for a while, and then judge whether it hit them. When the program is stable and well understood, that works well, and most of the methods this series has covered are built for it. But a great deal of real work is not like that. Genuinely new initiatives are still finding their shape. Complex efforts operate inside systems that keep shifting under them. Pilots are meant to learn, not to prove. Hold any of these against a fixed plan and you end up measuring a moving target with a frozen ruler.

    The mismatch is deeper than it first appears. The two familiar modes of evaluation both assume there is a settled model to work with. Summative evaluation asks whether the program met predefined outcomes; formative evaluation helps improve the program toward its intended design, on the way to a later summative judgment. Both take the design as given. But for a true innovation, the design is exactly what is not yet settled, because building the model is the work. Forcing predefined outcomes and fidelity to a frozen plan onto such a program does more than miss the point: it punishes the adaptation the program should be doing, and it can lock in a design before anyone knows what actually works.

    Michael Quinn Patton named the alternative developmental evaluation. Instead of judging a program against a fixed standard at the end, the evaluator works alongside the effort and supplies rigorous, timely evidence that feeds continuous adaptation. The purpose is development, not a verdict, and the guiding question changes from did it meet the targets to what is emerging, what is that telling us, and how should the program change in response. Crucially, this is not the same as formative evaluation, a distinction Patton stresses. Formative work improves a program toward a known model; developmental evaluation supports a program that has no settled model yet, and may never hold still long enough to acquire one.

    It is worth being clear about what this is not, because the approach is easy to abuse. It is not a license to drop rigor, skip measurement, or make things up as you go. The evidence stays disciplined and documented; what is allowed to evolve is the goals and measures themselves, as understanding grows. Nor does it retire summative evaluation for good. Once an innovation stabilizes into a model that can be named and repeated, the standard questions return in full force: does this work, and is it worth it. Developmental evaluation is for the developmental phase, not a permanent posture.

    The hard part is the tradeoff it forces on independence. To feed real-time evidence into a program, the evaluator has to sit close to it, embedded in the team rather than standing outside as a detached judge. That closeness is what makes the approach useful, and it is also its central risk: an evaluator this involved can slide from candid feedback into cheerleading, losing the honesty that makes the evidence worth having. Staying useful and staying truthful at once is the real skill. And the approach fits complexity and innovation specifically; using it where a stable, well-understood program already exists would just be avoiding accountability under a fashionable name.

    For those of us working with federal programs, this matters because agencies increasingly fund innovation, pilots, and complex cross-system initiatives, and then commission evaluations designed for stable programs. That mismatch sets the evaluation up to measure the wrong thing and sets the program up to look like a failure precisely for adapting. Choosing the evaluation approach to fit the program’s actual nature, settled or still developing, is a design decision that should come before any choice of method. Sometimes the most rigorous option is the one that can keep pace with a program still finding its form.

    So here is my question. When you evaluate something still being invented, do you hold it to a fixed plan it has outgrown, or match the evaluation to a program that is still taking shape?

  • Survey Modes Matter: How Question Delivery Affects Responses

    Survey Modes Matter: How Question Delivery Affects Responses

    You compare this year’s survey to last year’s, and the numbers have moved. Or you compare a phone sample to an online one, and they disagree. The natural reading is that something changed in the world, or that one of the samples is off. But there is a quieter explanation that is easy to overlook: you changed how you asked. The mode of a survey, whether it is administered by phone, on the web, in person, or on paper, is not a neutral pipe that delivers the question and returns the answer untouched. It shapes the answer.

    The reason is that each mode puts the respondent in a different situation. The most consequential difference is whether another person is present. When an interviewer is listening, on the phone or across a table, people edit their answers toward what looks acceptable, so questions about sensitive matters draw more flattering responses than the same questions answered alone on a screen. This is the social desirability effect, and the mode turns it up or down. Self-administration, with no one watching, tends to produce more candid answers to sensitive questions, and it also changes other behavior: without an interviewer to keep the pace, respondents are freer to rush, skip, or take mental shortcuts.

    Presentation matters too. On a screen or a page you see all the options at once and can reread them; on the phone you hear them in sequence and have to hold them in memory. That difference nudges which options get chosen, with a tendency toward the last options heard in a spoken list and the first ones seen in a visual one, though the evidence on exactly when this happens is mixed. Long lists of choices are simply harder to process by ear, which invites the kind of satisficing shortcuts this series has discussed. Even the way people use a rating scale shifts, with interviewer modes drawing more agreement and telephone respondents reaching more often for the extremes.

    Put all this together and you arrive at the uncomfortable part: mode differences can look exactly like real differences. There is a striking illustration in the research literature. A study that established the instrument measured the same construct across modes, and controlled for who selected into each, still found that people interviewed face to face reported systematically better psychological functioning than people answering on the web. The gap was not a measurement artifact in the usual sense, and not a difference in who responded. It was the mode itself, most likely social desirability from the interviewer’s presence. The way the question was delivered moved the answer.

    This is why switching modes over time is so dangerous. Many surveys have moved from phone to online in recent years, to cut costs and to fight the long decline in response rates. That is often the right call. But if last year was phone and this year is web, a shift in the results can be the mode change rather than a change in the population, and you can report a trend that is really a measurement discontinuity. It is the comparability problem from the measurement-invariance post in a new guise: before you compare, you have to be sure you measured the same way.

    Mixed-mode designs, now common for the same cost and response-rate reasons, help coverage but add a twist. The people who answer by one mode may differ from those who answer by another, and the mode also nudges their answers, so mode and selection effects get tangled and are hard to separate. You cannot make any of this vanish, but you can manage it. Hold the mode constant when your aim is comparison over time, or change it deliberately and run an overlap study that measures the mode effect so you can adjust for it. Design questions to work equivalently across modes rather than optimizing each in isolation. And when you must mix modes, treat the mode as a variable in the analysis instead of pretending it is invisible. The one thing not to do is switch quietly and read the resulting change as news.

    So here is my question for the group. When your numbers move between two surveys, do you rule out the possibility that only the mode changed before you conclude that the world did?

  • The Risks of Misinterpreting Mediation Effects

    Knowing that a program worked is valuable. Knowing why it worked is more valuable still, because a mechanism you understand is one you can strengthen, cut, or carry to a new setting. So we naturally want to go further than the total effect and ask how much of it flowed through a particular pathway. Did the training raise earnings by building skills, by building confidence, or by signaling effort to employers? This is mediation analysis: splitting a total effect into the part that runs through a proposed mechanism and the part that does not. It is far more treacherous than its popularity suggests.

    For decades the default has been the approach Reuben Baron and David Kenny laid out in 1986, one of the most cited recipes in social science. Run a few regressions. Estimate the effect of the treatment on the outcome, then on the mediator, then put the treatment and mediator into the outcome model together and watch the treatment coefficient. If it shrinks, you conclude the mediator carries part of the effect. The logic is intuitive, the steps are simple, and it is taught almost everywhere.

    The problem hides in that last regression, where you control for the mediator. That gives a clean estimate of the mechanism only if you assume something strong: that nothing unmeasured causes both the mediator and the outcome. And here is the catch. Randomizing the treatment does not buy you that assumption. Randomization makes the treatment clean, so the treatment is unconfounded. But the mediator is never randomized. It is something that happened naturally after treatment, and the people who ended up with more of it, more confidence, say, may differ in unmeasured ways that also shape earnings. Those differences confound the mediator and the outcome even in a flawless experiment.

    It gets sharper still, because the mediator is a post-treatment variable, and that is exactly the kind of variable an earlier post in this series warned against controlling for. If the mediator and the outcome share an unmeasured common cause, then adjusting for the mediator, precisely what the recipe instructs, opens a spurious path and introduces collider bias. Controlling for the wrong variable does not clean the estimate; it contaminates it. The standard recipe can invent a direct effect that is not real, or erase one that is, and a randomized treatment does nothing to prevent it.

    Modern causal mediation analysis does not make these difficulties disappear, but it does make them honest. It defines the direct and indirect effects precisely, states the assumptions out loud, no unmeasured confounding of the treatment-outcome, treatment-mediator, or mediator-outcome relationships, and no mediator-outcome confounder that is itself affected by treatment, and it comes with sensitivity analysis to ask how badly a violation would have to bite before the conclusion flips. That is the discipline this series has urged elsewhere: you cannot make an untestable assumption true, but you can make it visible and probe how much weight it can bear.

    The honest takeaway is a reversal of the usual instinct. A clean mediation result deserves more scrutiny than a clean estimate of the total effect, not less, because it rests on assumptions the study design cannot secure. Whether a program worked can sometimes be settled by a good experiment. How it worked almost never can be, at least not by the experiment alone.

    For those of us who build logic models and theories of change, this matters, because those diagrams are full of mediation claims about which link is doing the work. Testing them quantitatively is worth doing, but it calls for naming the mediator-outcome confounders you fear, measuring the ones you can, and reporting how sensitive the mechanism claim is to the ones you cannot. A confident line like forty percent of the effect ran through this pathway should invite a hard look at what had to be assumed to say it.

    So here is my question. When you claim that a program worked through a particular mechanism, do you hold that claim to a higher standard than the claim that it worked at all, or to a lower one?