How prediction accuracy should be measured

A synthesis across all seven fields surveyed on this site — what honest validation looks like, and why a bare accuracy percentage, on its own, is close to meaningless.

Discrimination and calibration are not the same thing

Every mature prediction field in this survey — survival analysis, epidemiology, predictive maintenance — independently arrived at the same structural requirement: a prediction has to pass two separate checks, not one.

• Discrimination asks: does the model correctly rank items relative to each other? Does the patient who actually died first get ranked as higher-risk than the one who died later? This is what metrics like the C-index (see medicine) or AUC (see churn) measure.
• Calibration asks a completely different question: are the model's absolute predicted probabilities or times actually correct on average? If a model says "70% chance of survival to one year," are 70% of the patients it said that about actually alive at one year?

A model can score very well on one and poorly on the other. Survival analysis's own literature is explicit that the C-index is "insensitive to adding even clinically important predictors with large effect sizes," and that a high C-index does not imply the model's absolute predicted times are correct — calibration has to be checked separately (see medicine). Epidemiology found the same failure mode from the other direction: one Rt-estimation method produced a narrower-looking, more confident-seeming interval that in simulation actually covered the true value less often than a wider, more honest-looking alternative (see epidemics). A tighter number is not automatically a more honest one.

Interval coverage: the test behind the test

A confidence or credible interval is only meaningful if its stated coverage rate actually holds up against reality. A "95% confidence interval" that, when checked against thousands of real or simulated outcomes, only contains the true value 60% of the time is not a 95% interval in any meaningful sense — it is a miscalibrated one wearing a 95% label. Reliability engineering computes MTBF bounds via the chi-square distribution specifically so the stated confidence level holds up under audit (see machines); epidemiologists explicitly audit Rt credible-interval coverage against simulated ground truth for the same reason (see epidemics). Coverage checking — not just reporting an interval, but separately verifying that the interval means what it claims to mean — is the test behind the test, and it is the single most consistent practice across every rigorous field in this survey.

Why unsourced accuracy percentages are meaningless

A bare number like "85% accurate" answers none of the following questions, all of which are required before the number can be evaluated at all:

• Accurate at what task — ranking, magnitude prediction, or timing prediction? These are not interchangeable (see the content decay page for a field where this conflation happens routinely).
• Measured against what ground truth, and on a held-out sample the model never saw during fitting, or on the same data it was trained on (which inflates the number)?
• What is the calibration, separately from the discrimination or point-accuracy figure?
• What is the uncertainty around the number itself — is 85% the center of a tight interval or a wide one?
• Is the methodology disclosed and reproducible, or asserted without a visible dataset, holdout split, or definition of "accuracy"?

This is not a hypothetical concern. A review of commercial social-analytics platforms for this survey found that the major enterprise incumbents — Sprinklr, Brandwatch, Hootsuite, Rival IQ — publish no validated, methodology-disclosed accuracy number at all for virality or decay prediction; their public marketing describes qualitative capabilities only. Meanwhile, smaller non-enterprise marketing tools do publish specific-looking percentages (70–85%) with no visible methodology, holdout set, or ground-truth definition. The absence of a number from the market leaders, and the presence of unsourced numbers from smaller tools, is itself the finding: a specific-looking percentage is not evidence of rigor, and the lack of a published number from established players is informative, not just a gap in the research.

The field-norms table

BeltUp LLC's commitment