This piece argues that a common reporting practice (reducing a testing program to a single mean score per task or cohort) actively hides the most decision-relevant information in the data, and proposes what to report instead.

The hypothesis

Averages are supposed to summarize. In accessibility and usability testing, we suspect they frequently do the opposite: they take two groups of testers having completely different experiences (one group succeeding cleanly, another group failing badly) and report a single number that describes neither group’s actual experience. If that’s true, it means the most commonly reported statistic in the field (mean score) is systematically the least informative one for exactly the cases where it matters most: the products sitting in the “medium” range that everyone assumes need incremental polish rather than structural fixes.

We tested this by looking not at averages alone, but at the spread underneath them: specifically, the gap between the minimum and maximum score recorded for the same task and the same cohort in the same testing session.

What we found

Restricting the analysis to task-cohort groups with at least three testers (so a single outlier can’t manufacture an artificial spread), we have 148 groups to examine across the 15 studies in this dataset. The average spread (the difference between the highest and lowest accessibility score recorded within the same task, same cohort, same study) is 40.5 points on a 100-point scale, with a median of 43.1. Nearly 44% of these groups have a spread of 50 points or more. Roughly 7% have a spread of 80 points or more, meaning, within the same cohort doing the same task in the same study, one tester scored close to a total failure while another scored close to a flawless success.

That alone is a meaningful finding. But the sharper version of the hypothesis is about what happens specifically to groups whose average score looks unremarkable: not clearly broken, not clearly fine, just sitting in the middle of the scale where a report author would reasonably write “moderate usability, room for improvement” and move on.

We isolated every task-cohort group with an average accessibility score between 35 and 65, the range most likely to be read as “medium” and therefore lowest-priority. There are 60 such groups in this dataset. Within them:

  • 65% have a spread of 50 points or more.
  • 15% have a spread of 80 points or more, meaning a “moderate” 50-out-of-100 average was, in the specific cases we checked, sometimes literally the midpoint between one tester’s 0 and another tester’s 100.

In other words: a moderate-looking average score is not weak evidence of a moderate experience. In this dataset, it is more often than not evidence of a split experience that happens to average out to something that looks moderate: two very different outcomes, occurring in the same cohort, on the same task, that a single summary statistic renders indistinguishable from genuine consistency.

Why this happens, and why it might be good news

There’s a version of this finding that would be discouraging: if scores were this volatile purely at random, it would suggest accessibility outcomes are unpredictable and hard to design for. We don’t think that’s the right reading, and the same-cohort data specifically points elsewhere.

Look at blind testers specifically, the cohort with the lowest overall average accessibility score in this program (29.0 out of 100, reported separately). If a low average reflected a uniformly difficult, hard-ceiling experience for blind users regardless of the product, we would expect blind testers’ scores to cluster narrowly around that low number on every product. They don’t. Several individual task-cohort groups for blind testers in this dataset show a full 0-to-100 spread: one tester encountering a completely inaccessible flow, another tester on the same task in the same study rating it a full success. That range is consistent with excellent, fully accessible experiences for blind users being achievable within the same product category and the same general timeframe as the catastrophic ones, which would mean the difference is not an inherent property of testing with blind users, but a property of specific implementation choices that happened to go right in one place and wrong in another, sometimes within the same product.

That would reframe the volatility finding from a statistical curiosity into an actionable one: on this reading, high spread is evidence that the problem is fixable, rather than evidence that the cohort is simply hard to serve. A uniformly low score with low spread would be the more worrying pattern: it would suggest a structural ceiling nobody has broken through yet. A high-spread pattern says the ceiling has already been broken through, somewhere, by someone; the work is diagnosing what was different about the sessions that succeeded. We can’t fully adjudicate between that reading and the alternatives with this dataset alone: the same spread is equally consistent with between-tester variation (different screen readers, different browser and assistive-technology combinations, differing levels of expertise) or with inconsistent application of the scoring rubric by different reviewers. This interpretation should be treated as a hypothesis worth testing further, not a settled result.

What we think testing programs should report instead of (or alongside) the mean

If a moderate mean score is the least reliable signal in a testing report, at minimum it shouldn’t be reported alone. Three low-effort additions would have caught the volatile groups identified in this analysis:

Report the floor alongside the average. The minimum score for a task-cohort group is a direct answer to “what’s the worst this gets,” which a mean by definition cannot answer. In this dataset, the floor was frequently 0 even when the average looked acceptable, information a stakeholder reading only the mean would never see.

Flag spread, not just central tendency. A simple threshold (flag any task-cohort group where max-minus-min exceeds, say, 40 or 50 points) would have surfaced 44% of the groups in this dataset for closer manual review, disproportionately concentrated in exactly the “moderate average” band where a reviewer is least likely to look closely without a prompt.

Treat a moderate average as a request for a qualitative read, not a quantitative verdict. Given that 65% of moderate-average groups in this dataset carried a 50+ point spread, a mean sitting in the middle of the scale should trigger someone actually reading the underlying session notes or watching the sessions, rather than being treated as a settled, if unremarkable, number.

Limitations

This analysis is restricted to groups with at least three testers, which excludes a large share of the underlying data (378 of 526 task-cohort combinations) precisely because spread is unreliable to interpret with only one or two observations, a legitimate constraint, but one that means the 148-group sample here should be read as directional rather than exhaustive. We also have not tested whether spread itself correlates with anything actionable in advance (for example, whether certain task types or component categories are more prone to high-spread outcomes than others); that’s a natural next step this dataset could support with further analysis, but we haven’t done it here and don’t want to imply otherwise.

Open question

If high spread within a cohort signals that success is achievable but inconsistently achieved, the more interesting research question isn’t “how much does score vary”: it’s “what specifically differed between the highest-scoring and lowest-scoring session on the same task.” That’s a session-comparison question our current aggregate data isn’t structured to answer directly, but it’s the natural next study: pair the highest and lowest scoring sessions for the same task and cohort and look for what changed.

About this analysis

Figures in this piece are drawn from an anonymised aggregation of scoring data across 15 independent usability testing projects conducted by See Me Please between late 2025 and mid-2026. Spread is calculated as the difference between the minimum and maximum accessibility score recorded for a given task and cohort within a study. All client and participant identities have been removed.