Who said that, and is it normal for them?

The short version

We built two things, and the order matters.

First, a foundation. Every recording on our platform now has a canonical, word-by-word timing record, materialised into numbered sentences, with every sentence attributed to a speaker role: the participant, a moderator, an interpreter, a screen reader, an observer, or an honest "unknown". This is unglamorous plumbing and it is the most valuable thing in this article.

Second, the first measurement built on it. Every time a tester finishes a session, we measure how often they used language that signals confusion or low confidence, and compare that against how that specific person normally speaks. Unusual for them raises a hand for a researcher. Normal for them does nothing. It runs automatically on every recording and is live end to end.

What we are claiming, and what we are not

It is worth being precise about this up front, because it is the part a careful reader will go looking for.

Our hypothesis is that comparing a tester against their own speech history will surface sessions containing real product friction that a cross-person count would miss, and will stop penalising people simply for being more talkative than average. That hypothesis is well grounded in the psycholinguistic literature, which we cite at the end. It is not yet proven on our own data.

What the work described below delivers is the data ingestion and information infrastructure that makes the hypothesis testable at all: speaker-attributed sentences, word-level timing, per-person baselines, and a permanent record of every evaluation ever made. You cannot test a claim about individual speech patterns without first being able to tell whose speech you are looking at, and how that person usually sounds. Building the instrument is the precondition for finding out whether the instrument reads anything real.

So this article is a description of an instrument and the reasoning behind its design, not a results paper. The measurement is not reliable on its own and we do not treat it as though it were. It is one input among many, deliberately narrow, with documented blind spots. A flag is a prompt to go and watch four minutes of video. It is not a finding. The section near the end sets out exactly where it fails, and if you only read one part of this article, read that one.

Part one: the transcript is not a blob

Here is the thing that quietly invalidates a great deal of published user research.

Most platforms treat a session transcript as a single undifferentiated wall of text. Words came out of a recording; here they are. You can search it, you can feed it to a model, you can count things in it. What you cannot do is tell whose mouth any of it came from.

For a lot of sessions that is survivable. For the sessions that matter most, it is fatal.

Why this breaks accessibility research specifically

A screen reader talks constantly. It announces every link, every button, every state change, every heading level, in a flat synthesised stream that can easily outpace the human using it. If your transcript is a blob, that torrent of machine speech is indistinguishable from the tester's own words.

Now count anything. Words spoken. Hesitations. Confusion phrases. Time spent talking. Every blind and low-vision tester in your study will register as wordier, busier and more confused than every sighted tester, and not one bit of that difference will be about the product you are testing. It is an artefact of their software.

The same failure repeats in other shapes. An interpreted session contains an interpreter's voice. A moderated session contains a facilitator asking "does that bit confuse you at all?", a sentence which, counted naively, makes the tester look confused for having been asked. An observer's aside becomes tester speech.

This is not a refinement or a data-quality nicety. It is the difference between a valid measurement and an invalid one, and it lands hardest on precisely the cohorts an inclusive research platform exists to serve. A platform that cannot separate these voices is not measuring those users conservatively. It is measuring them wrongly, and in a consistent direction.

What we built instead

Three layers, each built on the one below.

• Words. Every recognised word and punctuation mark from every recording, with exactly when it started and ended, how confident the transcription was, and which detected voice produced it. One row per word, indexed, queryable. Not a flattened field that has to be re-parsed from scratch every time someone wants to look at it.
• Sentences. Those words grouped into coherent units by sentence-final punctuation or a long enough pause, each with its own start and end time, its own number, and a pointer back to the exact words underneath it. Sentence fourteen of a specific video is now a real, addressable thing.
• Roles. Each sentence attributed to participant, moderator, interpreter, screen reader, observer, or unknown, drawn from a canonical list rather than a hardcoded tester-versus-other binary.

Two design choices in there are worth dwelling on.

"Unknown" is a first-class answer. Where the system genuinely cannot tell who was speaking, it says so rather than guessing. A moderated session where only one voice is detected is itself a signal that something is off, and it gets recorded as unknown rather than optimistically labelled participant. We would rather hold visible uncertainty than invisible error.

A sentence can be marked as spanning a speaker change. Where a short back-and-forth gets mis-grouped into one sentence, that sentence carries a flag saying so, and anything reading the data can exclude it. Most of the damage done by transcript data is done by rows that look clean and are not.

Why this is the part worth copying

Everything downstream inherits this. An insight can cite sentence fourteen as its evidence rather than paraphrasing the transcript and hoping the paraphrase can be matched back, or asking a model to report a timestamp and simply trusting the number it returns. A clip can be trimmed to a real boundary. A quote can be attributed with confidence. A cohort comparison can be made without an assistive-technology confound sitting inside it.

And any measurement built on top starts from data that is actually about the person you think it is about. Which brings us to the first measurement we built.

Part two: why counting across people measures personality

Suppose you have solved the speaker problem. You now have a clean record of what the tester, and only the tester, actually said. The obvious next move is to count confusion phrases, rank the sessions, and look at the top of the list.

Do not do this. It does not work, and the reason it does not work is worth being precise about.

The variance is in the wrong place

How much people narrate, hedge and hesitate while concentrating varies enormously between individuals. Some testers keep up a running commentary like a sports broadcaster. Others go quiet and grim and produce eleven words in fifteen minutes. Some people say "not sure" as verbal punctuation, in the way other people say "right" or "okay". Some are naturally fluent under pressure; some hesitate constantly and are completely fine.

This is not folk wisdom. Corpus work on task-oriented conversation has measured disfluency rates varying substantially with the speaker's age, their gender, their role in the task, their familiarity with the person they are talking to, and how abstract the subject matter is. Differences of one to two disfluencies per hundred words between demographic groups are routine findings, before anyone has considered whether the thing being discussed was any good.

That between-person spread is large. The thing you are actually trying to detect, one person having a harder time than they usually do, is small by comparison. It sits well inside the range that ordinary differences in personality and speaking style already occupy.

So when you rank people against each other on raw phrase counts, the between-person variation swamps the signal completely. The top of your list is not the testers who struggled. It is the testers who talk the most and hedge the most, who would have appeared there regardless of what you put in front of them. You have built a leaderboard of verbosity and labelled it insight.

This is the single most common way language-based friction metrics go wrong, and it is not a tuning problem. No threshold fixes it, because the noise is structurally larger than the signal you are looking for.

The fix: everyone is compared to themselves

The only defensible move is to stop comparing people to each other. Each tester accumulates a personal profile, their own average marker rate, and their own natural session-to-session variation. Think of it as a resting heart rate. It is useful precisely because it is theirs, and it is meaningless averaged with anyone else's.

The question stops being "is this rate high" and becomes "is this rate high for them". The between-person variation that was drowning everything gets subtracted out before any comparison happens, because it is baked into the baseline you are comparing against. What is left is the within-person movement you actually wanted.

A tester who habitually hedges at a high rate is not flagged for hedging. That is just how they talk, and it tells you nothing about your product. The same tester dropping well below their own usual fluency does get flagged, even if their absolute rate is still higher than a quieter colleague's worst day. That inversion is the whole point.

Note the exact shape of the claim, though. Removing a known source of noise is a measurement improvement we can demonstrate directly. Whether the movement that remains actually tracks product friction is a separate question, and an empirical one that our own data has not yet answered. We have built the instrument. We have not yet published the reading.

Two gates before anyone is judged

A personal baseline is only as good as the history it rests on, so a profile does not become usable until it clears two bars. Both were chosen for a specific reason rather than because they were round numbers.

Eligibility gates before a tester's baseline is trusted, and the reasoning behind each

Gate	Why this, and not something else
At least 20 minutes of their own speech	An earlier draft used a video count instead, and a count of videos says nothing about how much speech you actually collected. Ten one-minute clips and ten twenty-minute sessions are not the same evidence. Measuring the thing you actually need, cumulative participant speaking time, removes that ambiguity entirely. Short sessions still contribute their minutes; there is no per-video minimum that would discard them.
Across at least 2 different client projects	This is the one that shows the reasoning. A tester who has only ever recorded for a single client may have produced hours of footage, and their "baseline" would still be contaminated by that one product. A confusing interface makes a person sound confused, and with one project in the history there is no way to separate the person's speaking style from that product's effect on it. Two projects is the minimum at which a baseline describes a person rather than a product.

Until both are true, the profile is explicitly marked as insufficient data and no session is ever flagged against it. We do not judge someone until we actually know them, and "we don't know yet" is stored as a real state rather than quietly rendered as a baseline of zero.

One further consequence worth stating: a baseline belongs to the individual, not to the client. A tester who works across three organisations gets one coherent baseline, not three fragments that each half-know them. That is only possible because our testers are an ongoing panel rather than strangers recruited fresh for every study. It is also the reason this approach is difficult to copy on a platform where nobody is ever seen twice.

How the measurement actually works

With both foundations in place, the mechanics are deliberately simple. We scan only the participant's sentences for sixteen markers in three families: seven comprehension phrases, five confidence phrases, and four disfluency sounds ("um", "uh", "ah", "like") that carry no friction classification of their own.

A rate, not a tally

Marker count divided by words actually spoken. And if a tester did not speak at all, we leave the rate blank rather than writing a zero. A zero says "we checked, and they were completely fine". Blank says "we do not know". Those are different statements, and a dataset that confuses them will mislead someone eventually.

The comparison

Take the session's rate, subtract the tester's own average, divide by their own normal variation. At 2.33 or above, the session is raised for internal review.

Maya usually runs at about two confusion phrases per hundred words, wobbling by around 0.8 either way. One session she jumps to 4.5. That is (4.5 − 2) ÷ 0.8 = 3.1, past the line. The flag does not claim "Maya was confused". It says the narrower and more useful thing: Maya does not usually sound like this.

Order of operations matters more than it looks. We compare the session against the tester's history before folding that session into it. Mix it in first and a genuine spike averages itself into "normal" before anyone notices it happened.

Isn't one shared cut-off the opposite of personal?

Reasonable objection: if everyone is judged against 2.33, where is the personalisation? It already happened. The average and the variation in that calculation come entirely from that individual's own history, so by the time we reach the comparison we are asking "how many of this person's own typical wobbles above this person's own normal is this". Once a measurement is expressed in someone's own units, there is no reason the threshold should differ person to person. 2.33 is just the standard cut-off for "this would happen by chance about one time in a hundred". The personalisation is in the yardstick.

What this actually sounds like

The examples below are illustrative rather than real tester speech, but they are the shape of what the pipeline reads. Markers are shown in bold.

A stretch of a session

"Right, so I'll go into settings, um, and then... uh... hang on. I'm not sure what this toggle actually does. Um. Is it like a master switch for all of them, or just this one? Yeah, no idea."

Six markers in roughly forty-five spoken words, a rate of about 0.13. For a tester whose personal average sits near 0.03, that is a substantial outlier and flags immediately. For a tester who habitually thinks aloud at 0.14, it is an entirely ordinary Tuesday and nothing happens. Same words, same forty-five seconds, opposite conclusions. That is the argument for per-person baselining in one paragraph.

Comprehension markers, in context

• "I'm confused, is this the same thing as the one on the previous page?"
• "What does 'concession status' mean here?"
• "That doesn't make any sense to me."
• "The wording's a bit unclear, I think?"

Confidence markers, in context

• "I don't know where to go from here, honestly."
• "I'm not quite sure whether that saved or not."
• "I think I'm a bit lost now."
• "I don't know what to click next."

Disfluency, in context

• "Um... uh... ah, there it is."
• "So I'd like, um, probably tap that one?"

Disfluency is the softest of the three families and we treat it accordingly. "Um" is not a complaint about your product. It is a person thinking. It earns its place only as a rate that moves relative to that person's own habitual rate of thinking out loud.

What deliberately does not count

This list matters more than the ones above it, because it is where naive implementations quietly fall apart.

• "I like this layout." Here "like" is a verb, and the sentiment is positive. It was the single riskiest item on our list precisely because it is also just a normal English word, which is why it was hand-validated against real matches rather than assumed safe.
• "Uh-huh." Backchannel agreement, not hesitation. Someone politely signalling they are still listening is the opposite of someone struggling.
• "Link. Settings. Button. Collapsed." Screen-reader output. Never scanned, because it is not the tester speaking. This is part one doing its job.
• "Does that bit confuse you at all?" A moderator's question, excluded by speaker role, which also stops a facilitator's vocabulary contaminating a tester's baseline over time.
• "Well, that's not confusing at all." Sarcasm and polarity inversion are genuinely hard, and pattern matching will never be trustworthy here. Which is the point of the next section.

Decisions we would be happy to be interrogated on

Design choices behind the pipeline, and the reasoning for each

Decision	Why it is built that way
Evaluate before updating	A spike cannot quietly absorb itself into the average before anyone sees it.
Permanent evaluation snapshot	Every check, flagged or not, writes an immutable record of what the baseline looked like at that moment. Profiles move. Without the snapshot, "why was this flagged in March" becomes unanswerable by April.
Frozen, versioned phrase list	Changing the marker list creates a new pipeline version rather than editing the old one in place. Historical measurements stay honest instead of being silently rewritten by today's definitions.
Vendor-neutral timing schema	Every word row records which transcription engine produced it. Changing engines later replaces the producer of the data, never the features that consume it.
Measurements outlive their sources	Deleting a video detaches the pointer but never destroys the measurement. The extracted data point is the expensive, non-reproducible artefact.
Precision checked by hand	Two hundred real matches manually labelled before this ran at scale, plus a regression suite of adversarial cases: sarcasm, polite backchannel, Australian and British idiom, screen-reader phrasing.

Why this is a very long way from a survey

The dominant model in user testing is still: run the session, ask the participant how it went, aggregate the answers into a number. Three problems, none of them small.

Surveys measure the end of the session, not the session

A post-task rating is a memory of an experience, compressed into one number, after the difficult part is over and the relief has set in. This is one of the better-evidenced findings in the psychology of experience. Retrospective evaluations of an extended episode are dominated by its most intense moment and by how it ended, and are strikingly insensitive to how long the unpleasant part actually lasted. People remember the last thirty seconds, they remember whether they eventually succeeded, and they tend to blame themselves rather than the interface. The thing you wanted to know happened four minutes ago and has already been overwritten.

Speech produced during the task does not have that problem, and the distinction is a well-established one in cognitive psychology: verbalisation of what someone is attending to while they are attending to it is treated as legitimate data, while retrospective explanation of processes that were never consciously attended to is not. "I have no idea what that means" is emitted at the exact moment of difficulty, with a timestamp, inside a recording you can watch.

Survey fatigue is a measurement artefact, not an inconvenience

The industry treats declining response quality as an annoyance to be managed with shorter forms and better incentives. It is worse than that: it is a systematic distortion. When answering a question properly would take real cognitive effort, a predictable share of respondents stop optimising and start satisficing, which shows up as picking the first plausible option, agreeing with whatever the question asserts, failing to differentiate between items, or selecting at random. None of that looks like missing data. It looks like answers.

Nothing here asks the tester for anything. No extra step, no form, no cognitive tax, and therefore no fatigue curve to correct for. The signal is a by-product of the session that already happened.

"The average user" is a fiction, and averaging makes it worse

Aggregate enough ratings and you converge on a description of nobody. This is particularly destructive in accessibility research, where the cohorts that matter most are exactly the ones a mean will flatten. A blind tester, a Deaf tester working through an interpreter, and a sighted tester on a laptop are not three noisy samples of one underlying user. They are three people having three different experiences.

Per-person baselining refuses to average across them in the first place. Nobody's difference from a norm is mistaken for difficulty, and nobody's genuine difficulty is hidden inside someone else's normal.

One input among many, and never the loudest

None of this replaces our existing insight generation. It sits alongside it, and the division of labour is deliberate.

A language model reads each session with full context. It handles negation, sarcasm, the task the tester was given, and the difference between "this is confusing" and "this used to be confusing". Sixteen pattern matches will never out-read that, and we did not try.

What a language model structurally cannot do is remember you. It meets each tester for the first time every single session, and cannot know that eight hesitations is unremarkable for this person and alarming for that one. Carrying someone's entire speech history into every prompt is not a realistic fix.

So the split is clean. The model answers "does this express confusion?" The baseline answers "is this unusual for them?" Neither can do the other's job, and a marker rate read in isolation would be exactly the kind of decontextualised number this whole design argues against. It is one modality alongside what the tester did, where they went, how long they took, what they said in context, and what the session was actually asking them to accomplish.

Communication is deeply personal. Treating it as one measurable modality among many, with its own baseline and its own honest error bars, is a more respectful way to study it than reducing it to a rating.

Where this signal is weak, stated plainly

Every measurement has a failure mode. Most vendors discover theirs in public and describe it afterwards as a learning. Here are ours, in advance.

• It is internal, and stays internal. A flag does not appear on any client report, does not enter any score, and does not override the model's own reading of a session. It raises a hand for a researcher. Expanding that job is not on the roadmap.
• The core hypothesis is not yet tested on our data. Everything here rests on the proposition that within-person movement in confusion language tracks real product friction. The literature supports it. Our own correlation study has not run. Until it does, treat this as a well-motivated instrument rather than a validated metric.
• It catches spikes better than slides. Once a flagged session is folded into someone's profile, a similar session soon after is less likely to flag, because it has become part of their normal. Good for a one-off rough session. Weaker at a slow, sustained decline, which is exactly the pattern a human reviewer spots better than a threshold does.
• The profile is a lifetime average. It grows more confident the longer we know someone, but a genuine long-term change in how a person speaks gets diluted by their own history rather than tracked. A tester who was hesitant two years ago and is fluent now carries that old self around in their baseline. First thing on the list to revisit.
• Speaker detection has a ceiling. Our current transcription configuration reliably separates two voices. A session with a moderator and an interpreter both speaking falls back to "unknown" rather than guessing. Honest uncertainty is better than confident fiction, but it is still a gap, and closing it properly means changing how sessions are recorded rather than how they are analysed.
• Language is not a reliable proxy for difficulty. Sarcasm, polarity inversion, regional idiom, and someone cheerfully saying "no idea!" about something they do not need to know are all real. The marker rate does not attempt to resolve any of them. That is the model's job, and the reason the two signals are read together rather than one standing in for the other.
• Deaf and ESL cohorts are scoped separately. Applying a spoken-disfluency model to signed or interpreted communication without thinking very carefully first would be exactly the shortcut this design exists to avoid. That work is being designed on its own terms rather than retrofitted.

Read that back and the honest summary is this: a confusion flag is a reason to watch four minutes of video. It is not a reason to believe anything yet.

How we intend to test the hypothesis

The pipeline is live on every new recording, the historical corpus has been processed, and the signal is in front of the people who can act on it. That is the instrument built. The experiment starts here.

The test is a comparison between two independent methods that have never seen each other's output. On one side, the sessions this pipeline flagged as unusual for the individual. On the other, the friction our insight generation identified by reading each session in full context. Neither knows what the other concluded.

If the two agree at better than chance, the hypothesis survives its first real contact with our data, and we have a cheap early-warning signal that costs a tester nothing and asks them for nothing. If they disagree, that is the more interesting result, because it means one of two independent methods is seeing something the other is blind to, and we would very much like to know which. Either outcome is publishable and we intend to publish it, including the unflattering version.

But the foundation underneath holds regardless of how that comparison lands. Knowing exactly when every word was spoken, and by whom, is not a feature. It is the precondition for every honest claim a research platform makes about what happened in a recording, and for every hypothesis it might want to test next. Most of us have been making those claims without it.

Sources

The design reasoning above draws on the following. Each entry notes which claim in the article it supports.

Bortfeld, H., Leon, S. D., Bloom, J. E., Schober, M. F., & Brennan, S. E. (2001). Disfluency rates in conversation: effects of age, relationship, topic, role, and gender. Language and Speech, 44(2), 123 to 147.

https://pubmed.ncbi.nlm.nih.gov/11575901/

Supports the central variance argument. Measured disfluency rates in task-oriented conversation varying with speaker age, gender, task role, familiarity between speakers and abstractness of the topic, and found disfluency rates rising with planning difficulty.

Brennan, S. E., & Williams, M. (1995). The feeling of another's knowing: prosody and filled pauses as cues to listeners about the metacognitive states of speakers. Journal of Memory and Language, 34, 383 to 398.

https://www.semanticscholar.org/paper/e85cc2c429111c7b8c5c4a4f0f344a374a127f81

Supports treating filled pauses and hesitation as carrying information about a speaker's confidence rather than as noise to be stripped out.

Ericsson, K. A., & Simon, H. A. (1980). Verbal reports as data. Psychological Review, 87(3), 215 to 250.

https://www3.nd.edu/~ghaeffel/Ericsson&Simon(1980).pdf

Supports the distinction between speech produced while a person attends to a task, which is treated as legitimate data, and retrospective explanation of processes that were never consciously attended to, which is not.

Redelmeier, D. A., & Kahneman, D. (1996). Patients' memories of painful medical treatments: real-time and retrospective evaluations of two minimally invasive procedures. Pain, 66(1), 3 to 8.

https://doi.org/10.1016/0304-3959(96)02994-6

Supports the claim that a rating given after an experience is dominated by its peak and its ending, and is largely insensitive to how long the difficult part lasted.

Krosnick, J. A. (1991). Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology, 5(3), 213 to 236.

https://onlinelibrary.wiley.com/doi/abs/10.1002/acp.2350050305

Supports the argument that survey fatigue is a distortion rather than an inconvenience. Sets out satisficing, where respondents facing cognitive effort choose the first plausible option, agree with the question's assertion, fail to differentiate between items, or answer at random.

W3C Web Accessibility Initiative. Involving Users in Evaluating Web Accessibility.

https://www.w3.org/WAI/test-evaluate/involving-users

Supports the position that conformance evaluation alone does not surface the usability problems disabled and older users actually encounter, and that evaluation should include those users directly.