Eric Bergman
July 11, 2025
“Human Factors is more of a game of words than a game of numbers”
– Ron Kaye
Many years ago, I attended a talk by Ron Kaye who had recently retired from leading FDA’s human factors team at the Center for Devices & Radiological Health (CDRH). Ron made a compelling argument that words are more important than numbers in the medical device human factors engineering domain. His talk was so engaging that I jotted down the quotation above. When I read it again a few weeks ago it stimulated thoughts on this topic.
Ron was right about human factors engineering being a “game of words”. Like many practitioners I was educated in research (in my case, Cognitive Psychology). I took graduate statistics and my doctoral thesis relied on quantitative analysis. Yet decades into a career in human factors engineering, usability, and user experience design I find that quantitative analysis is rarely required and often misleading.
Don’t get me wrong – sometimes numbers and statistics (or at least understanding them) can be critical. However, serious quantitative analysis tends to be the exception. I get excited when I get to scrape the rust from my quantitative toolset and put those tools to work. It doesn’t happen often.
As the cliché goes, anecdote is not the singular of data. Still, stories tell us a lot…
A recurring theme over my career has been demand for numbers associated with findings from user research in the field and usability tests in the lab. A common scenario is something like this: we identify a usability issue with one or two participants in a large group. A member of the project team downplays the importance of the issue by arguing that it was “only” one person or that one use error “isn’t statistically significant”. I took graduate level statistics. Don’t statsplain significance to me!
What’s the answer to such pushback? If you’ve been at this for a while, you already know. The answer is that people objecting to numbers or demanding better data (whatever that means) are often asking the wrong question. The question isn’t what is the significance of the number of people, but rather what is the significance of use error itself? What are the root causes of what was observed and (words, not numbers) is it plausible to believe that these conditions will be repeated in real world use?
“Of all the gin joints in all the towns in all the world, she walks into mine.”
– Rick Blaine (played by Humphrey Bogart in Casablanca)
Probability and statistics aren’t needed for many questions about use error, especially in medical device usability testing, because if you observe a use error in your study you shouldn’t ask what is the probability of this happening in the world, but rather, what is the probability of this happening only in my study and never again? What are the chances that the one person in the world who was confused about this part of the interface happened to show up today and was one of 15 participants? When you ask the question that way, the answer becomes clear. It is unlikely that any usability related event you observe in a study will be the only time it ever happens. You don’t need statistics to figure that out.
The exception to this rule is if the error occurs due to the study design. Of course, you must guard against that possibility in the study design and evaluate the root causes of observed use errors in your data analysis to exclude that possibility. However, that analysis requires no numbers, only the game of words.