Discovering the critical number of respondents to validate an item in a questionnaire: The Binomial Cut-level Content Validity proposal

The question that drives this research is: “How to discover the number of respondents that are necessary to validate items of a questionnaire as actually essential to reach the questionnaire’s proposal?” Among the efforts in this subject, \cite{Lawshe1975, Wilson2012, Ayre_CVR_2014} approached this issue by proposing and refining the Content Validation Ratio (CVR) that looks to identify items that are actually essentials. Despite their contribution, these studies do not check if an item validated as “essential” should be also validated as “not essential” by the same sample, which should be a paradox. Another issue is the assignment a probability equal a 50% to a item be randomly checked by a respondent as essential, despite an evaluator has three options to choose. Our proposal faces these issues, making it possible to verify if a paradoxical situation occurs, and being more precise in recommending whether an item should either be retained or discarded from a questionnaire.

💡 Research Summary

The paper addresses a long‑standing problem in questionnaire development: determining how many expert respondents are needed to validate an item as essential. Traditional content validity methods, especially Lawshe’s Content Validity Ratio (CVR), assume a 50 % chance that an expert will label an item as “essential” even though the response format typically offers three options (essential, important but not essential, unnecessary). Subsequent refinements by Wilson et al. (2012) and Ayre & Scally (2014) attempted to correct the critical values but retained the flawed probability assumption and relied on normal approximations that are inaccurate for small samples.

The authors propose the Binomial Cut‑level Validation (BCV) method. BCV first defines the success probability p as the reciprocal of the number of response options (p = 1/k), so for a three‑option scale p = 1/3. For each item, the number of “essential” responses (n) out of total experts (N) is modeled with a binomial distribution B(N, p). A two‑tailed hypothesis test is performed: H0 : p = 1/k versus Ha : p ≠ 1/k. The resulting p‑value determines whether the item can be classified as “essential” (significant in the upper tail), “unnecessary” (significant in the lower tail), or “paradoxical” (both tails non‑significant, indicating inconsistency).

BCV also provides an explicit formula for the minimum number of essential responses required for a given sample size, confidence level, and significance threshold, allowing researchers to plan the exact number of expert raters needed before data collection. By using the exact binomial distribution, BCV avoids the distortions introduced by normal approximations, especially when N is small.

A comprehensive literature review shows that CVR remains widely used across fields such as education, healthcare, nutrition, production economics, and quality‑of‑life research. The authors argue that despite its popularity, the statistical shortcomings of CVR can lead to over‑ or under‑estimation of required panel sizes and may mask contradictory judgments among experts.

The paper’s contributions are threefold: (1) it replaces the arbitrary p = 0.5 assumption with a theoretically justified p = 1/k; (2) it introduces a double‑check mechanism via two‑tailed testing, enabling simultaneous validation of “essential” and “unnecessary” status and detection of paradoxical results; (3) it supplies a clear, sample‑size calculation based on the binomial model, eliminating the need for pre‑computed tables.

Limitations include the lack of empirical case studies demonstrating BCV in practice, and the need for further guidance when the number of response options exceeds three (e.g., Likert scales). The additional step of testing the “unnecessary” direction may increase respondent burden in some contexts. Future work should involve simulation studies across various k values, development of user‑friendly software, and real‑world applications to confirm the method’s robustness.

In summary, the Binomial Cut‑level Validation method offers a statistically sound, transparent, and flexible alternative to the traditional CVR, improving the rigor of content validity assessments and helping researchers determine the precise number of expert respondents required for reliable questionnaire item validation.