Rationale awareness for quality assurance in iterative human computation processes

Human computation refers to the outsourcing of computation tasks to human workers. It offers a new direction for solving a variety of problems and calls for innovative ways of managing human computation processes. The majority of human computation tasks take a parallel approach, whereas the potential of an iterative approach, i.e., having workers iteratively build on each other’s work, has not been sufficiently explored. This study investigates whether and how human workers’ awareness of previous workers’ rationales affects the performance of the iterative approach in a brainstorming task and a rating task. Rather than viewing this work as a conclusive piece, the author believes that this research endeavor is just the beginning of a new research focus that examines and supports meta-cognitive processes in crowdsourcing activities.

💡 Research Summary

This paper investigates whether making human workers aware of the rationales behind previous workers’ contributions can improve the performance of iterative human‑computation processes. Building on Little et al.’s (2010) finding that iterative workflows can raise the average quality of crowd‑generated ideas, the author introduces a meta‑cognitive element—rationale awareness—and tests two hypotheses: (1) exposing prior rationales during a brainstorming task will increase the quality of the ideas generated, and (2) providing rationales during the evaluation (rating) task will reduce inter‑rater variability.

The experimental platform is Amazon Mechanical Turk, with Turkit used to orchestrate the iterative workflow. Six fabricated company descriptions are used; each description undergoes six iterative rounds, each round requiring a Turker to propose five company names and to write a short rationale explaining why each name fits the description. Two conditions are compared in the brainstorming phase: (a) only the previously generated names are shown (control), and (b) both the names and their rationales are shown (treatment). Strict quality controls are applied (97 % approval rate, no brand names, no duplicate names, complete rationales). The control condition yields 180 unique names; the treatment condition yields 179 unique names after discarding incomplete entries.

After the brainstorming phase, a rating task is posted. Each generated name is evaluated by ten independent Turkers under two rating‑condition variants: (i) the name alone (control) and (ii) the name together with the original rationale (treatment). Raters are prohibited from rating names they themselves generated. In total, 1,800 ratings per condition are collected.

Statistical analysis (paired t‑tests) shows that the presence of rationales in the rating interface does not significantly affect average scores (p = 0.57). Regarding hypothesis 1, average ratings across iterations are plotted for both brainstorming conditions; the curves are virtually indistinguishable, indicating that showing rationales does not raise the mean quality of generated ideas. Moreover, the best‑rated name for each company description shows only negligible differences between conditions (e.g., a 0.0‑0.8 point swing), contradicting the expectation that rationale exposure would produce higher‑quality “best” ideas.

For hypothesis 2, the standard deviations of the ten‑rater scores are compared. Although a paired t‑test on the SDs yields a statistically significant difference (t = ‑3.49, p = 0.001), the direction is opposite to the hypothesis: the treatment condition (rationale shown) has a larger average SD (2.53) than the control (2.43). This suggests that providing additional information does not foster a shared understanding among raters; instead, it may introduce divergent interpretations, increasing rating dispersion.

The paper discusses several methodological limitations. First, the task domain (company‑name generation) is narrow; results may not generalize to other creative crowdsourcing tasks such as story writing, design, or problem solving. Second, the quality, length, and clarity of rationales were not controlled, potentially leading to heterogeneous informational value across workers. Third, the cognitive load imposed by reading rationales was not measured, leaving open the possibility that “information overload” contributed to the observed increase in rating variance.

From a design perspective, the findings caution against assuming that meta‑cognitive supports automatically improve crowd work outcomes. While rationale sharing can be beneficial in structured group settings, in an open, anonymous crowd environment it may add noise without guaranteeing shared interpretation. Future work should explore (a) structured or template‑based rationales to standardize informational content, (b) mechanisms for assessing rationale quality before dissemination, and (c) broader task categories to test the robustness of the effect. Additionally, investigating how rationale sharing influences collaborative decision‑making, learning, or feedback loops in human‑computer interaction could yield richer insights.

In summary, the empirical results demonstrate that exposing previous workers’ rationales neither raises the average nor the best quality of ideas in an iterative brainstorming task, and it may even increase variability in subsequent evaluations. These outcomes highlight the need for careful balancing of information provision and cognitive load when designing iterative human‑computation systems.