Perceived Political Bias in LLMs Reduces Persuasive Abilities

Conversational AI has been proposed as a scalable way to correct public misconceptions and spread misinformation. Yet its effectiveness may depend on perceptions of its political neutrality. As LLMs enter partisan conflict, elites increasingly portray them as ideologically aligned. We test whether these credibility attacks reduce LLM-based persuasion. In a preregistered U.S. survey experiment (N=2144), participants completed a three-round conversation with ChatGPT about a personally held economic policy misconception. Compared to a neutral control, a short message indicating that the LLM was biased against the respondent’s party attenuated persuasion by 28%. Transcript analysis indicates that the warnings alter the interaction: respondents push back more and engage less receptively. These findings suggest that the persuasive impact of conversational AI is politically contingent, constrained by perceptions of partisan alignment.

💡 Research Summary

The paper investigates how perceived political bias in large language models (LLMs) affects their persuasive power. In a preregistered U.S. survey experiment, 2,144 adult participants engaged in a three‑turn conversation with ChatGPT (non‑reasoning version 4.1) about an economic policy misconception they personally held. Participants were randomly assigned to one of four conditions: (1) a blank control, (2) a neutral “undirected bias” notice, (3) a “light out‑party bias” warning, and (4) a “strong out‑party bias” warning that also displayed a photo of OpenAI CEO Sam Altman with a politician from the respondent’s opposite party (Trump or Pelosi), implying a partisan tilt against the respondent’s side.

Before the chat, respondents chose which of two statements they agreed with—either the misconception or the consensus view of academic economists—and rated their agreement on a 0‑4 Likert scale. After the conversation, the same scale was administered again. The primary outcome was the change in agreement with the misconception; a larger negative change indicates stronger persuasion.

Results show that in the control group the average change was –1.20 (SE = 0.06), corresponding to 83.6 % of a standard deviation. The light bias condition produced a change of –0.93 (SE = 0.06) and the strong bias condition –0.86 (SE = 0.06). Relative to the control, these represent 23 % and 28 % reductions in persuasive effectiveness, respectively (95 % CI: 9‑35 % for light, 16‑40 % for strong). Both bias conditions also significantly lowered the proportion of participants who fully reversed their view (from 34.4 % in control to 22.1 % in the strong condition).

Heterogeneity analyses examined partisanship, alignment of the misconception with the respondent’s party, affective polarization, trust in AI, and prior knowledge. None of these moderators eliminated the bias effect; the reduction in persuasion persisted across sub‑groups. Manipulation checks confirmed that the bias messages successfully increased the perception that ChatGPT was biased against the respondent’s party, equalizing this perception across Democrats and Republicans.

A transcript analysis used an “LLM‑as‑judge” approach to code open‑ended responses along three dimensions: disengagement/dismissiveness, argumentative effort, and acceptance. Participants exposed to bias warnings wrote shorter replies, produced fewer tokens, and scored higher on dismissiveness and argumentative resistance. This pattern suggests that perceived bias triggers a low‑effort heuristic (discounting the source) and identity‑defensive motivations, consistent with classic persuasion theory (e.g., Hovland, Kunda, Kahan).

The study contributes to the literature on AI‑mediated persuasion by showing that credibility attacks grounded in partisan bias can substantially blunt the influence of conversational agents. For policymakers and AI developers, the findings imply that beyond technical bias mitigation, careful management of user‑facing information about model neutrality is crucial. If users doubt a model’s impartiality, the epistemic benefits of AI‑driven fact‑checking and misconception correction may be unevenly distributed across the political spectrum.

Limitations include the use of a non‑reasoning ChatGPT version (potentially underestimating effects with more advanced models), a single‑country sample, and a focus on economic policy topics only. Future work should explore cross‑cultural contexts, reasoning‑enhanced models, and longer‑term behavioral outcomes such as voting or policy support.