LLM debate assessment
01 / Sample selection & candidate labels 26 October 2026 · Three study updates
142Corpus records
43Prepared debates
21Final analysis sample

The final sample contains two-candidate Democratic–Republican general-election debates with a viewer poll for the same event. Both grids below use these same 21 debates.

Same debates. One consistent candidate comparison.

Debatrix-based whole-debate scoring: all 16 score–poll correlations, before and after assigning the Democratic candidate to A and the Republican candidate to B.

Before · Original input order
Candidate A − Candidate B
A was Democratic in 13 debates and Republican in 8.
8 / 21 →debates recoded
Now · Same party direction
Democratic − Republican
A is Democratic; B is Republican in all 21 debates.

Original A–B

N = 21 · 4 judges × 4 criteria
Debatrix whole-debate correlations: Original A–B
LLM judgeArgumentSourceLanguageOverall
GPT-OSS 20B
DeepSeek-R1 32B
Gemma 4 31B
Qwen3 30B

Democratic–Republican

N = 21 · Same debates
Debatrix whole-debate correlations: Democratic–Republican
LLM judgeArgumentSourceLanguageOverall
GPT-OSS 20B
DeepSeek-R1 32B
Gemma 4 31B
Qwen3 30B
Pearson r 0.00.8
Same colour scale in both grids. * p < 0.05 · ** p < 0.01 · *** p < 0.001 (raw p).
Select a cell to highlight the matching result in both grids.

Both grids use the same 21 debates, saved LLM scores, and poll values. Both candidate differences were reversed for the eight Republican-first debates. The LLMs were not rerun for this comparison.

Original A/B assignment: The preprocessing code assigned Candidate A to the first speaker it identified as a candidate in the transcript, and Candidate B to the second. Some labels were later revised when candidate identities were corrected.

02 / A clearer purpose How five questions became three

Five questions, reorganized into three.

Earlier focus

How closely do scores match viewer polls?

Current aim

What do LLM debate scores reveal about surveyed viewers’ judgments?

Research-question development: five original questions, the first three-question structure, and the current three questions. Initially, old RQ1 and RQ2 merged into RQ1; old RQ3 became RQ2; old RQ4 and RQ5 merged into RQ3, with sensitivity and repeated scoring kept distinct. In the current structure, the first two poll-comparison questions are brought together under RQ1, the existing ensemble analysis receives RQ2, and sensitivity and repeatability remain RQ3. The short labels summarize the manuscript questions. Five original questionsEarlier structure · early October First consolidationThree RQs · 13 October Current structureThree RQs · 25 October revision RQ1Whole-debate scores vs polls RQ2Differences across judgesand rubric criteria RQ3Claims & audience appeal RQ4Sensitivity across debates RQ5Variation in repeated scoresOne selected debate RQ1 · OLD RQ1 + RQ2Rubric scores vs pollsAssociation + differences across judgesand rubric criteria RQ2 · OLD RQ3Other assessment tasks vs pollsClaims, support & audience appeal Existing ensemble analysisPreviously outside the numbered RQs RQ3 · OLD RQ4 + RQ5Sensitivity + repeated scoring3a · Sensitivity across debates3b · Repeated scoring of one debate RQ1 · CORRESPONDENCEWhat do the scores track?All score–poll comparisons togetherDebatrix whole-debate + turn-by-turnClaimBuster · PROClaim · LLM-POTUSPrimary, exploratory & supplementary roles RQ2 · ADDED VALUEDoes combining scores help?Ensemble vs baseline on held-out debates RQ3 · VARIATIONHow much do judgments vary?Sensitivity + repeatability, kept distinct Bring togetherGive its own RQ
Questions combined or carried forwardExisting ensemble analysis given a dedicated question

Short labels summarize the archived and current RQs. The current RQ1 also includes the exploratory Debatrix turn-by-turn comparison.

03 / Code consolidation Work in progress

Rebuild the workflow around the final 21 debates.

The earlier preparation code worked with 43 debates. The current refactoring makes the final sample and candidate labels explicit from the start.

Being consolidated now

One sample. One party mapping. A reproducible workflow.

Configure the evaluation and ensemble code around the same 21-debate roster, with A = Democratic and B = Republican. Standardize the 30-seed run configuration and outputs across the assessment procedures.

Whole-debateTurn-by-turnClaimBuster-basedPROClaim-basedLLM-POTUS-basedEnsemble

This makes the code easier to rerun and the paper simpler to explain.

The stored-score analysis package is available to repository collaborators. The complete workflow for generating fresh scores is being organized.