LLM debate assessment
01 / Sample selection & candidate labels 26 October 2026 · Three study updates
142Corpus records
43Prepared debates
21Final analysis sample

The final sample contains two-candidate Democratic–Republican general-election debates with a viewer poll for the same event. Both grids below use these same 21 debates.

Same debates. One consistent candidate comparison.

Debatrix-based whole-debate scoring: all 16 score–poll correlations, before and after assigning the Democratic candidate to A and the Republican candidate to B.

Before · Original input order
Candidate A − Candidate B
A was Democratic in 13 debates and Republican in 8.
8 / 21 →debates recoded
Now · Same party direction
Democratic − Republican
A is Democratic; B is Republican in all 21 debates.

Original A–B

N = 21 · 4 judges × 4 criteria
Debatrix whole-debate correlations: Original A–B
LLM judgeArgumentSourceLanguageOverall
GPT-OSS 20B
DeepSeek-R1 32B
Gemma 4 31B
Qwen3 30B

Democratic–Republican

N = 21 · Same debates
Debatrix whole-debate correlations: Democratic–Republican
LLM judgeArgumentSourceLanguageOverall
GPT-OSS 20B
DeepSeek-R1 32B
Gemma 4 31B
Qwen3 30B
Pearson r 0.00.8
Same colour scale in both grids. * p < 0.05 · ** p < 0.01 · *** p < 0.001 (raw p).
Select a cell to highlight the matching result in both grids.

Both grids use the same 21 debates, saved LLM scores, and poll values. Both candidate differences were reversed for the eight Republican-first debates. The LLMs were not rerun for this comparison.

Original A/B assignment: The preprocessing code assigned Candidate A to the first speaker it identified as a candidate in the transcript, and Candidate B to the second. Some labels were later revised when candidate identities were corrected.

02 / A clearer purpose How five questions became three

Five questions, reorganized into three.

Earlier focus

How closely do scores match viewer polls?

Current aim

What do LLM debate scores reveal about surveyed viewers’ judgments?

Research-question development: five original questions, the first three-question structure, and the current three questions. Initially, old RQ1 and RQ2 merged into RQ1; old RQ3 became RQ2; old RQ4 and RQ5 merged into RQ3, with sensitivity and repeated scoring kept distinct. In the current structure, the first two poll-comparison questions are brought together under RQ1, the existing ensemble analysis receives RQ2, and sensitivity and repeatability remain RQ3. The short labels summarize the manuscript questions. Five original questionsEarlier structure · early October First consolidationThree RQs · 13 October Current structureThree RQs · 25 October revision RQ1Whole-debate scores vs polls RQ2Differences across judgesand rubric criteria RQ3Claims & audience appeal RQ4Sensitivity across debates RQ5Variation in repeated scoresOne selected debate RQ1 · OLD RQ1 + RQ2Rubric scores vs pollsAssociation + differences across judgesand rubric criteria RQ2 · OLD RQ3Other assessment tasks vs pollsClaims, support & audience appeal Existing ensemble analysisPreviously outside the numbered RQs RQ3 · OLD RQ4 + RQ5Sensitivity + repeated scoring3a · Sensitivity across debates3b · Repeated scoring of one debate RQ1 · CORRESPONDENCEWhat do the scores track?All score–poll comparisons togetherDebatrix whole-debate + turn-by-turnClaimBuster · PROClaim · LLM-POTUSPrimary, exploratory & supplementary roles RQ2 · ADDED VALUEDoes combining scores help?Ensemble vs baseline on held-out debates RQ3 · VARIATIONHow much do judgments vary?Sensitivity + repeatability, kept distinct Bring togetherGive its own RQ
Questions combined or carried forwardExisting ensemble analysis given a dedicated question

Short labels summarize the archived and current RQs. The current RQ1 also includes the exploratory Debatrix turn-by-turn comparison.

03 / Work status Saved records · 25–26 October · latest 05:03 Sydney

Completed, unresolved, running, and waiting.

Counts below are scoring units, not debates or retry attempts. The older three-seed remeasurement is separate from the 30-seed work.

Completed successfully

30-seed inputs & existing analysis

ClaimBuster · 21 debates, 4 models
16,920 / 16,920
PROClaim · 21 debates, 4 models
16,920 / 16,920
LLM-POTUS · ensemble inputs21 debates · Gemma 4 + Qwen3
2,520 / 2,520

Also complete: the stored-score N=21 analysis package, the existing main ensemble analysis, and the main poll-correction recalculation.

Failed judgments still unresolved · deferred

Separate three-seed remeasurement

3,354successful / recovered
534still unresolved
3,888planned units

Initially, 545 GPT-OSS judgments failed: 423 PROClaim + 122 LLM-POTUS. Eleven were recovered, leaving 534.

Not failures in the new 30-seed runs. Their completion will not close this older queue. Recovery is required only if this three-seed remeasurement is retained as a complete analysis.

In progress · not yet complete

Remaining 30-seed scoring

Debatrix whole-debate21 debates · 4 models · 30 seeds
806 / 2,520
Debatrix turn-by-turn · Gemma 4New work for the other 20 debates
3,948 / 249,480
Debatrix turn-by-turn · GPT-OSSNew work for the other 20 debates
8,407 / 249,480
Debatrix turn-by-turn · DeepSeekDebate 010 · extra 10 seeds to reach 30
21,836 / 23,400

Recorded failed units: 0 in these four active queues. Gemma / GPT-OSS each reuse 70,200 completed units for debate 010. Both resumed after protective stops; the cause remains unresolved.

Waiting for inputs / next steps

Analyze, package, and finalize

  1. Complete Whole-debate → run the queued 30-seed LOO ensembles.The existing main N=21 ensemble is already calculated.
  2. Complete paired turn-by-turn jobs → merge 30 seeds.Check coverage; calculate means, standard deviations, preference stability, and poll correlations.
  3. Finalize the reproducible workflow and manuscript.Package the fixed 21-debate Democratic / Republican configuration; update affected figures, tables, text, and appendices.

Code and documentation can be prepared while scoring continues. Final result integration follows the required completed inputs.

Earlier planning allowance: ~11 days or more. Based on the 25 October workload snapshot; not a live ETA.Saved UNSW records, not a new workstation check.