LLM debate assessment / 26 October 2026
01 / 09What changed
in the study.
A common direction for candidate differences. Three connected questions about what LLM debate scores mean.
A presentation brief on cumulative revisions from earlier drafts to the current manuscript. The changes include reanalysis, clearer descriptions of existing procedures, and a revised structure for interpreting the findings.
21US general-election debates
12Election cycles
4LLM judge models
3Connected research questions
RQ1 · Association with viewers→RQ2 · Value of combining scores→RQ3 · Sensitivity and repetition
Change 01 / Reanalysis
02 / 09Use one candidate direction throughout.
Candidate A and B previously followed the original input order. The current analysis gives those labels the same party meaning in every debate.
Common analysis labels
A Democratic candidate
B Republican candidate
Democratic − RepublicanScore margin: candidate score difference, in the procedure’s own units.
Poll margin: viewer-poll percentage difference, in percentage points.
The original A candidate was Republican in 8 of the 21 debates. Both score and poll margins were reversed for those eight events.
Every positive margin now favours the Democratic candidate; every negative margin favours the Republican candidate.
Interpretation changedRecoding can change the pooled correlation.
Reversing selected events changes the pooled means and covariance. This is more than replacing the names “A” and “B.”
Qwen3 30B · Whole-debate Overall
0.490→0.172
Pearson r: original mixed order → Democratic-minus-Republican direction.
Existing LLM scores were reused. Correlations were recalculated and ensemble regressions were refitted after recoding. The LLM judges were not rerun for this change.
The original candidate presentation order is retained for sensitivity analysis. Recoding saved scores does not reverse the order the LLM originally saw.
Change 02 / Research purpose and structure
03 / 09Connect every analysis to the research aim.
Examine what LLM-based debate scores reveal about surveyed viewers’ judgments of performance.
RQ1 / Association
What do the scores track?
Compare Debatrix-based whole-debate and paired-turn score margins with the same polls across the same 21 debates.
Include ClaimBuster-based, PROClaim-based, and LLM-POTUS-based scores as supplementary, task-specific comparisons.
Meaning: establish how different assessments relate to perceived debate performance.
RQ2 / Added estimation value
Does combining scores help?
Test whether a ridge ensemble estimates held-out poll margins more accurately than a calibrated mean of the four judges’ whole-debate Overall margins.
Examine whether adding audience-score margins helps.
Meaning: assess the additional value of learning how to combine scores.
RQ3 / Sensitivity and repetition
How much do the findings vary?
Examine how Debatrix score–poll associations change with sample selection, debate grouping, and candidate direction.
Separately, measure score variation and candidate judgments across repeated evaluations of one debate.
Meaning: distinguish sensitivity across debates from variation across repeated runs.
Changes 03 and 08 / Assessment procedures
04 / 09Make the scoring procedures explicit.
03 / Comparison made more prominentWhole-debate and paired-turn scoring
Whole-debate: assess complete candidate text blocks in a joint prompt.
Paired-turn: assess one turn from each candidate, then average scores within each debate.
Both Debatrix-based procedures are compared on the same 21 debates. The paired-turn analysis contains 2,664 turn pairs.
Whole-debate scoring remains primary; paired-turn scoring is the exploratory RQ1 comparison. The debate is the unit used in correlations with polls.
08 / Assessment targets clarifiedDifferent scores answer different questions.
- ClaimBuster-based LLM scoringWhich selected statements are worth fact-checking?
- PROClaim-based support assessmentDo passages from the same debate support or contradict a selected statement?
- LLM-POTUS-based audience appealHow appealing does the model estimate each candidate to be to specified audiences?
These are adaptations of the named frameworks. A weak correlation with an overall-performance poll does not, by itself, show that the intended task was performed incorrectly.
Implementation clarification. Original Debatrix uses chronological analysis and memory updates. The study’s whole-debate implementation unintentionally departed from that procedure. The manuscript now explains the difference and the reasons for retaining whole-debate scoring: fewer model calls and a debate-level output corresponding to the unit of the viewer poll.
Changes 04 and 05 / Ensemble evaluation
05 / 09Give the ensemble a clear test and comparator.
04 / A distinct RQ2Learn how to combine saved scores.
16 inputs4 LLM judges × 4 whole-debate rubric dimensions
+64 optional inputsDetailed LLM-POTUS audience-score margins
Ridge regression estimates the viewer-poll margin. The nested procedure selects the feature pool and penalty within the training data.
The comparator is a calibrated mean of the four judges’ Debatrix-based whole-debate Overall margins, fitted using training data only.
05 / Validation made explicitTwo ways to hold out data
- Primary: election-cycle LOGO-CVLeave one entire election cycle out; fit on the other 11 cycles. Repeat for all 12 cycles.
- Secondary: debate-level LOOCVLeave one debate out; fit on the other 20. Repeat for all 21 debates.
- Nested selectionPreprocessing, calibration, coefficient fitting, and model selection use training subsets only.
LOGO-CV prevents debates from the same election cycle appearing in both training and test sets.
Nested ridge ensemble versus the calibrated-mean baseline| Validation design | Change in mean absolute error | Interpretation |
|---|
| LOGO-CV · primary | 0.287 lowerPoll percentage points | A small descriptive improvement. |
| LOOCV · secondary | 1.220 higherPoll percentage points | The ensemble did not improve on the baseline here. |
Changes 06 and 07 / Variation and uncertainty
06 / 09Look beyond a single correlation or significance label.
06 / Repeated scoringA 30-seed design for one debate
The selected event is the 29 September 2020 Biden–Trump debate. The design covers five LLM assessment procedures, separately from the cross-debate analyses.
- Score variationMean, sample standard deviation (SD), and observed range.
- Candidate-direction countsRuns favouring the Democratic candidate, the Republican candidate, or neither.
- Execution periodsWithin-batch variation and differences between batches are distinguished.
A score can correlate with polls across debates while the higher-scored candidate changes when one debate is assessed repeatedly.
07 / Statistical emphasisEmphasize estimates and sensitivity.
Interpretation now focuses on the magnitude of associations, uncertainty intervals, and changes under alternative analyses.
- Delete individual debates or election cycles.
- Resample groups defined by election cycle or recurring candidates.
- Compare original input order with Democratic-minus-Republican margins.
- Keep raw p values and BH-adjusted q values, with the full details in appendices.
BH adjustment is supporting information. Counting “significant” results is not the study’s main objective.
Coverage and interpretation. Complete paired-turn 30-run summaries are available for three judges; the DeepSeek-R1 32B summary is unavailable in the current manuscript. Different scales, generation settings, and averaging rules prevent a direct reliability ranking based on SD alone.
The findings to emphasize
07 / 09Three results, three different meanings.
RQ1
Correspondence with viewers depends on the judge and task.
Both Debatrix-based procedures had positive sample correlations with polls, but their magnitudes and uncertainty varied. Paired-turn correlations were higher in seven matched model–dimension comparisons and lower in nine. Neither procedure showed a uniform advantage.
RQ2
Combining scores does not guarantee better estimation.
The nested ensemble improved slightly on the calibrated-mean baseline under election-cycle validation, but had higher error under debate-level validation. The value of combining scores depends on the comparator and validation design.
RQ3
Repeatability and agreement across judges are different.
In whole-debate repetitions, GPT-OSS and Qwen changed which candidate received the higher Overall score. Each of the three judges with complete paired-turn summaries kept one direction, but they favoured different candidates. Stable repeated outputs can therefore coexist with disagreement across models.
Suggested presentation summary
“We standardized candidate differences and connected the analyses through three research questions: what the scores reveal about viewer judgments, whether combining scores improves held-out estimates, and how the findings vary with analysis choices and repeated scoring.”
Change 09 / Traceability and reproducibility
08 / 09Make the sample, sources, and reproducible work visible.
Document the final 21-debate sample.
The manuscript explains how the working corpus was restricted to two-candidate general-election debates with a viewer poll for the same event.
The final sample contains 17 presidential and 4 vice-presidential debates, spanning 1976–2024 and 12 election cycles.
Appendices identify the debates, candidates, parties, dates, selected poll percentages, and source links. Original-order results are retained for comparison.
Separate statistical reproduction from fresh inference.
The versioned N=21 analysis package contains stored scores, poll values, candidate metadata, and scripts for recalculating the statistical analyses.
A companion archive records repeated-run outputs and coverage.
Authorized repository users can reproduce the analyses from stored outputs. A complete fresh-inference workflow and public access remain under preparation.
Current manuscript status · 26 October 2026
- Updated in Overleaf
- Introduction, Literature Review, Methods, Results, Discussion, and related appendices and figures.
- Kept unchanged
- Abstract, Conclusion, and existing professor comments.
- Presentation note
- Use the updated body for this briefing. The unchanged Overleaf Abstract and Conclusion have not yet been synchronized with the latest framing.
Additional experiments / Recorded progress
09 / 09Four workstations were running additional evaluations.
Snapshot: 25 October 2026, 8:00 pm. All four workstations were reported active. PCs 276 and 440 had paused after GPU conflicts were detected; they resumed with the same settings, and their successful-completion counts were increasing again.
Recorded progress and remaining-time estimates at the snapshot| PC | Task | Progress | Estimated time remaining |
|---|
| 276 | Gemma 4 · Turn-by-turn20 additional debates | 0.67% | About 15 daysAt the workstation’s solo processing rate |
| 440 | GPT-OSS 20B · Turn-by-turn20 additional debates | 1.88% | About 10.5 daysAt the workstation’s solo processing rate |
| 424 | DeepSeek · 10 additional seeds | 83.9%19,630 / 23,400 completed | About 15 hours 20 minutesEstimated finish: 26 October, around 11:20 am |
| 450 | Whole-debate scoringCurrently processing DeepSeek at the snapshot | 30.4%766 / 2,520 completed | About 14 daysDisplayed estimate; highly variable |
Planned allocationCompleted workers join the longer jobs.
PC 424 → PC 276’s workload after the additional DeepSeek seed runs finish.
PC 450 → PC 440’s workload after whole-debate scoring finishes.
Both reassignment plans remained scheduled at the snapshot.
Dependencies and failuresTwo LOO jobs await whole-debate outputs.
PC 440’s two LOO jobs were waiting for whole-debate scoring to finish.
0 recorded failures across the four active jobs at the snapshot.
How to read the estimates. PC 450’s estimate applied DeepSeek’s slow processing rate to the remaining models, so its accuracy was low. The PC 276 and 440 estimates were taken shortly after restart and before other workstations joined their workloads.