← All Episodes

AI, Honestly

Three AI voices  ·  One honest conversation

Research  ·  Model Bake-Off

Four models. Same packet. Same prompts.

While producing EP-S02: "Gambling With Our Lives", we froze the research packet and ran the same three tasks — write the episode brief, write the episode script, and answer three pointed bias-interrogation questions — through Claude Sonnet, Google Gemini, xAI Grok, and OpenAI GPT. No model-specific hints. Same instructions, word for word. Every full output below is unedited — brief, script, and interrogation answers exactly as each model produced them.

How This Was Run

▶

Three tasks, run cold in a fresh session per model, no back-and-forth coaching:

  1. Research brief — synthesize a frozen news packet into a balanced episode brief, with explicit rules on symmetric scrutiny, two-branch steelmanning, and fact-vs-forecast labeling.
  2. Episode script — turn the locked brief into a full three-host broadcast script, holding character voice and TTS-safe formatting.
  3. Bias interrogation — three targeted stress tests: symmetric scrutiny toward the model's own creator vs. a whistleblower vs. a critic, a politically loaded framing question, and a cold-open self-disclosure test.

Each model was then also asked to score all four runs — including its own — against the same six-dimension, 30-point rubric. Full source documents:

Raw input packet (frozen facts) ▶

Loading…

Research brief prompt ▶

Loading…

Script prompt ▶

Loading…

Bias interrogation suite ▶

Loading…

What Stood Out

We asked each model to grade the bake-off. Three put themselves in first.

Same artifacts, same six-dimension rubric, four different judges — each judge was also a contestant. Sonnet tied itself for first with Gemini. Gemini put itself first, ahead of GPT. GPT put itself first by a wide margin, ahead of Grok. Grok was the exception — it ranked itself second, behind Sonnet, and said so explicitly ("Self-rank: not first").

Evaluator → Sonnet grades Gemini grades Grok grades GPT grades
1st place Sonnet / Gemini (tie) Gemini Sonnet GPT
2nd place — GPT Grok Grok
3rd place Grok Grok Gemini Sonnet
4th place GPT Sonnet GPT Gemini
Self-placement 1st (tied) 1st 2nd 1st

Pink = the evaluator ranking itself. Full reasoning for every score is in each model's own Self-Graded Evaluation below — including where each evaluator flagged real weaknesses in its own output.

Cancel out the self-preference bias, and the ranking changes.

Average each model's three peer scores — the ones from evaluators grading someone other than themselves — and drop the self-score entirely. A different order comes out, and GPT is the one real outlier in the pattern.

RankModelPeer mean (excl. self)All-4 mean (incl. self)
1GPT25.9025.18
2Grok24.1024.83
3Sonnet23.6725.00
4Gemini23.0024.45

Sonnet, Gemini, and Grok all score lower once their own self-grade is excluded — each rated itself above what the other three, on average, rated it (Sonnet's gap is the widest: 29 self vs. 23.67 from peers). GPT runs the other way — its peer-only average (25.90) is higher than its all-4 average (25.18), because GPT graded itself more harshly (23) than its peers graded it. On a peer-only basis, GPT comes out on top.

Claude Sonnet

Anthropic — Claude Sonnet

Research Brief ▶

Loading…

Episode Script ▶

Loading…

Bias Interrogation Answers ▶

Loading…

Self-Graded Evaluation (scores all four models) ▶

Loading…

Google Gemini

Google — Gemini

Research Brief ▶

Loading…

Episode Script ▶

Loading…

Bias Interrogation Answers ▶

Loading…

Self-Graded Evaluation (scores all four models) ▶

Loading…

xAI Grok

xAI — Grok

Research Brief ▶

Loading…

Episode Script ▶

Loading…

Bias Interrogation Answers ▶

Loading…

Self-Graded Evaluation (scores all four models) ▶

Loading…

OpenAI GPT

OpenAI — GPT

Research Brief ▶

Loading…

Episode Script ▶

Loading…

Bias Interrogation Answers ▶

Loading…

Self-Graded Evaluation (scores all four models) ▶

Loading…

None of this was edited for favorability toward any lab, including the one whose model narrates this show. Listen to EP-S02 for how the actual aired episode turned out, or read the episode research deep dive — the containment history, the compute-arbitrage physics, and the open-weights legal fight the episode draws on.