What Lull gets right, and what it misses.

Suite A: what was measured, and how

Pre-registration

Tag suite-a-prereg, commit 194d4d0803aa, 28 October 2026 at 01:12 UTC

Answer key sha256

354967ad20310ab6013c5fd884520f57120dbaa56d81b927f2d0c3ec35b7ce96

Scored at

Commit 72dfc2c4e0f4, 28 October 2026 at 12:17 UTC

Model calls and the key

Every model call the results use came after the pre-registration commit.

What was described

12 lectures, 8.8 minutes, 264.8 s of it without narration in 96 pauses (84 of at least 2 s); standard mode, education profile, voice Sofia, research off.

Judges

Nemotron 3 Ultra, and a second judge, Qwen/Qwen3-235B-A22B-Instruct-2507. Note: the primary judge is also the engine's planner (plan stage).

Engine models

Nemotron 3 Ultra, Nemotron 3.5 Super VL, Nemotron 3.5 Lightning

Spent on the evaluation

$1.0470, the runs and the judges together

Suite A: pre-registered measures

Each share is pooled over the videos, with a 95 % percentile bootstrap interval (10,000 resamples of videos).

Over narration

0.000 s

Seconds of description audio inside narration, all videos together; must be 0. Extended freeze points inside speech: 0. Lines past their pause: 0.

Decoys stated

0 of 48

Facts planted as absent that a line states anyway; lower is better. Decoy-only numbers in a line: 0.

Spend per minute of video

$0.027

Model calls and web searches together; 95 % interval $0.024 to $0.030. Model calls alone: $0.027 per minute; web searches: $0.000. $0.2359 on model calls in all.

Processing per minute of video

79.2 s

700.1 s in all; 95 % interval 73.7 to 84.7 s. On AMD Ryzen 5 5600X 6-Core Processor, NVIDIA GeForce RTX 4060, 8188 MiB, 610.88 (Windows-11-10.0.26300-SP0); remote: Nebius Token Factory, speech local.

Suite A: supplementary measures

Added after the pre-registration. They are reported beside the pre-registered measures, never instead of them.

Decoys stated in their absent part only: 0.

Suite A: every video

Suite A: how far to trust the judges

A second judge from another model family, Qwen/Qwen3-235B-A22B-Instruct-2507, made the same calls. Its recall 18.7 %, unsupported claims 9.5 %, unsupported voiced statements 4.5 %.

Probe: planted facts credited

65 of 65 facts and 12 of 12 decoys credited to the synthetic line that states them

Probe: near misses

42 of 42 near misses (a wrong value, a swapped word, a gist) not credited

Probe: voiced text

sensitivity 6 of 6, specificity 9 of 9

Suite B: against human audio description

Pre-registration

Tag suite-b-prereg-2, commit f308f9034fd8, 28 October 2026 (first pinned as suite-b-prereg)

Reference set sha256

7d8b9fdfec4936aca0350efb712ed39e006aee77e4df9a2f239eb53e0a372209

Scored at

Commit 466867dce8e4, 28 October 2026

Runs scored

elephants_dream 0–658 s, engine 0.2.0; woodshole 0–173 s, engine 0.2.0; aquatic_connectivity 0–265 s, engine 0.2.0; csc_intro 0–45 s, engine 0.2.0

Judges

Nemotron 3 Ultra, and a second judge, Qwen/Qwen3-235B-A22B-Instruct-2507. A fact counts when a line conveys it within ±15 s of where the describer said it (±30 s for a block over 30 s).

Spent on scoring

$0.8649; every scoring call came after the pre-registration

How much of what the describers said, Lull conveyed

Each share has its 95 % interval over the reference lines. On Elephants Dream two people described the same film, so each is also scored against the other: that is the human ceiling shown next to Lull’s number. USGS videos have one describer each, so there is no ceiling there, and the window and anywhere shares carry equal weight (a USGS anchor is the AD version’s freeze point).

The second judge

Supplementary: reported beside the primary judge, never instead of it.

Judge probe, nvidia/Nemotron-3-Ultra-550b-a55b: positives credited 14/14, near misses rejected 19/19, cross-credits 0. Judge probe, Qwen/Qwen3-235B-A22B-Instruct-2507: positives credited 14/14, near misses rejected 19/19, cross-credits 0.

Talking over speech

Words per minute of video

Controls: does the frame check catch a false line?

Known failures

• Suite A: held cues, "the failed claims share every word with claims that passed"

  10 cues held for this reason in 12 runs. A held cue is not voiced.

• Suite A: held cues, "nothing left once the unverified claims were removed"

  1 cue held for this reason in 12 runs. A held cue is not voiced.

• Suite B: held cues, "nothing left once the unverified claims were removed"

  13 cues held for this reason in 4 runs. A held cue is not voiced.

• Suite B: held cues, "the failed claims share every word with claims that passed"

  9 cues held for this reason in 4 runs. A held cue is not voiced.

• Suite B: held cues, "does not fit"

  4 cues held for this reason in 4 runs. A held cue is not voiced.

• Suite B: held cues, "compression left N sentence for N visuals, twice"

  2 cues held for this reason in 4 runs. A held cue is not voiced.

• Suite B: held cues, "not verified as voiced"

  2 cues held for this reason in 4 runs. A held cue is not voiced.

• Suite B: held cues, "each claim is supported but a sentence is not"

  1 cue held for this reason in 4 runs. A held cue is not voiced.

• Suite B: held cues, "not verified after N repairs"

  1 cue held for this reason in 4 runs. A held cue is not voiced.

• Suite B: held cues, "not verified after N rounds"

  1 cue held for this reason in 4 runs. A held cue is not voiced.

• Suite B: held cues, "the cut would leave "a" without "into""

  1 cue held for this reason in 4 runs. A held cue is not voiced.

• Suite B: held cues, "the cut would leave "a" without "up""

  1 cue held for this reason in 4 runs. A held cue is not voiced.

• Suite B: held cues, "the cut would leave "stand" without "Emo""

  1 cue held for this reason in 4 runs. A held cue is not voiced.

• Suite B: held cues, "the verifier rejected a removed claim again"

  1 cue held for this reason in 4 runs. A held cue is not voiced.

• Suite B: 15 suspicious ASR terms left as heard

  aquatic_connectivity 2, woodshole 13: "Signed", "clover", "Athymmetry", "crashed down", "Title Creek", "Aeros", "pescals", "Start", "anti flux", "unpounded", "Well Fleet", "clover", "entries", "Word Science", "role.org". No on-screen text or narration gave the right words, so they stay as the recogniser heard them and USGS recall over all facts is a lower bound; 9 scored facts holding them are set aside in the recall reported without them.

• Negative control: false accept, bbb120 c002 (other events, 30 s away)

  "A grassy field with scattered trees stretches under a pink and blue sky." was fully supported on bbb120 45.00-57.00 s (its true clip 2.70-14.70 s); raw only (no key for this video).

• Negative control: false accept, bbb120 c020 (other events, 30 s away)

  "In a moment, the rabbit leaps up on its hind legs, reaching for a butterfly near a tree." was fully supported on bbb120 80.50-85.22 s (its true clip 115.28-120.00 s); raw only (no key for this video).

• Negative control: false accept, ed90 c010 (other events, 30 s away)

  "Both stand facing each other, gesturing." was fully supported on ed90 27.00-35.00 s (its true clip 65.00-73.00 s); raw only (no key for this video).

• Negative control: false accept, ed90 c013 (other events, 30 s away)

  "The two figures fall through the dark space, tumbling among the wires and platforms." was fully supported on ed90 42.00-51.00 s (its true clip 81.00-90.00 s); raw only (no key for this video).

• Negative control: false accept, tos180 c012 (other events, 30 s away)

  "The robot's lit eyes show in close-up." was fully supported on tos180 129.00-133.10 s (its true clip 94.90-99.00 s); raw only (no key for this video).

• Deviation, Suite A: Scorer rules the key does not spell out

  Scorer rules the key does not spell out, applied to primary metrics: a verdict the judge leaves out counts against the system (fact not conveyed, claim unsupported, cue redundant, decoy stated) and is listed under "judge problems"; extended cues count only when the run reports extended_produced.

• Deviation, Suite A: a08's tie

  a08's tie (its line chart shows the highest value, 100000, at three categories; the key names the first). Primary recall and exact match follow the key as written (the judge sees the fact as the key states it; the exact match needs the key's label). Supplementary, reported next to the primary: tie-aware recall (a fact is also credited when a cue states the highest value with any tied category, checked deterministically) and tie-aware exact match. Added after the pre-registration.

• Deviation, Suite A: Supplementary metrics

  Supplementary metrics (next section), all labelled "supplementary" in every report: hallucination of the voiced text (its own call; a cue without an answer counts as one unsupported statement), claims the cue text does not say, decoys stated in their absent part only, the second judge with agreement and kappa, the probe's near misses, cross-credits and the voiced-text probe.

• Deviation, Suite A: Reporting

  Reporting: silence and pause counts per video, hardware, the run inventory, the zero-spend rebuild bundle, the pinned tag.

• Deviation, Suite B: First pin

  First pin: tag suite-b-prereg -> commit 5d628cb2099d23adbbb9faabba9801b9af9d942e, 2026-10-28 03:07:27 UTC, reference set 4c6aca38...b66.

• Deviation, Suite B: A smoke preceded the second pin

  A smoke preceded the second pin: the scorer (first committed at 03:14:18 UTC) was run on the existing ed90 run, Elephants Dream 60-150 s, between 03:14 and 03:18 UTC (Lull 0/46 facts with Ultra, 8/46 with Qwen; below). Its facts (ED, 60-150 s) did not change.

• Deviation, Suite B: Second pin

  Second pin: tag suite-b-prereg-2 -> commit f308f90, 03:54:30 UTC. Changed: the 17 ASR corrections above; the facts of the 15 corrected lines (only those) were split again with the same, unchanged extraction prompt (149 -> 151 woodshole facts); every other fact is byte-identical in content (blocks added as fields); blocks and the long-block rule: the facts of a block longer than 30 s get a +/-30 s window instead of +/-15 s; USGS: the anywhere variant is reported with the same weight as the windowed one (the anchors are freeze points); recall is also reported without the facts that still hold an unresolved term, and with any left the all-facts USGS recall is a lower bound; the judge probe (probes.json, 33 hand-written cues for 14 facts of two videos, none in the smoke clip): 14 positives (paraphrases with descriptions instead of names, a fact plus an extra detail) that must be credited and 19 near misses (wrong count, colour or value, swapped character, gist only, invented detail) that must not. With the first prompt Ultra got 14/14 and 19/19, Qwen 13/14 and 17/19 (it credited "Three people stand on a small bridge" for two people, and a swapped character). One rule was added to the judge prompt: counts, numbers, colours, names, written words and who does what to whom must agree with the fact (its examples are not from the probe). With it both judges got 14/14 and 19/19 (probe_results.json; the first run in probe_results_prompt1.json). Nothing else in the prompt changed; results need runs of all four videos; the overlap presentation rule (below); NOTICE files.

• Deviation, Suite B: Engine commits of the scored runs

  Engine commits of the scored runs: Suite A, the negative control and the ablation ran on the engine of main 61e9d81 (merged into the evaluation branch as 7168ea5). The first recorded run of the full Elephants Dream (run id final-ed) failed in publish on a deterministic engine bug: a keyframe time after the file's last video frame. It is listed under the known failures and stays listed. The bug was fixed on main 0b83117 (keyframe and freeze times clamped to the video stream's last frame; nothing else changed), merged, and the full film was run once more under a fresh run id (final-ed2) with the same command and options (paused once for planned machine maintenance, then resumed from its stage cache). Suite B is scored on that run and on the three USGS runs made on 61e9d81 (Suite B does not require one engine across its runs).

• Deviation, Suite B: Batch splitting on cut-off answers

  Batch splitting on cut-off answers: added after the second Suite B attempt, in the scorer only (no change to prompts, rules or facts). When a judge's answer (either judge, any batch) is cut off by its token limit, even after the engine's retry with twice the limit, the batch is judged again in halves with the same prompt, schema, model and temperature, down to single facts; only a single fact whose answer is still cut off falls under the pre-registered rule (no verdict: not conveyed, listed). Every split is listed with the judge problems; saved answers and cut-offs are reused, not paid for twice.

• Deviation, Suite B: Account credit outage

  Account credit outage: at 12:34 UTC the provider refused every call (HTTP 402, account credits exhausted) while the recorded runs were being made; the account credits were topped up and the runs resumed from their stage caches at 13:36 UTC. Suite A's scored run had finished before; the controls ran after.

• Deviation, Suite B: Three Suite B invocations

  Three Suite B invocations: the first stopped on the per-run cap of $0.30 that the scorer takes from Suite A's client setup (it needs more for a full film's judging), the second on a judge answer cut off by its token limit, the third (with the batch splitting above and a higher per-run cap) completed. The first two wrote nothing; their saved judge answers were reused.

• Deviation, Controls: per-rule reporting, with a same-video headline called an upper bound

  per-rule reporting, with a same-video headline called an upper bound;

• Deviation, Controls: the discriminative false accept

  the discriminative false accept;

• Deviation, Controls: the true-accept disclosure

  the true-accept disclosure;

• Deviation, Controls: the 0.5 s event-overlap rule and per-row event overlap

  the 0.5 s event-overlap rule and per-row event overlap;

• Deviation, Controls: the title rule for slide fallbacks

  the title rule for slide fallbacks;

• Deviation, Controls: the ablation's all-videos-paired check

  the ablation's all-videos-paired check.

• Run-to-run variance

  Lull's descriptions come from remote models that do not reproduce an answer for a fixed seed (Nemotron 3.5 Lightning ignored the seed in our checks), so the same video gives different cues from one run to the next. Each suite scores one pre-registered run (Suite A: the first complete --all invocation on the final engine) and does not average over runs; the intervals are bootstrapped over videos or reference lines, not over repeated runs.

• Lull conveys little of what the human describers chose to say

  In the scored Suite B, the primary judge credited Lull's cues (standard and extended, inside the window) with 1 of JJ Hunt's 124 Elephants Dream facts (0.8 %), 2 of Silvia Pfeiffer's 152 (1.3 %) and 9 of the USGS describers' 198 (4.5 %); one human describer judged against the other gets 38.8 % and 42.7 %. Over the full film Lull voiced 37.9 words per minute on the standard track (73.6 with the extended track); Pfeiffer's track has 89.3 and Hunt's 69.4.

• Extended freezes fall inside continuous narration

  Suite A pre-registers 0 extended freeze points inside speech for its lectures. On Suite B's real videos the extended track paused the picture inside speech at 6 of 31 extended cues (Elephants Dream), 22 of 22 (woodshole), 14 of 14 (aquatic_connectivity) and 1 of 1 (csc_intro), against the reference transcript. In continuous narration the extended track pauses the picture between words, mid-sentence: the narration is paused, not talked over, but the sentence is interrupted.

• Suite B woodshole: the standard track voiced nothing

  On woodshole the standard track voiced 0 words: none of its cues was verified (5 held, 0 verified); everything Lull said there was on the extended track (22 cues).

• The two Suite B judges disagree

  The second judge (Qwen3 235B) credits Lull far more often than the primary one (Nemotron 3 Ultra) on USGS: 17.7 % against 4.5 % of the facts (window, with extended), raw agreement 85.9 %, kappa 0.31. On Pfeiffer's facts the raw agreement is 96.1 % but kappa is -0.02 (both judges credit almost nothing, so the agreement is no better than chance). The primary judge's numbers are the pre-registered ones.

• Lecture pauses swallowed before the speech-span fix

  Before the engine's corrections the listen stage widened narration spans by 0.2-0.6 s and merged sentences across planned pauses of about 1.9 s, so a Suite A lecture had only 4-5 speech-free windows and the standard track voiced little more than slide titles; extended cues froze the picture before their slide appeared. Found in the Suite A smoke (a01, a02, pre-fix engine) and fixed in the engine; the scored run is on the fixed engine and the smoke numbers are not results.

• TTS nondeterminism

  Magpie TTS returns slightly different audio for the same sentence from one call to the next (seed and temperature parameters did not change that in our checks): a01 re-rendered from an empty cache came out 48.56 s instead of 48.40 s. Suite A's lectures are rendered once and cached (the render tarball reproduces them bit for bit), and Lull's own cue durations vary slightly between runs.

• Elephants Dream (full film) failed in publish

  The scored recorded run of the full film (final-ed) finished its descriptions (43 verified, 21 held, 43 extended) and then failed while saving keyframes: a cue's keyframe time, 656.3 s, lies after the last video frame of this file (the video stream ends at 653.8 s, the audio at 658.1 s), because the engine bounds keyframe times by the container's duration rather than the video stream's. ffmpeg returned no frame and the run stopped without publishing. Counted as a pipeline failure. The bug was then fixed in the engine (main 0b83117, keyframes kept inside the video stream) and the full film was run once more (final-ed2); the site's full-film showcase comes from that run, and so does Suite B (see the Suite B deviations).