Structure-aware evaluation

• Judge: an independent audio-understanding model, Gemini-2.5 (audio-native). It is a different model family from the PE-A-Frame encoder used inside SemanticAudio, so the evaluation is not circular.
• Input: the clip audio (and, for editing, the source + edited clips) plus the text prompt. The judge hears the audio directly — no reference to how the audio was produced, and clips are unlabeled by method.
• Transcribe-then-score: the judge first lists the sound events it hears with onset/offset timestamps, then scores — reducing hallucinated judgments and giving an audit trail (see the event transcripts in results_taskA.csv).
• Metrics. Generation: Temporal-Order Match (0–1, does the event order match the prompt), Temporal-Order Accuracy (fraction of clips fully correct), event recall/precision, and holistic Alignment (1–5). Editing: edit-success, non-target preservation, temporal preservation, naturalness (1–5).
• Aggregation: each clip is judged 3× and averaged; temperature 0–0.2, strict-JSON output.
• Full prompts & raw outputs: the exact judging prompts are in eval/LABELING_PROMPTS.md; all per-clip JSON in eval/raw/.

SemanticAudio attains the best temporal ordering and alignment of all systems, and improves temporal-order match over the parameter-matched Base Model from 0.64 to 0.86. Sanity check: ground-truth audio scores highest under the same judge (order match 0.97, alignment 4.83), confirming the metric is meaningful.

Full numbers

TangoFlux renders individual events marginally better (recall) but orders them less reliably — the acoustic-vs-semantic distinction our method targets. Ground truth is scored on a subset containing reference audio by the same judge, and is shown only as a sanity-check reference (not on the same prompt set as the four generative models).

Our edits achieve strong success (4.37/5) while keeping non-target content (4.70/5) and temporal structure (4.84/5) largely intact — precise, not destructive. Cross-method edit-success (ΔCLAP) is reported in the paper (Ours +0.094 > TangoFlux +0.081 > Resonate +0.065 > Base +0.054).

• aggregate_taskA.csv — temporal ordering / alignment, per model
• aggregate_taskB.csv — editing preservation, aggregated
• results_taskA.csv — per-clip alignment (with the judge's event transcripts)
• results_taskB.csv — per-pair editing scores
• results_taskC.csv — per-clip captions
• eval/raw/ — individual judge JSON outputs (3 runs/clip)