# Gemini audio-labeling prompts (LLM-as-judge for the rebuttal) Use Gemini 2.5 Pro (audio-capable). Send audio inline/File API. Temperature 0–0.2. Two tasks: (A) temporal-ordering / event-alignment scoring — answers XtA5-W1 & jQgw-W5; (B) editing evaluation with success/preservation separated — answers XtA5-W2. Grounding trick used in both: force the model to FIRST transcribe events + timestamps, THEN score. This reduces hallucinated judgments and gives you an audit trail. Ask for STRICT JSON. ================================================================================ ## PROMPT A — Temporal-ordering & event-alignment judge ## Inputs per call: {caption} + one generated audio clip (10 s) ================================================================================ You are an expert audio annotator evaluating a text-to-audio generation system. You are given (1) a TEXT CAPTION that was used to generate an audio clip, and (2) the AUDIO CLIP. Your job is to judge, strictly from what you HEAR, how well the audio matches the caption — with special attention to (a) which sound events are present, and (b) their TEMPORAL ORDER. Work in two steps. STEP 1 — Transcribe events. List every distinct sound event you actually hear, in the order they begin. For each, give: a short label, approximate onset and offset in seconds, and a confidence in [0,1]. Do NOT use the caption to guess events you cannot hear. STEP 2 — Score against the caption: - event_recall: of the events named in the caption, what fraction are clearly audible? [0,1] - event_precision: of the events you heard, what fraction are actually implied by the caption? [0,1] - temporal_order_match: does the ORDER of events in the audio match the order stated/implied by the caption? Use: 1.0 = order fully correct; 0.5 = events present but order wrong/unclear; 0.0 = required events missing so order is undefined. If the caption implies no ordering (single event or simultaneous), set temporal_order_applicable=false and temporal_order_match=null. - overall_alignment: holistic 1–5 (1=unrelated, 3=partially matches, 5=faithful match incl. order). Rules: base every judgment ONLY on the audio. If unsure, lower the confidence, do not invent events. Output ONLY valid JSON, no prose outside it. Output JSON schema: { "events_heard": [ {"label": "string", "onset_s": number, "offset_s": number, "confidence": number} ], "caption_events": ["string", ...], "event_recall": number, "event_precision": number, "temporal_order_applicable": boolean, "temporal_order_match": number | null, "overall_alignment": integer, "notes": "one short sentence of justification" } > For the W1 reversed-prompt test: run this on BOTH the correct-order caption and the reversed > caption for the same audio; a model that respects order should get higher overall_alignment / > temporal_order_match on the correct caption. Aggregate temporal_order_match across the set as > "Temporal-Order Accuracy" per model. ================================================================================ ## PROMPT B — Editing judge (success vs. preservation, separated) ## Inputs per call: {source_caption} + {target_caption} + SOURCE audio + EDITED audio ## (send two audio clips, labeled clearly in the order below) ================================================================================ You are an expert audio annotator evaluating a text-guided AUDIO EDITING system. You are given: a SOURCE CAPTION, a TARGET CAPTION, the SOURCE AUDIO (first clip), and the EDITED AUDIO (second clip). The intended edit is the difference between source and target captions (usually a single attribute, e.g. "man speaking" -> "woman speaking"). Judge, strictly from what you HEAR, whether the edit achieved the target change WITHOUT disturbing everything else. Work in two steps. STEP 1 — Describe both clips. For SOURCE and for EDITED separately, list the sound events you hear with approximate onset/offset (s). Then state, in one phrase, the single attribute the target caption asks to change. STEP 2 — Score on three INDEPENDENT axes (do not let one leak into another): - edit_success [1–5]: did the EDITED audio realize the target attribute? (5=clearly changed to target, 1=no change / wrong change). - nontarget_preservation [1–5]: is everything OTHER than the edited attribute (background, other events, timbre, scene) preserved from the source? (5=well preserved, 1=heavily altered). - temporal_preservation [1–5]: is the timing/structure (number and order/rhythm of events) preserved relative to the source? (5=same structure, 1=structure destroyed). - naturalness [1–5]: does the edited audio sound natural/artifact-free? - verdict: one of "good_edit" | "over_edit" (target changed but too much else changed) | "under_edit" (too little changed) | "failed". Rules: judge only from audio. edit_success and nontarget_preservation are a trade-off — score them independently and honestly. Output ONLY valid JSON. Output JSON schema: { "source_events": [{"label": "string", "onset_s": number, "offset_s": number}], "edited_events": [{"label": "string", "onset_s": number, "offset_s": number}], "intended_change": "string", "edit_success": integer, "nontarget_preservation": integer, "temporal_preservation": integer, "naturalness": integer, "verdict": "string", "notes": "one short sentence" } > Aggregate per method: mean edit_success, mean nontarget_preservation, mean temporal_preservation. > This fills the W2 table (success vs. preservation as separate columns). Run identically on > every method's edited audio (Ours / TangoFlux / Resonate / Base) with the SAME source clip. ================================================================================ ## PROMPT C — Audio captioning / labeling (for sharing clips with others) ## Inputs per call: one audio clip (10 s), no caption given ================================================================================ You are an expert audio annotator. You are given one AUDIO CLIP (~10 s) with no text. Describe ONLY what you actually hear. Do not speculate about causes you cannot hear. Provide: - caption: one natural-language sentence describing the clip, in the style of an AudioCaps caption (mention the main sound events and, if clearly audible, their order). - events: the distinct sound events with approximate onset/offset in seconds and a confidence. - tags: 3–6 short keyword tags (e.g. "dog bark", "rain", "speech", "engine"). - ordered: true if the clip has a clear temporal sequence of distinct events, else false. - quality_flags: any audible issues ("clipping", "silence", "artifacts", "muffled"), or []. Output ONLY valid JSON: { "caption": "string", "events": [{"label": "string", "onset_s": number, "offset_s": number, "confidence": number}], "tags": ["string", ...], "ordered": boolean, "quality_flags": ["string", ...] } > Use this to attach a text description to every clip before sending to others, and to auto-index > the downloaded audio. The "ordered" flag helps you mine candidate temporal-sequence samples. ================================================================================ ## Building the ordered-event prompt set (XtA5-W1 — you need to construct this) ================================================================================ You do not yet have explicit "A followed by B" prompts, so build them: 1. Pick ~8–12 clearly distinguishable event classes from AudioSet (dog bark, car horn, door knock, applause, gunshot, glass breaking, baby cry, thunder, whistle, footsteps ...). 2. Form ordered pairs ", followed by " and the reversal ", followed by ". Target ~50–100 prompts (both directions), avoiding acoustically confusable pairs. (Optionally use GPT-4/Gemini to phrase natural captions; keep the ordering explicit.) 3. Generate audio for each prompt with EACH model (Ours d=128 / Base / TangoFlux / Resonate). 4. Judge every clip with PROMPT A against BOTH its own caption and the reversed caption. 5. Metric — Temporal-Order Accuracy (TOA): fraction of clips where the judge's temporal_order_match=1.0 on the correct caption AND the correct caption scores higher overall_alignment than its reversal. Report TOA per model; expect Ours > Base. > Release this prompt set + the audio on the demo page — also answers the "no dataset" ding. ================================================================================ ## Practical notes ================================================================================ - Blind the judge: never tell Gemini which method produced a clip; randomize clip order. - Run each clip 3x and average (or majority verdict) to reduce variance; report mean. - Keep a CSV: [task, method, sample_id, caption(s), all json fields] for the paper tables. - Independence caveat for the rebuttal: state that the judge (Gemini) is a DIFFERENT model family from PE-A-Frame (the generator's encoder), so the temporal-order eval is not circular.