SemanticAudio
Audio Generation and Editing in Semantic Space
SemanticAudio decouples text-to-audio generation into semantic planning and acoustic synthesis. A Semantic Planner first predicts compact frame-level semantic latents that describe what sound events happen and when; an Acoustic Synthesizer then renders those plans into high-fidelity audio. The same semantic space enables training-free, text-guided FlowEdit for targeted audio editing.
Two-stage generation: semantic planning first, acoustic synthesis second.
Core Idea
Plan what happens before rendering how it sounds.
Instead of asking a single model to solve event semantics and waveform detail at once, SemanticAudio predicts compact frame-level semantic latents, then synthesizes DAC acoustic latents conditioned on that semantic plan. The editing method reuses the semantic velocity field for training-free text-guided changes.

Semantic FlowEdit steers source audio toward target text in the semantic latent space.
Generation Demos
AudioCaps test prompts
Fifteen representative AudioCaps samples were selected for event diversity and temporal structure. Each row compares SemanticAudio with strong acoustic-space baselines and the controlled Base Model.
SemanticAudio d=128
TangoFlux
Resonate
Base Model
| Prompt | SemanticAudio | TangoFlux | Resonate | Base Model |
|---|---|---|---|---|
| 01 A man speaking as frogs croak and crickets chirp while a motorboat engine runs alongside several plastic clacks and clanging |
||||
| 02 A motorboat engine running as water splashes and a man shouts in the background followed by birds chirping in the distance |
||||
| 03 Food and oil sizzling followed by metal rattling as a man is talking while guitar music plays and water trickles in the background |
||||
| 04 A man talking followed by a goat baaing then a metal gate sliding as ducks quack and wind blows into a microphone |
||||
| 05 A man talking as another man speaks in the background followed by birds chirping and leaves rustling while wind blows into a microphone |
||||
| 06 A man baaing followed by a goat baaing as a child and a woman talk while metal pots clank in the background |
||||
| 07 A power tool drill running before turning off and on as a man is talking and wood is clanking and scrapping in the background |
||||
| 08 Faucet water flowing as plastic clanks and a man speaks followed by another plastic clank and a kid laughing in the background |
||||
| 09 A large motor vehicle engine accelerates and then slows and idles, while an adult male speaks in the foreground |
||||
| 10 A series of electronic dings while a vehicle horn honks twice followed by a plastic click while a dog barks in the distance |
||||
| 11 A dog whimpering followed by a dog growling and barking as metal jingles and footsteps squeak on hard surface |
||||
| 12 A man talking as a metal thumps followed by a power tool sanding as a metal pot clanks |
||||
| 13 An electronic device buzzing followed by paper tearing and then crumpling as a bell rings before a toilet flushes |
||||
| 14 Music plays and an adult female speaks, a flush occurs and water splashes and gurgles, and a young female speaks |
||||
| 15 A man laughing followed by another man yelling in the distance then wind blowing in a microphone as a motor hums and birds chirp in the background |
Generalization
TTABench prompts
These prompts come from TTABench, a broader text-to-audio benchmark. The selected samples span transport, human activity, animals, weather, tools, impacts, and music-like events.
SemanticAudio d=128
TangoFlux
Resonate
Base Model
| Prompt | SemanticAudio | TangoFlux | Resonate | Base Model |
|---|---|---|---|---|
| 01 A sustained car horn blaring from gridlocked traffic. |
||||
| 02 A train is chugging along blows its whistle and steam escapes from its engine |
||||
| 03 A dog barks, prompting a baby to cry, then the owner's apologetic soothing. |
||||
| 04 A blender whirs briefly as glasses clink, the doorbell rings, someone answers it, and a baby coos in the background. |
||||
| 05 A motorcycle revs aggressively, a trash can clangs against a wall, and a dog barks persistently. |
||||
| 06 A dog's persistent barking erupts, followed by the jingle of its owner quickly shortening the leash. |
||||
| 07 Sheep bleating intermittently while wind rustles through the wheat stalks. |
||||
| 08 The driver argues over hands-free phone while tires hydroplane through water. A child's tablet plays cartoons and seat fabric squeaks. |
||||
| 09 Rain pelts windows while wipers struggle, heater blows air, passenger snores softly, and a phone alarm chirps repeatedly. |
||||
| 10 Simultaneous sounds of typing, paper rustling, clock ticking, and distant traffic create a work ambiance. |
||||
| 11 Cyclist's heavy breathing overlaps with the rustling of a windbreaker. |
||||
| 12 A hair dryer runs while the cabinet door opens, a comb is used, and someone taps the mirror. |
||||
| 13 A single drum maintaining a steady military cadence during a parade performance. |
||||
| 14 Overlapping sounds of bouncing basketball, sneaker squeaks, cheers, and a whistle blow. |
||||
| 15 Threat is shouted, furniture overturns, items crash, door slams, and car speeds away. |
Training-Free Editing
Semantic FlowEdit pairs
Nineteen curated edit pairs show source audio and the SemanticAudio edit. This focused subset keeps the clearest verified transformations across voice, event presence, environment, atmosphere, spatial, intensity, and vehicle edits.
Semantic FlowEdit
TangoFlux FlowEdit
Resonate FlowEdit
DAC FlowEdit
| Edit Type | Source / Target Captions | Source Audio | SemanticAudio Edit |
|---|---|---|---|
| 01Subject Replace | Source: A man speaks with some light cracking Target: A woman speaks with some light cracking |
||
| 02Subject Replace | Source: A man talks while water burbles Target: A woman talks while water burbles |
||
| 03Subject Replace | Source: A male speaking over a microphone Target: A female speaking over a microphone |
||
| 04Sound Addition | Source: Rhythmic ticking is occurring Target: Rhythmic ticking is occurring with water flowing nearby |
||
| 05Sound Addition | Source: Engine noise with other engines passing by Target: Engine noise with other engines passing by followed by a bell ringing |
||
| 06Sound Addition | Source: A woman delivering a speech Target: A woman delivering a speech with birds chirping in the background |
||
| 07Sound Removal | Source: Camera muffling followed by a popping explosion as a group of people talk and silverware clacks in the background Target: Camera muffling |
||
| 08Sound Removal | Source: A walking in the grass and a pig oink sound Target: A walking in the grass |
||
| 09Sound Removal | Source: Wood brushing followed by a door creaking then wood clanking Target: Wood brushing |
||
| 10Weather Change | Source: Heavy wind followed by yelling and cheering Target: Heavy rain followed by yelling and cheering |
||
| 11Environment Add | Source: Male yelling loudly and forcefully Target: Male yelling loudly and forcefully in a large empty room |
||
| 12Environment Add | Source: Bus coming to a stop and door opening Target: Bus coming to a stop and door opening outdoors with wind blowing |
||
| 13Atmosphere Change | Source: A woman is speaking followed by a dog growling and barking Target: A woman is speaking followed by a dog growling and barking echoing in a large space |
||
| 14Atmosphere Change | Source: Wind blows and people speak with some rattling and hissing Target: Wind blows and people speak with some rattling and hissing distorted and eerie |
||
| 15Intensity Change | Source: A woman sneezing repeatedly Target: A woman sneezing briefly |
||
| 16Spatial Change | Source: Birds chirping in the foreground and background as a dog barks in the distance Target: Birds chirping in the foreground and background as a dog barks nearby |
||
| 17Spatial Change | Source: A man talking in the foreground as another man talks in the background followed by a crowd of people applauding Target: A man talking in the foreground as another man talks in the foreground followed by a crowd of people applauding |
||
| 18Vehicle Replace | Source: A car horn honks once shortly and then two long honks while another car honks Target: A truck horn honks once shortly and then two long honks while another car honks |
||
| 19Vehicle Replace | Source: A truck engine revving followed by tires skidding as a crowd of people talk in the background Target: A car engine revving followed by tires skidding as a crowd of people talk in the background |