SemanticAudio

SemanticAudio

Audio Generation and Editing in Semantic Space

Anonymous Authors

SemanticAudio decouples text-to-audio generation into semantic planning and acoustic synthesis. A Semantic Planner first predicts compact frame-level semantic latents that describe what sound events happen and when; an Acoustic Synthesizer then renders those plans into high-fidelity audio. The same semantic space enables training-free, text-guided FlowEdit for targeted audio editing.

SemanticAudio generation framework

Two-stage generation: semantic planning first, acoustic synthesis second.

Core Idea

Plan what happens before rendering how it sounds.

Instead of asking a single model to solve event semantics and waveform detail at once, SemanticAudio predicts compact frame-level semantic latents, then synthesizes DAC acoustic latents conditioned on that semantic plan. The editing method reuses the semantic velocity field for training-free text-guided changes.

Semantic FlowEdit framework

Semantic FlowEdit steers source audio toward target text in the semantic latent space.

Generation Demos

AudioCaps test prompts

Fifteen representative AudioCaps samples were selected for event diversity and temporal structure. Each row compares SemanticAudio with strong acoustic-space baselines and the controlled Base Model.

SemanticAudio d=128

FD19.1IS9.13CLAP-L0.381MOS3.72

TangoFlux

FD22.6IS12.05CLAP-L0.361MOS3.85

Resonate

FD16.0IS11.04CLAP-L0.403MOS3.78

Base Model

FD25.0IS7.11CLAP-L0.318MOS3.42
PromptSemanticAudioTangoFluxResonateBase Model
01

A man speaking as frogs croak and crickets chirp while a motorboat engine runs alongside several plastic clacks and clanging

AudioCaps test split
02

A motorboat engine running as water splashes and a man shouts in the background followed by birds chirping in the distance

AudioCaps test split
03

Food and oil sizzling followed by metal rattling as a man is talking while guitar music plays and water trickles in the background

AudioCaps test split
04

A man talking followed by a goat baaing then a metal gate sliding as ducks quack and wind blows into a microphone

AudioCaps test split
05

A man talking as another man speaks in the background followed by birds chirping and leaves rustling while wind blows into a microphone

AudioCaps test split
06

A man baaing followed by a goat baaing as a child and a woman talk while metal pots clank in the background

AudioCaps test split
07

A power tool drill running before turning off and on as a man is talking and wood is clanking and scrapping in the background

AudioCaps test split
08

Faucet water flowing as plastic clanks and a man speaks followed by another plastic clank and a kid laughing in the background

AudioCaps test split
09

A large motor vehicle engine accelerates and then slows and idles, while an adult male speaks in the foreground

AudioCaps test split
10

A series of electronic dings while a vehicle horn honks twice followed by a plastic click while a dog barks in the distance

AudioCaps test split
11

A dog whimpering followed by a dog growling and barking as metal jingles and footsteps squeak on hard surface

AudioCaps test split
12

A man talking as a metal thumps followed by a power tool sanding as a metal pot clanks

AudioCaps test split
13

An electronic device buzzing followed by paper tearing and then crumpling as a bell rings before a toilet flushes

AudioCaps test split
14

Music plays and an adult female speaks, a flush occurs and water splashes and gurgles, and a young female speaks

AudioCaps test split
15

A man laughing followed by another man yelling in the distance then wind blowing in a microphone as a motor hums and birds chirp in the background

AudioCaps test split

Generalization

TTABench prompts

These prompts come from TTABench, a broader text-to-audio benchmark. The selected samples span transport, human activity, animals, weather, tools, impacts, and music-like events.

SemanticAudio d=128

CLAP-M0.459AES-CU5.187AES-PQ5.936

TangoFlux

CLAP-M0.464AES-CU5.006AES-PQ5.701

Resonate

CLAP-M0.459AES-CU5.258AES-PQ5.942

Base Model

CLAP-M0.426AES-CU5.119AES-PQ5.897
PromptSemanticAudioTangoFluxResonateBase Model
01

A sustained car horn blaring from gridlocked traffic.

TTABench
02

A train is chugging along blows its whistle and steam escapes from its engine

TTABench
03

A dog barks, prompting a baby to cry, then the owner's apologetic soothing.

TTABench
04

A blender whirs briefly as glasses clink, the doorbell rings, someone answers it, and a baby coos in the background.

TTABench
05

A motorcycle revs aggressively, a trash can clangs against a wall, and a dog barks persistently.

TTABench
06

A dog's persistent barking erupts, followed by the jingle of its owner quickly shortening the leash.

TTABench
07

Sheep bleating intermittently while wind rustles through the wheat stalks.

TTABench
08

The driver argues over hands-free phone while tires hydroplane through water. A child's tablet plays cartoons and seat fabric squeaks.

TTABench
09

Rain pelts windows while wipers struggle, heater blows air, passenger snores softly, and a phone alarm chirps repeatedly.

TTABench
10

Simultaneous sounds of typing, paper rustling, clock ticking, and distant traffic create a work ambiance.

TTABench
11

Cyclist's heavy breathing overlaps with the rustling of a windbreaker.

TTABench
12

A hair dryer runs while the cabinet door opens, a comb is used, and someone taps the mirror.

TTABench
13

A single drum maintaining a steady military cadence during a parade performance.

TTABench
14

Overlapping sounds of bouncing basketball, sneaker squeaks, cheers, and a whistle blow.

TTABench
15

Threat is shouted, furniture overturns, items crash, door slams, and car speeds away.

TTABench

Training-Free Editing

Semantic FlowEdit pairs

Nineteen curated edit pairs show source audio and the SemanticAudio edit. This focused subset keeps the clearest verified transformations across voice, event presence, environment, atmosphere, spatial, intensity, and vehicle edits.

Semantic FlowEdit

CLAP-L0.351Delta CLAP-L0.094MOS3.71

TangoFlux FlowEdit

CLAP-L0.339Delta CLAP-L0.081MOS3.54

Resonate FlowEdit

CLAP-L0.322Delta CLAP-L0.065MOS3.48

DAC FlowEdit

CLAP-L0.29Delta CLAP-L0.054MOS3.32
Edit TypeSource / Target CaptionsSource AudioSemanticAudio Edit
01Subject Replace

Source: A man speaks with some light cracking

Target: A woman speaks with some light cracking

Delta CLAP +0.243Target CLAP 0.259
02Subject Replace

Source: A man talks while water burbles

Target: A woman talks while water burbles

Delta CLAP +0.195Target CLAP 0.435
03Subject Replace

Source: A male speaking over a microphone

Target: A female speaking over a microphone

Delta CLAP +0.143Target CLAP 0.348
04Sound Addition

Source: Rhythmic ticking is occurring

Target: Rhythmic ticking is occurring with water flowing nearby

Delta CLAP +0.179Target CLAP 0.539
05Sound Addition

Source: Engine noise with other engines passing by

Target: Engine noise with other engines passing by followed by a bell ringing

Delta CLAP +0.204Target CLAP 0.365
06Sound Addition

Source: A woman delivering a speech

Target: A woman delivering a speech with birds chirping in the background

Delta CLAP +0.148Target CLAP 0.360
07Sound Removal

Source: Camera muffling followed by a popping explosion as a group of people talk and silverware clacks in the background

Target: Camera muffling

Delta CLAP +0.368Target CLAP 0.397
08Sound Removal

Source: A walking in the grass and a pig oink sound

Target: A walking in the grass

Delta CLAP +0.352Target CLAP 0.336
09Sound Removal

Source: Wood brushing followed by a door creaking then wood clanking

Target: Wood brushing

Delta CLAP +0.206Target CLAP 0.309
10Weather Change

Source: Heavy wind followed by yelling and cheering

Target: Heavy rain followed by yelling and cheering

Delta CLAP +0.180Target CLAP 0.473
11Environment Add

Source: Male yelling loudly and forcefully

Target: Male yelling loudly and forcefully in a large empty room

Delta CLAP +0.065Target CLAP 0.319
12Environment Add

Source: Bus coming to a stop and door opening

Target: Bus coming to a stop and door opening outdoors with wind blowing

Delta CLAP +0.042Target CLAP 0.414
13Atmosphere Change

Source: A woman is speaking followed by a dog growling and barking

Target: A woman is speaking followed by a dog growling and barking echoing in a large space

Delta CLAP +0.085Target CLAP 0.486
14Atmosphere Change

Source: Wind blows and people speak with some rattling and hissing

Target: Wind blows and people speak with some rattling and hissing distorted and eerie

Delta CLAP +0.088Target CLAP 0.265
15Intensity Change

Source: A woman sneezing repeatedly

Target: A woman sneezing briefly

Delta CLAP +0.076Target CLAP 0.451
16Spatial Change

Source: Birds chirping in the foreground and background as a dog barks in the distance

Target: Birds chirping in the foreground and background as a dog barks nearby

Delta CLAP +0.140Target CLAP 0.390
17Spatial Change

Source: A man talking in the foreground as another man talks in the background followed by a crowd of people applauding

Target: A man talking in the foreground as another man talks in the foreground followed by a crowd of people applauding

Delta CLAP +0.058Target CLAP 0.419
18Vehicle Replace

Source: A car horn honks once shortly and then two long honks while another car honks

Target: A truck horn honks once shortly and then two long honks while another car honks

Delta CLAP +0.118Target CLAP 0.380
19Vehicle Replace

Source: A truck engine revving followed by tires skidding as a crowd of people talk in the background

Target: A car engine revving followed by tires skidding as a crowd of people talk in the background

Delta CLAP +0.087Target CLAP 0.437
The editing page uses the verified curated SemanticAudio outputs with exact source/target caption metadata. The released code directory also includes the final 200-pair editing benchmark used for paper evaluation, and a verification file confirming its captions align with the packaged evaluation results.