To learn more please check out our ICML'25 paper: "FLAM: Frame-Wise Language-Audio Modeling"
arxiv.org/abs/2505.053...
Big congratulations to Yusong Yu, @tsirif.bsky.social and the whole team from @adobe.com research, @mit.edu and @mila-quebec.bsky.social
FLAM is trained jointly on instance (global) and frame-wise (local) objectives.
The secret sauce: A memory-efficient and calibrated frame-wise objective with logit adjustment to address spurious correlations, such as event dependencies and label imbalances during training
Enter FLAM: Frame-Wise Language-Audio Modeling.
A model trained to produce a calibrated likelihood for *any* text prompt.
FLAM outperforms prior self-supervised models on both closed-set and open-set SED, while preserving strong retrieval and zero-shot classification accuracy
Our goal is for the model to detect *any* sound via free form text queries.
"So use CLAP", some of you will say.
The problem is its output likelihoods are not calibrated for different prompts :(
That's ok ranked retrieval, but for detection it's a no go.
Sound Event Detection models, ie finding sounds in audio/video recordings, are typically constrained to a predefined "closed" set of sounds, like in this (old!) model below for urban sound detection.
It has some applications, but it doesn't address general purpose sound search.
I think we finally cracked it? FLAM can detect *any* sound via text prompts
arXiv (ICML'25): arxiv.org/abs/2505.053...
demos: flam-model.github.io
Led by Yusong Wu, with @tsirif.bsky.social Ke Chen, Cheng-Zhi Anna Huang, Aaron Courville, @urinieto.bsky.social @pseeth.bsky.social
Generative Extend in Premiere Pro just won *five* awards at NAB 2025, including the NAB Show Product of the Year award! SODA, our group, created the audio GenAI model in charge of audio extensions in the feature. Couldn't be more proud of the team!
w/ @urinieto.bsky.social @pseeth.bsky.social
Generative Extend just released in Premiere Pro! Use GenAI to extend your video *and audio* clips for a perfectly timed edit.
The audio model was built by our team, the Sound Design AI (SODA) group at Adobe Research w/ @pseeth.bsky.social and @urinieto.bsky.social 🙌
www.youtube.com/watch?v=_Bv5...
We didn't expect this... our Sketch2Sound demo video has gone viral on IG with more than 5.2 million views 🤯
Amazing job @hugofloresgarcia.bsky.social @pseeth.bsky.social @urinieto.bsky.social
I should've done my hair...
www.instagram.com/reel/DEEBRhd...
Really cool to see our AI for bioacoustics work in the MIT Technology Review! BirdVox was a fantastic project that beyond posing a fascinating research problem, introduced me to the world of bird watching and deepened my appreciation of the natural world
www.technologyreview.com/2024/12/18/1...
@waspaa.com is on Bluesky!
Big changes are coming for WASPAA 2025!
Sketch2Sound is out!
Takes a text prompt + vocal (or sonic) imitation and generates sound effects that perfectly match the energy and dynamics of your voice.
It's an extremely intuitive (and fun!) way to create SFX that are perfectly timed to your video.
Led by @hugofloresgarcia.bsky.social 👏
I'm hiring a Research Engineer for my team at Adobe Research.
Full details in the link in the post below, DM me if interested.
Here's another example of work from our group:
MultiFoley, a Video-to-Audio model that generates perfectly synced audio for video at 48 kHz and supports multimodal conditioning.
More on MultiFoley here: bsky.app/profile/czya...
📢 Audio AI Job opportunity at Adobe!
The Sound Design AI Group (SODA) is looking for an exceptional research engineer to join us in building the future of AI-assisted audio and video creation.
Strong ML background, GenAI experience a plus.
Details: adobe.wd5.myworkdayjobs.com/external_exp...
New model from the SODA group at Adobe and UMich!
MultiFoley generates perfectly synced audio for video at full 48 kHz and supports multimodal conditioning.
You can define the generated sound via a text prompt, an example SFX, or audio you want to extend.
Led by our intern @czyang.bsky.social 👇
