Seeing Where Sounds are Coming From

We worked on sound location detection to support product placement and marketing.  Also syncing audio and video experience.

MIT PixelPlayer “Sees” Where Sounds Are Coming From
Synced

The “cocktail party effect” describes humans’ ability to hold a conversation in a noisy environment by listening to what their conversation partner is saying while filtering out other chatter, music, ambient noises, etc. We do it naturally but the problem has been widely studied in machine learning, where the development of environmental sound recognition and source separation techniques that can tune into a single sound and filter out all others is a research focus.

MIT CSAIL researchers recently introduced their PixelPlayer system, which has learned to identify objects that produce sound in videos. The system uses deep learning and was trained by binge-watching 60 hours of musical performances to identify the natural synchronization of visual and audio information.

The team trained deep neural networks to concentrate on images and audio and identify pixel-level image locations for sound sources in the videos.    ... "