Chromosonia

Chromosonia is an installation for the facade of the Ars Electronica Center. We provide the means for anyone with a portable music player to visualize the music they are currently listening to on the Ars Electronica facade featuring three levels of mapping: perceptual, informational and social.

When a passer-by connects her music player to our system, our software perceptually analyzes the contents of the acoustic signal in real time. Based on a psycho-acoustic model of the human hearing system and neural networks we interpret the music as spatially distributed forms resulting in a novel way of sonic visualization. The adaptive and highly dynamic nature of our algorithm makes it possible for certain musical features to light up specific parts of the building. This dynamic visual pattern organically reflects rhythmic and harmonic changes in the music.

In addition to this perceptual mapping, there are also informational and social layers in the visualization scheme. The information layer conveys features extracted by matching the audio signal with a database, using music recognition technology. From the detected artist name, track title, and genre we create a color fingerprint of the track, which is used as an additional layer on top of the perceptual visualization.

When no music is plugged into the facade, the social visualization enters the frame. In response to an audio device being unplugged, the perceptual and informational visualization occupying the whole building fades and shrinks to one window pulsating to the recorded rhythm of the music. This pulsation still bears the color coded music genre. After a few seconds all previously visualized tracks fade in on their own window, each pulsating in their own rhythm, using their own color coded information. The nodes are organized on the facade according to style, so that certain genres occupy certain areas of the building. The pattern of this topographical organization of all encountered music conveys the musical taste of the visitors.

Credits
Alex Berman, Eszter Bircsák, Zoltán Csík-Kovács, Gáspár Hajdu, Gábor Papp, András Turi.