Synesthesia

TouchDesigner, VCVRack, MIDI Virtual Cable, 2 Audio Virtual Cables

This project was developed during the course Computational Creativity in 2021. The aim of the project was to explore, reproduce and study different types of synesthesia. In order to do so, I implemented a generator which tries to visualize and recreate 3 types of synesthesia: Chromesthesia, Pitch-Size and Pitch-Location Synesthesia.

For an explanation of the technical specifications of the generator refer to files and papers at the end of this post. Briefly, the audio for the model is generated through a Markov Chain probability model, which is learnt in VCVRack. The model is then triggered from a pseudo-random clock, and sends back notes according to the probability distribution. Since the inspiring file was made out of 3 MIDI channels, the model learns 3 different probability distributions. The system is thought to be used in real-time. The file I used as inspiring set is the song "Everything you do is a Balloon" from Boards of Canada, but every MIDI file would work fine.

In order to study the 3 different types of Synesthesia, I have implemented 4 different ways of visualizing the music.
The top-left corner of the 4 videos tries to visualize the Pitch-to-Color Synesthesia (Chromesthesia) and the Pitch-to-Altitude Synesthesia.
The top-right quadrant is trying to visualize Chromesthesia and Pitch-to-Size Synesthesia.
The bottom-right quadrant is visualizing both Pitch-Size and Pitch-to-Color Synesthesia.
The bottom-left quadrant is the most complete among the other ones, trying to visualize all the 3 types of synesthesia described.

Audio/Visual Generator

README (Technical Specification)

Synesthesia


Here you can see a demonstration video