San José City Hall Rotunda Concert and Sound Projection on April, using real-time generative AI Collaborate with SJSU Orchestra

In April, I participated in the live music and visual performance at the San José City Hall Rotunda Concert.

My Master’s program colleague Mona Farrokhi and I created a real-time visualization for the performance, using TouchDesigner (a node-based visual programming language for real-time interactive multimedia) with StreamDiffusion (a diffusion pipeline designed for real-time interactive generation) for the event.

The program takes external gestures and images, processes them with prompts used for AI to create new images, and renders them into video. TouchDesigner performs secondary programming to detect objects in the video and draw their edges, incorporating them into another video. We built a MIDI console to control parameters of the program, such as crossfading between different layers of video, adjusting color and contrast, switching between prompts and seeds, and more. We projected the visual onto the wall of San José City Hall and controls it in real time on the station.

Additionally, I used some images taken by the composer from downtown San José to train a LoRA (Low-Rank Adaptation) model. This gave the AI a concept of 'signs of San José,’ creating a distinctive visual that was relevant to the music and different from the basic model.

The performance utilized the characteristics of the venue itself, allowing the sound to spread to every corner with a distinctive trajectory. People could move freely around the venue and enjoy the music from different angles.

Special thanks to the performers, the composer Pablo Furman and Christopher Luna-Mega, my colleague Mona Farrokhi, and my advisor Andrew Blanton.