MELOVIZZ DEMONSTRATOR (PHENICX, 2015)
Music Technology Group, UPF

Melovizz is a web-based visualisation tool developed in the context of the EU FP7 PHENICX Project. PHENICX is about innovating the classical music concert experience with a multi-modal, multi-perspective and multi-layer interactive engagement, before, during and after the concert. This particular demo is related with the post concert experience.

Melovizz allows playing a musical piece, and following the automatically estimated melody in a piano-roll canvas. A scrolling curve shows pitch values (y-axis), while time is represented horizontally (x-axis). A different colour is used for each instrument (or section), and the intensity of each colour is variable, mapped to the estimated probability that a given instrument is playing the melody. A vertical line refers to the current playing time, and pitches estimated in a short time window around it are displayed, both in the past and future. Variable size text labels are additionally displayed at the top, showing the instrument/s that contribute to the melody. The name of the instrument considered predominant is displayed in its corresponding colour.

This tool shows the results obtained with two melody extraction approaches. The first one has been submitted to MIREX 2015, and is based on a source-filter model, combined with a method based on pitch contour characterisation. The second one takes advantage of the aligned music score to guide the melody estimation. The probability that each of the instruments is playing the melody is automatically estimated combining audio and score information.

Royal Concertgebouw Orchestra - Beethoven's 3rd Symphony (Eroica)



HELP: Click on one of the movements to see the live demo.

ORCHESTRA LAYOUT VIEW DEMONSTRATOR (PHENICX, 2014)
Music Technology Group, UPF

The Orchestra Layout web-based demonstrator has been developed in the context of the EU FP7 PHENICX Project. The Phenicx project is about innovating the classical music experience providing them with a multi-modal, multi-perspective and multi-layer interactive engagement, before, during and after the concert. This particular demo is related with the post concert experience. During the project we have recorded a set of classical pieces performed by top level orchestras, including some data modalities like multi-channel audio, video, motion capture sensors, midi and text. Once all data streams have been time-synchronized, we have performed the following analysis on the data:

- Low-level and high-level audio descriptors for each individual audio source
- Description of conductor gestures based on the motion capture sensors
- Score to performance alignment
- Audio source separation
- Musical structure analysis of the performed piece

All these data are available in the repovizz web repository that allows visualization and sharing of the data over the network. A set of customized web-based visualizations have been designed to build the prototype shown in this demo. Multi-modal data streams are accessed on-line by the demonstrator using the repovizz web API and html5 is used for the visualizations of the multi-modal data and descriptors extracted from the performances. The web-audio API is used to handle the audio rendering in the client to mix between the different audio sources.

In this particular demo you will see an orchestra layout image representing the orchestra musicians and while playing the piece, colour lights will highlight the sections that are playing at each time and the intensity of the lights will correspond to the dynamics of each instrumental section. This results have been achieved by aligning the musical score of the piece to the actual performance. Selecting one of the instruments (top part in the demo) you will be able to listen to the sound of each isolated instrumental section achieved using source separation techniques from the stereo performance recording.

Royal Concertgebouw Orchestra - Beethoven's 3rd Symphony


Orquestra Simfònica del Vallès - Beethoven's 9th Symphony



HELP: Click on one of the movements to see the live demo.