For my final project, I would like to combine the makey makey audio-visual project as well as the face-tracking technology to make a musical instrument for individuals to play. One user would interact with the face tracking input to change the pitch of the audio playing as well as add visual effects while another individual would use the buttons on the makey makey to change the base audio/visual component.
Together the audiovisual experience is meant to be reminiscent of “Ghost Choir”, a song that brings me the upmost joy, especially during the spooky season.
Artist Questions:
For the audio component, would it be stronger to have each button be a tone that would be shifted rather than a clip of audio so that it is clearer that the changes in the audio are due to the individual rather than shifts in the “song”?
For the face tracking capabilities, would “mouth-openness” be a strong enough controller to change the pitch of the audio?
For the face tracking capability, mouth-openness could definitely work, other ideas can vary from eyebrow-raising, or eyes being open/shut!
To answer your first question, having each button be a tone to be shifted would likely be easier for the user to understand.
For face tracking capabilities, mouth-openness and eyebrow raising seems to be the most common ways I’ve seen.
I think tone shift would be noticeable and that mouth-openness would work well, especially with those singing ghosts.
For audio being a tone or not, I think that might require a test run of both options. Sometimes the impact of the audio clip you choose is important enough to include it.
To answer both questions, buttons would be a strong signifier that the user is changing the tones and the opening the mouth to change the pitch would also be a strong controller.
Would each change in pitch also be reflected visually? As a response to the question, I think mouth opening would be strong enough.
I think it would be fun if the face tracking user was also controlling the little ghosties’ faces (even in a basic way)
Mouth openness would work well, another input you could try is how close the user’s face is to the camera