The Sound of Music — what you see and how you move, affects what you hear

[Nov 19, 2021: Mari Lilleslatten, University of Oslo]

When you hear a melody, your perception is formed by the shapes and movements you associate with it.

When you hear Beyoncé sing, how long does it take before you visualize her dancing across the stage? Or when Jimi Hendrix's guitar solos are pounding out of the speakers—can you see Hendrix posing with his guitar?

Whether you're miming in front of the mirror and using your hairbrush as a microphone or listening with one ear while cleaning the house, the movements you associate with the music play a role in what you actually hear.

That's because music is more than a good lyric or melody. Music is the interplay between everything you sense.

"Just think about how you feel when someone is singing really high notes," says music researcher Tejaswinee Kelkar, and continues:

"What we actually notice is the effort being made by the singer. We recognize it because it's physical. We don't even need to see the singer, because we are so fine-tuned to interpreting nuances in the voice—which, for example, tells us about the singer's feelings."

Kelkar studies the shapes we associate with music and has investigated this by studying what gestures people make when listening to music. Facial expressions and the position of one's legs are just a few of the things that she believes affect one's listening experience.

Related Stories:

When singing, you also use your arms

Tejaswinee Kelkar is herself a performing singer, and her interest in gestures developed when she became aware that there is a difference between how we use our hands when singing Western and Indian music.

"As a child, I learned to sing North Indian music. It is common there to use hand gestures in order to help children when they are learning to sing. When you're on stage, you should sing in the same way as you do when you rehearse. You focus on the song as being sung between you and the room, instead of thinking about the audience."

When she was later trained in Western classical singing, she began using similar hand gestures.

"But I was told quite firmly that that was not how it was done."

She became curious: What did the hand gestures really mean?

"You might think that these movements provide you some kind of anatomical assistance, or that they shape how you use your voice. In that case some hand gestures may be suitable for Indian music and not for Western music."

To Kelkar, the different rules that applied to gestures also served as proof of something else: in order to understand music, you need to think about how it fills the space in a room.

Melodies can have different shapes

Music plays on all our senses because it is multimodal—it takes place in different modes. For Kelkar, the main mode is spatiality. She refers to the mathematician René Thom who says: "In order to understand something, we need to understand the geometry of it."

"I believe he's right: everything has a spatiality. Time, which is important in music, is a good example. We relate the past and future to our bodies—that something lies in front of or behind us."

In order to understand more about how music is perceived spatially, Kelkar has conducted several experiments. In one of these, she asked the participants to listen to the same piece of music several times and draw or explain what they were visualizing in their minds.

"People often perceive specific shapes or use movement metaphors. Several of the participants described or drew the music as a wave that was passing by them—like sound waves or ECG (electrocardiography)," she explains.

"While others visualized a circle, especially if they noticed that one motif in the melody kept recurring."

In another experiment, she asked participants to move their arms in a way that they felt matched the music. She documented everything by using motion capture tracking technology in order to search for patterns.

"Several tried to draw the contours of the music going up or down. This reflected the pitch, but also other characteristics of the music, such as timbre, motifs and patterns."

Facial expressions affect the sounds we hear

The fact that our brains make a connection between what we see and what we hear is something that has already been studied by language researchers.

"This has been observed as linguistic phenomena, including one called the McGurk effect," says Kelkar.

The McGurk effect describes how we can listen to a sound while observing a face that expresses a different sound and hear something which is sort of in between. A classic example of this is when we hear a B pronounced while seeing a face expressing a G, we will end up hearing a D.

Kelkar recently conducted a study with her colleagues Bruno Laeng and Sarjo Kuyateh which was designed to see if the same thing happens when we sing:

"We actually found signs to indicate that what the singer does with his or her face affects how people perceive a melody. The interval between two tones may sound different if the singer's mimicry varies."

Shazam for movement

Just as melody and rhythm allows us to recognize music, the shapes that we associate with a melody can help us with the same thing. In her research, Tejaswinee Kelkar has fed artificial intelligence with documentation of the different movements derived from her experiments, thus allowing the technology to use the movements to recognize music. This type of technology can be used for developing new tools.

"Imagine a technology similar to the music-recognition app 'Shazam' but imagine it scanning movements instead of sounds. If you were to make gestures that would be match 'Happy Birthday,' such an app might be able to find the song for you."

Although this technology is relatively new, searching for melodies based on contours is an old idea. In 1975, Denys Parsons published his "The Directory of Tunes and Musical Themes" in which he cataloged about 15,000 classical pieces based on their melodic contours—how the pitch moves up and down. The identification of music based on its contours is thus called the "Parsons Code."

Kelkar's method is also similar to other tools that are available online.

"For example, today we have musipedia.org which allows you to search for music based on its contours," she says.

Listeners change the music

Sound, text and space are some of the modes included in the music. The same applies to actions, or simply thinking about them.

"If you visualize dancing, the music you hear will probably sound different to how it would if you didn't."

The music researcher also highlights how we experience concerts, something which has changed a lot over the years.

"We have musical genres where people sit still and listen respectfully, but that's something new, because many of our classical composers created music for dancing. We now play mazurkas and minuets in concert halls while the audience sits there watching respectfully. The same applies to jazz, which was a club music genre that was intended for people to dance to."

She points out that this says something fundamental about the multimodality of music.

"Your perception of music is shaped by what you see and do, but the opposite also applies; what you do will affect the music."

Like these kind of feel good stories? Get the Brighter Side of News' newsletter.

Tags: #New_Discovery, #Music, #Cognitive_Science, #Hearing, #Feelings, #Science, #Research, #The_Brighter_Side_of_News