What’s the fascination with Virtual Reality? Until you’ve tried it for yourself it’s hard to appreciate why strapping a phone to your face has anything to offer beyond gimmickry. Yet it’s what happens to both body and mind when you’re ‘inside’ the VR world that defies expectations. We encounter all the physiological and emotional reactions that we do in the real world. Perched on the edge of a virtual cliff, you’re seized by fear no matter how much you tell yourself “this is not real!”. Take this recent example, where viewers find themselves having to rescue a helpless kitten high on a ledge.
Simply put, VR is the first wholly new form of media to emerge since 1910. Where film was about manipulating time, VR is about manipulating space and our place in it. Before it we’d seen only enhancements to existing forms of moving image and sound. That elusive nirvana of every creative conversation — how to create ‘immersive’ experience — is suddenly obsolete. Because in VR you are immersed, like it or not. The gap is gone.
VR is also a completely new medium for telling stories, for giving experiences, and for social interaction. It is different from film because all of a sudden we are in the movie (we have presence), and we are free to look and move around inside it the way we choose (we have agency). A whole new language needs to written for the creators of VR and for its users, and a key part of this vocabulary is sound.
The Role of Sound in VR
Many of the new VR production companies emerging are offshoots of visual effects outfits, for whom expanding imagery from a forward placed screen to a 360º field is a relatively accessible process. Because their skillsets are primarily visual, sound was getting left behind. But ask any VR developer, from Jaunt to Industrial Light and Magic, and they will tell you: sound is far more important in VR than it ever was in film or tv. In the vast majority of VR experiences, we are using only two of our senses: sight and hearing. So sound is 50% of the experience to be had, yet it conveys almost all the emotion. In the VR experience just as in life, it can convey feelings or drive a story without even needing to look at it, because sound is by nature immersive, and our ears never blink. In this way, sound has found an important role in the VR world: to lead the user’s attention towards something you want them to see that may be out of view (the human visual system sees only a 180º field of view, and just 114º of that in 3D.)
Man Made Music in VR
As long-time evangelists for the power of sound, Man Made Music embraced the sonic possibilities of the VR space early on, building partnerships with developers of new audio tools and creating unique workflows to enhance the sound & music component of VR experiences. Great advances in the simulated positioning of sound (spatialization) mean that we can place sounds anywhere in space, even when listening on regular headphones. In the VR experience this spatialized soundscape can now track head movement correctly so the sound remains in place, just as in real life. We can go further, creating an ‘inner experience’ for the user, an internal dialogue or song, by making it sound distinct from the sounds that are ‘out there’.
Our long experience in creating meaningful & memorable music and sonic identities for entertainment, general market brands and physical spaces from stores to stadiums, finds a perfect home in the VR world. We can bring these diverse elements together, creating experiences that use space in entertaining and engaging ways. As with all our work at Man Made Music, strategy is our foundation, and this is ever more important in VR, where the canvas is seemingly limitless. Our development process begins with understanding the larger concept before moving into any creative development. What emotion do we want to evoke? What story are we telling?
If the VR project requires live location recording, we use ambisonic microphones to capture a ‘soundfield’, which together with detailed post-produced soundscapes can then be refined and manipulated to accurately represent a true-to-life, responsive sonic experience. On the music side, we are able to go beyond the conventions of stereo and surround, and create real musical interactions with the user. When it comes to playback, we then work with our clients to determine the ideal delivery platforms, whether GearVR, Oculus Rift, HTC Vive, Google Cardboard and beyond, and facilitate integration of the necessary codecs.
Some of our recent work which we are particularly proud of includes Studio Transcendent’s VR film A Brief History of Flight. This unique piece was nominated for the 2015 Proto Award for Best Educational VR Experience at the most recent VRLA Conference. Viewers are brought on a cinematic journey through the history of aviation, from witnessing the Wright Flyer’s first leap, to being strafed by WWI fighters, buzzed by the leading-edge F22 Raptor, and nearly engulfed by the world’s largest plane. Every sound of every aircraft is historically accurate, carefully spatialized to give the user a real-life experience of this amazing airshow. The experience is underscored by a soaring orchestral soundtrack composed by Man Made Music’s team.
In our work on Applied VR’s Pain RelieVR app — a post-surgery VR experience currently being tested in hospitals in the US — we drew from research in music therapy to create a patterned musical soundscape that helps distract patients without overwhelming them. Placed in the VR setting with a simple, visual task-oriented game, patients experience as much as a 40% reduction in pain. This project, developed in collaboration with Cedars-Sinai and Children’s Hospital Los Angeles is now entering its next phase based on patient feedback, and we have begun work on a second VR patient experiences aimed at anxiety management.
Our live-action pieces with SilVR Thread challenge users to speed down a ski terrain park, go zip-lining or sailing. Throughout these journeys, the music and sound amplifies the thrill and the realism. So much so, that sometimes this happens!
We’re deliberately pushing the envelope to usher in a new kind of experience — Sonic VR: 360º experiences of space that are driven by sound and music, with all the meaning and emotionality they bring. The possibilities are endless and exciting. So go ahead & strap that phone to your face, but don’t forget the headphones!
Joel Douek is Creative Director, West Coast and Chief Scientist at Man Made Music. Follow him on Twitter to learn more about VR: @joeldouek.
We receive sound at such a deep subliminal level, often we don’t realize that in the real world it is a fully 3D experience, replete with directional meaning and information. Unlike vision, it doesn’t matter where we look in order to perceive sounds. Binaural sound captures all this information with a specialized array of microphones positioned on a “dummy” head to closely match how we hear. Together with psychoacoustic processes, we are then able to simulate a fully 360º 3D sound experience through just a pair of regular headphones. Sounds can move up, down, behind you as well just left and right. Binaural Sound has most recently found its rightful place in the world of virtual reality, and you don’t even need goggles. Take a sonic journey with us, through a dense jungle in search of lost treasure!
Joel Douek is Creative Director, West Coast and Chief Scientist at Man Made Music. Follow him on Twitter to learn more about sonic journeys: @joeldouek.
If a tree falls in the forest, and nobody is there, does it make a sound?
If we not rush to reflexive conclusions, we find that this age-old question unearths a fascinating look at the nature of sound, of hearing, and further still, our place in reality.
How is sound produced? It is a disturbance, a ripple in some medium, whether air or water, earth or steel. Branches and tree parts falling through the air and hitting the ground create pulses of air. Rapid pressure variations that spread in ripples at 750 miles per hour. There is no sound implicit in them – it is just wind moving.
Yet if someone armed with an ear and an auditory cortex happens to be nearby, these puffs of air will hit their eardrum, vibrating it and setting off a cascade of mechanical and electrical activity down the auditory nerve and into the brain, resulting in the perception of a noise. For the sound to be perceived, however, the frequency of the air puffs must fall between 20 and 20,000 times a second – the particular range for which the ear and neural system are designed. Either side of this – there is just silence.
So an observer is every bit as essential to create the sound experience. When a tree falls in the forest and no one is there, there are just silent pulses of air. Sound is a shared experience, a symbiosis between the world outside and our conscious perception. They are indivisible. Indeed this is one of the most fundamental and puzzling discoveries of Quantum Physics – that reality is a process, which involves, and is even influenced by, our consciousness.
When we are immersed in the sonic landscape of our lives, replete with songs and sirens, we are full participants in the process of creating it. When music stirs our emotions, the little puffs of air stimulate ancient neural pathways, conjuring memories and emotions both personal and universal. Perhaps this is what makes life all the more remarkable: that while everything is ultimately in our heads, what we feel in a song or a sound can be so deeply shared by other people, who we may have never even met. Music and sound forge a bridge from our individual experience to the individual experiences of countless others. Now that is magic.
And if you still believe in the firm reality of sound outside of your brain, check out some of these bewildering Sonic Illusions.
Joel Douek is Creative Director + Sonic Strategist at Man Made Music. Talk to him on Twitter @joeldouek.