Hollywood, CA (September 15, 2016)—Star Wars director George Lucas has famously been quoted as saying, “Sound is 50 percent of the moviegoing experience.” But in virtual reality (VR), sound may be much more important, as presenters explained at a recent meeting of the Hollywood section of the Society of Motion Picture and Television Engineers (SMPTE).
“We see forward, but we hear all around us. We can tell direction from sound. So sound may be more than 50 percent of the storytelling toolkit between sound and visuals when you shoot VR,” observed Phil Lelyveld, VR/AR (augmented reality) initiative program manager at the University of Southern California’s Entertainment Tech Center.
Lucas Wilson, a VR producer with developer Jaunt VR and his own SuperSphere Productions, noted that a cinema-goer receives visual information from the screen, but a viewer’s attention could be anywhere in a VR experience. Sound can be a useful tool to direct, or misdirect, attention and also helps solve the puzzle of how to insert cuts into this 360-degree visual medium.
“One of the first things I ever did in VR was a band video,” said Wilson, who has over 60 VR projects under his belt, including a new documentary short series featuring Paul McCartney. “I put one VR camera in the middle of a rehearsal space. There’s a crash behind you, and your limbic system does not allow you to not turn around. Instantly, you whip your head around and you see that the drummer dropped a cymbal.
“There’s a sound back here,” he continued, snapping his fingers. “You turn around, and now you’re on stage at Madison Square Garden. When you follow the cues that your brain is telling you to follow, it’s magic.”
Consequently, he said, VR need not be a continuous, real-time experience simply because of its 360-degree field of view. “With audio, you can cue people to look around and make them look places, not in a gimmicky way, but by guiding the eye in that world in a way that you can’t [in any other way].”
On a current project on which he is working, Wilson continued, sound is being used to enhance the storytelling through interactivity with the viewer’s gaze. “When you look at the bad guy, the entire scene shifts in color and it goes darker. There’s a mid-range boost in the audio, so that things get a little more muddled. Everything about the scene that you traditionally associate with bad things influences that scene. It’s influencing people’s emotions in ways that you don’t traditionally think of and which are not available to you” in other visual media, he said.
Although VR’s roots can be traced back many decades, with early examples including flight simulators and certain arcade video games, VR has only been widely, and globally, commercialized over the past three years. Lelyveld offered a rapid-fire tour through the technology and techniques that began with the basics—what is augmented reality versus virtual reality—and concluded with some of the tools required for a successful VR experience. “It’s an emerging art form. We’re developing the language and the tech at the same time,” he commented. “Nobody knows anything yet. Don’t worry about it; you have not missed any sort of boat.”
Indeed, standards for VR and AR have yet to be published, he said: “There are efforts to standardize bits of it.” For instance, Google’s Spatial Audio RFC, an open metadata scheme utilizing MP4 containers accommodating spatial and non-diegetic (with no visible source) audio “seems to be moving along. But nothing has come into clarity yet.”
There are three ways to capture VR sound, Lelyveld enumerated, beginning with ambient or spatial audio from the camera position. “You might also want to mic the actors, so you can understand them. But you also have to track their positions, so when you get into post their sound is properly positioned. And third, sound with no location, like music or voiceover.”
VR content creators have bemoaned the lack of a comprehensive and cohesive set of spatial audio postproduction tools. Days before the SMPTE event, Facebook unexpectedly announced that it had acquired Edinburgh, Scotland-based audio software developer Two Big Ears. The developer’s immersive audio post production, mixing and rendering platform, rebranded as the Facebook 360 Spatial Workstation, is now available as a free download.
The platform encompasses a plugin suite enabling the positioning of sounds in an immersive environment within DAWs, including Pro Tools, Nuendo and Reaper. Sources may include tracks recorded with mono, stereo and Ambisonic microphones and arrays, music and other elements. Time-synchronized VR video allows content to be previewed on the device via the DAW. The renderer supports Android, iOS, Windows, OSX, HTML5 and game engines.
The VR world is continually evolving and new tools are constantly emerging. Cameras and camera arrays, which can be 2D or 3D, producing 360-degree images natively or through the application of algorithms after the fact, are becoming more readily available. For example, Nokia recently launched Ozo, reported Lelyveld: “It not only captures 360, but it has eight microphones, so you’re capturing spatial audio as well.”
He commented, “The global consumer market for VR is growing rapidly. The price points now start at $200. This isn’t a walled garden.”
“I think it’s absolutely a valid and compelling medium. Buy a cheap camera, go out there and start experimenting,” urged Wilson.
“When you immerse somebody in another world, and you immerse all their senses in another world, it creates a greater degree of emotional impact,” said Wilson. “And if you’re not in this business to create a greater degree of emotional impact, then why are you doing it?”
Society of Motion Picture and Television Engineers
Facebook 360 Spatial Workstation