MONTREAL, CANADA—As the audio industry has evolved from mono to stereo to quad to surround to immersive sound formats, there have always been those willing to blaze a trail into the unknown. Now, with immersive audio beginning to take off, Jean-Pascal Beaudoin and a team at Apollo Studios in Montreal are pioneering what is sure to be the next big thing—sound for virtual reality, or VR.
Beaudoin, director of special projects at the creative music and sound company, which has won multiple awards for its advertising and experiential presentation work, is one of the first to wrestle with how to create a realistic soundfield to match interactive, all-encompassing, live-action virtual reality images. A meeting with virtual reality startup company Oculus VR at South by Southwest in early 2013, at which Apollo Studios demonstrated its first audio-for-VR project, has since catapulted Beaudoin and his team to the forefront of the new format.
Virtual reality is rapidly picking up momentum. Earlier this year, Facebook acquired Oculus VR for $2 billion, while Google sank nearly $550 million into another VR startup, Magic Leap, this past summer. Meanwhile, VR cameras are beginning to appear on the market from 360Heroes, Jaunt (with investment from UK broadcaster BSkyB), Samsung and others, including a DIY cardboard device from Google that works with a smartphone. For its part, Apollo produced a live-action VR trailer (VR has so far been largely computer-generated) to demonstrate a special edition of Samsung’s Gear VR headset, developed in partnership with Oculus, which was officially announced on November 11.
But audio tools applicable to VR have been lagging behind. First and foremost, how do you capture audio that not only offers a 360-degree perspective but can change perspective as the viewer’s head swivels? Beaudoin went back to fundamentals, adapting a system first introduced in 1881: binaural sound. “Obviously, the new thing with VR is that it has to be 360 degrees, which does bring a level of complexity,” he says.
For that 2013 SXSW project, a short performance film of Canadian singer/songwriter Patrick Watson made by Felix Lajeunesse and Paul Raphaël of Felix & Paul Studios, Beaudoin had a custom microphone made. “It’s very simple—quad binaural. It’s similar to the one used for the Beck experience, I learned afterwards,” he says, referring to an interactive film by music video director Chris Milk for car maker Lincoln which featured Beck performing in the round with 160 musicians. That film debuted the month before SXSW 2013. An ambisonic microphone system or the VisiSonics 5/64 audio camera—a sphere encapsulating five HD cameras and 64 mic elements— could also be worth trying, he says.
But the real challenge comes during mixing. “So far, it’s been only recording and doing some EQ and leveling; not that much post production. It’s basically 3D positioning, but the only 3D positioning tools that exist are for cinema. So we’re really at a point where it’s a totally new venture for sound.”
For example, having captured a bed of multiple binaural recordings, “The only way you can assemble this into a seamless 360 experience is into a game engine such as Unity or Unreal. The way we’ve been working so far is with zero degrees, 90 degrees, 180, 270, and then, in the game engine, cross-fading those mixes. It’s basically head tracking.
“But then you have to do the post production, because you want to add sound design. I’m going into Pro Tools and working on it, and listening to it, and back into Unity, so it’s a very tedious process.”
Nevertheless, so far the results of his workflow have been good, he says. “It’s seamless. But probably the best way to do this is to work with gaming middleware, like Wwise and FMOD—more like an audio-for-gaming workflow.” Ultimately, he says, object-based cinema formats that can render to a binaural output could offer a solution. “I would bet that in six months, my workflow will have changed. And it does bring someone new into the equation of mixing, which is the sound programmer. I think it’s going to foster a new type of hybrid mixing engineer that isn’t purely gaming—it’s mixing very, very real sound with post.”
There are mix tools emerging such as the VisiSonics RealSpace 3D Gaming Engine and the 3D AfterEffects Engine, which enables offline immersive audio mixing via a VST plug-in. In October, Oculus announced it had licensed the technology, which was developed at the University of Maryland and combines head-related transfer functions, head tracking and room models. “A 3D positional plug-in will do the job instead of having to do it by hand, which is how I’m having to do it right now,” says Beaudoin.
“Every project, there’s a technological iteration. The way we’re working, recording, doing post production, every project, we learn something very important and then move on to the next. We’re really not at that level where it’s just regular business. It’s so exciting.”
The Oculus plug-in licensing deal is also an indication that VR developers are aware of the importance of audio. “They really want to elevate the quality of sound, because they do understand that sound is fundamental to having the best VR experience.”