First-of-its-kind musical VR experience re-imagines 'Hallelujah' in brand new light

— -- While virtual reality might still be at the genesis of its development, some innovative minds have put its future potential on dizzying display at the Tribeca Film Festival today with the release of the groundbreaking VR music experience "Hallelujah."

In an interview and preview screening with ABC News, award-winning director and "Hallelujah" creator Zach Richter sat down alongside its technological and creative producer Chrissy Szczupak to discuss the first-of-its-kind VR musical set piece based around the Leonard Cohen classic.

"Music is very digestible and emotionally resonates," Richter said about why he wanted to create something around this song. "Music drives creativity and experimentation."

The project was launched as partnership between Within, a story-based virtual reality company, and revolutionary imaging platform technology company Lytro, where Szczupak is based.

Richter worked with childhood friends Eames Kolar and Bobby Halvorson to reimagine the late Cohen's most iconic song, and produce an original music experience with an uncompromising sense of presence.

"It all started when my buddy [Eames Kolar] texted me and said: 'Wouldn't it be cool if there were five Bobby's standing around you singing?'" Richter recalled. It was that kernel of an idea back in January 2016 that sparked a journey to produce this musical composition unlike anything previously attempted.

Halvorson, a musician with a vast background in production, composition, singing and playing, was instrumental to the process of recreating the famed song.

Staying true to the song's roots in a capella with heavy religious influence, Halvorson utilized a medieval composition technique called hocket. That process uses multiple voice parts to sing rapidly alternating notes and short phrases, creating a "hiccuping effect."

Over the course of six months, Halvorson worked with the Within team to compose his rendition of the song that has been famously covered by "nearly everyone," Richter said. Cohen was said to have originally written 81 verses and Halvorson similarly took his time to strip his rendition all the way down from nine minutes to nearly five.

The production team utilized Lytro's Immerge technology, which has a one-meter wide camera with a hexagon full of multiple camera lenses. The process captures every angle in space using light field technology around a subject to create a sense of volume, space and dimension that standard 360 degree video cannot provide.

The field of visuals that this provides, paired with the highest quality audio recording, makes the depth of the VR experience feel completely human.

Lytro seamlessly blends the physical and digital worlds in the post-production process by isolating the exact distance from the viewer and the source with high-end spatialization, so when the viewer moves within the experience, sound in virtual time and space changes as it would in real life.

"It's a lot of math," Szczupak said laughing. "The omni-directional technology then projects sounds from exactly where the viewer is standing."

The "Hallelujah" experience begins with Halvorson's voice directly in front of the viewer. He emerges before the user standing alone with nothing but his single melody, a baseline. Then his lowest baritone pops up to the left and as you turn to follow the sound, more of his own vocal counterparts chime in piece-by-piece, filling the space with the perfect pitch.

"We wanted to strip down the components of the piece completely down to the essentials to make you feel fully immersed," Richter said. "Using this technology creates an incredible sense of presence."

As the song begins to approach the final verse, there is a fade-to-black followed by a bright reveal that moves the viewer to the altar of a cathedral. There, Halvorson stands front and center, leading a formal choir dressed in white robes.

One of the lyrics from Cohen's song depicts a marble arch, and with Lytro’s 6DoF technology, the viewer can actually look down to see sparkling marble floors of St. Ignatius Church in San Francisco. At the same time, the user can turn around and have voices projecting from behind as empty church pews occupy the visual space.

The final product, which the entire California-based team said they are extremely proud of, is a beautifully simple and organic music experience unlike anything yet utilized in the VR format.

"Hallelujah" flips the technology of virtual reality on its head, pushing the boundaries to the point where the cutting edge technical aspects almost becomes an afterthought.

As Szczupak summed it up, "When you experience 'Hallelujah' it allows you to forget about the technology and melt into the song."