AV’s virtual viewpoint

VR’s magic lies in its ability to transport the subject into a remote environment where they become physically present in a non-physical world. Zoe Mutter explores the engaging and empathetic virtual realm.

Virtual reality has the power to take you anywhere, giving access to extraordinary experiences that might otherwise only be available to a select few. The medium has been developing for decades – one of the earliest incarnations being Morten Heilig’s multi-sensory Sensorama machine in the ‘50s – but is 2016 the year VR becomes a viable product?

“The over-hyped attempts of the mid ‘90s such as Nintendo’s Virtual Boy couldn’t deliver truly immersive experiences and in many cases left you feeling nauseous. Well, now you can believe the hype,” says Sol Rogers, ceo of VR production agency, Rewind.

With major advancements in graphics, screen resolution, fields of view, head and spatial tracking and computer processing powers, headsets can now achieve the real magic ingredient of VR – presence. This equates with immersion; an incredibly powerful sensation that’s unique to VR.

VR in action

Gaming craves immersive experiences, making it a no-brainer VR application. When film studios and broadcasters first adopted the technology the audience was restricted due to the lack of mass-market, high quality VR headsets. Now that these devices are becoming more affordable and improving in quality these experiences should find a larger audience.

According to Dejan Gajsek, VR evangelist and marketing lead at VR company VIAR, Hollywood’s experimentation with VR has poured gasoline on the fire, with the help of ambassadors like actor and director, Kevin Spacey, and director and ceo of VR company Vrse, Chris Milk who refers to it as the “ultimate empathy machine”. VIAR is developing the Viar360 web platform to help VR and 360˚ content producers create better stories backed by analytics.

But while VR’s influence on the entertainment world is undeniable, its applications extend far beyond to impact nearly every pro AV sector; from architecture and property to education and visitor attractions.

VR use by enterprises and the public sector is increasing while architectural and construction firms use VR to bring blueprints to life, ironing out flaws before anything is built and avoiding costly revisions. This means architects can design within a 3D environment and potential property buyers can virtually visit a property.

The initial kickoff for VR in the industrial environment started in the ‘90s when companies in the oil and gas, automotive and education industries began exploring the benefits of using VR.

“Where consumer VR has allowed these industries to explore the benefits of VR at a low investment point, they now see the value of professional VR and can consider the next investments,” says Kurt Doornaert, business and market director virtual reality worldwide, Barco.

All sectors can benefit from using VR for visualisation, says Allan Rankin, business development consultant at VR production company, Inition: “We’ve facilitated everything from real estate viewing of fly-throughs of existing buildings through to demonstrating the process of going through a copper mine.”

Similarly VR software company WorldViz has created full-scale walkthroughs of hospital construction projects and helped a pharmaceutical company visualise specialised machinery for design and marketing purposes.

Companies such as Marxent are focused on VR’s impact on retail and sales applications, using it to help with visual merchandising, product visualisation and configuration. Lowe’s Holoroom uses Oculus software developer’s kit combined with Marxent’s 3D content management system VisualCommerce to allow shoppers to design a kitchen or bathroom on an iPad, view it in 360 degree VR using an in-store Oculus Rift headset and export the design to YouTube 360 to view at home with Google Cardboard.

“From reinventing visualisation and product development to changing shopping behaviours and engaging customers in immersive brand experiences, leading brands are already starting to integrate VR solutions,” says innovation director, Amelia Kallman, Flux, an innovation lounge created by creative consultancy agency, Engage Works.

VR provides insight and speeds up build times in the construction industry and when analysing complex assembly issues too, making it possible to see which workflows will succeed before making a final decision. Holovis also uses it to create simulators that reinforce potential dangers people face to reduce complacency.

Elsewhere, agricultural and construction equipment company, Case New Holland carries out collaborative design reviews via its eight ActiveWall VR systems, plus an HMD-based ActiveSpace, both systems from pro VR company, Virtalis, whilst mining company, Vale, in Brazil, is using Virtalis’ GeoVisionary software to view stereoscopic 3D data in VR, enabling it to plan and marshal resources.

The automotive industry is benefitting not only in terms of virtual driving tests and test drives, it is also integrated into the complex manufacturing process. While Ford and Honda have used headsets to aid design and development, Audi is enhancing the customer experience using HTC Vive and Oculus Rift technology at select showrooms. Visualise’s VR driving experience showing off the Audi A4’s Virtual Cockpit involves a custom rig placing the camera at the driver’s eye height.

In the medical sphere, VR is allowing people with phobic disorders to face their fears in a safe environment too. And although the medium has been adopted for military and pilot training for years, it is also aiding training within the medical sector and to test implants and devices.

Elsewhere in the education sector, Google has teamed up with teachers and content partners to create Expeditions – a virtual reality platform for the classroom. Expeditions’ guided VR tours include trips to Buckingham Palace or the Great Barrier Reef in an experience produced by Alchemy VR and Sir David Attenborough.

VR is enhancing the experience for museum goers too, thanks to attractions such as Dreams of Dali at the Dali Museum in Florida – created by agency Goodby Silverstein & Partners. The visitor is placed within the painting via an Oculus Rift paired with an audio element for a multi-sensory experience. Similarly the Tank Museum at Bovington in Dorset, UK features a VR video walkthrough of a collection of tanks and in-game 360 VR video.

In the realm of visitor attractions, VR technology such as Holovis’ experiential design tool, RideView brings the attraction or whole site master plan to life in 1:1 scale and realtime, so people can walk through a park or ride an attraction from any user demographic. The technology is making theme park rides more immersive as well. Take Alton Towers’ Galactica, for example, a team effort between Samsung Electronics, Merlin Magic Making and Figment Productions (see our case study on page 63).

Filming techniques

VR content comes in the form of filmed material, computer generated imagery or reconstructed rendering. The filming process requires specialist camera equipment, capture position consideration, post production optimisation and attention to the user journey.

Camera placement is crucial because that is the spot the viewer has point of view from. While creative directors are used to working with a focal point for traditional filming, VR is about environment filming which requires thinking about softer situations and subtly weaving character and plot development.

As viewers absorb more of the intended story when watching film and TV compared to VR – which directs their attention in multiple directions – camera placement and obvious eye linked movement in the scene is pivotal to maintaining users’ attention.

Cameras suited to VR are capable of taking in 360 degrees, with a typical set-up comprising a rig holding multiple cameras shooting at a certain angle with a camera field that overlaps to eliminate gaps in the footage.

As the capture invokes a 360° viewpoint, lighting conditions vary from each angle so stitching footage together without massive disparity is an art. “This is crucial because when a viewer discerns abnormalities in content, the narrative, content and playback’s power is diminished,” says Inition’s Rankin.

Proximity to camera is imperative for MPC (Moving Picture Company) – a visual effects studio that has developed VR experiences to run alongside films such as The Martian. “It’s important to keep people at a distance so lens and stitch will work. Keeping the camera steady or static is necessary to avoid nausea,” says Tim Dillon, executive producer at MPC VR.

MPC works with many types of camera array and workflow and can match the technology to the brief and end product. “We developed new camera tools and CG workflows for our Goosebumps VR experience to cope with the new challenges of a hybrid live-action and CG production,” adds Dan Phillips, head of digital and interactive, MPC.

Frame rate and resolution

Resolution and frame rate are important for visual effects and creative studios such as Framestore to achieve lifelike experiences. “With a traditional PAL or NTSC frame rate, when you move your head around there are not enough frames of information for it to feel smooth,” says Karl Woolley, who leads Framestore’s digital, VR and immersive content team in London.

“You need to shoot at a high frame rate and resolution which is why we tend to use custom rigs we machine ourselves or have machined for us with professional grade cameras from manufacturers such as RED or Blackmagic.”

“Like shooting a traditional film or commercial, there’s never just one camera, lens or lighting combination that works. I can see why people use the low-end solutions – they’re affordable and flexible. However, they don’t offer the fidelity, quality, colour depth and frame rate we need for professional projects such as the Game of Thrones VR activation for HBO.”

Other professional options include the Jaunt ONE stereoscopic cinematic VR camera and Nokia’s Ozo which captures 360 spherical video and surround sound. Live action VR company NextVR’s broadcast platform captures the geometry of the environment around the camera and maps video to wireframes, recreating content that doesn’t need stitching. As well as producing live action content allowing fans to experience sporting events such as the NBA from the best seat in the stadium, NextVR is rolling out the first live VR production truck for events coverage, with one-third of the truck comprising a mobile audio mixing facility.

“Quality of footage is important because it means content is believable, comfortable and transfers the experience’s emotional value,” says David Cramer, svp corporate strategy, NextVR.

Facebook’s open source camera rig could have a huge impact in live action content too. Using a 17-camera array and accompanying web-based software, Facebook Surround 360 captures images in 360 degrees and renders them automatically. Lytro’s Immerge camera – a professional Light Field solution for cinematic VR – could have similar implications for immersive live action possibilities.

Cinematic VR experience producer, Jaunt Studios, is also exploring VR in live streaming broadcast and events, producing VR music video and film experiences for artists such as The Grateful Dead and Sir Paul McCartney. The company also collaborated with travel company, Mountain Travel Sobek, capturing VR films to be viewed via the Jaunt VR app.

Broadcaster Sky – which invested in Jaunt in 2013 – further committed to VR in March with the launch of its in-house VR production unit, Sky VR Studio. Its first pieces of fully immersive VR content transported viewers to the Formula 1 testing in Barcelona. Over the coming year, the studio will drive creation of cinematic, fully immersive VR content, producing more than 20 films, from news to sporting events.

Simon Reveley, md at VR and 3D film-makers Figment Productions, expects future cinematic VR experiences will be a blend of recorded and synthesised imagery and that future filmed VR content will involve light field technology, giving the viewer multiple viewpoints on a single scene.

“Future light field technologies from companies like Lytro and 4D video solutions from the likes of 8i and Uncorporeal have the potential to capture moving elements as if they were 3D objects in a scene,” he says. “When blended with CGI and reconstructed rendering, these new types of media could deliver realworld assets into VR scenes.”

As well as resolution and frame rate, a system with several sensors and lenses, positioned as close as possible is key, points out Alexandre Jenny, senior director immersive media solutions at camera manufacturer, GoPro.

To make VR filming accessible to consumers, GoPro’s VR system Omni builds on the fisheye lens-equipped GoPro unit and comprises a rig of six cameras synchronised at a pixel level. Kolor’s software then gives users control over each frame’s stitching. This single system is optimised for VR and acts in sync rather than several cameras acting independently within a rig.

Also aimed at consumers, LucidCam is a 3D, 180-degree VR camera that fits in your pocket. Videos can be uploaded and shared instantly and viewed through a headset.

Hardware and headsets

VR explorers are spoilt for choice when selecting a headset to transport them into a virtual world. Oculus founder, Palmer Luckey, kick-started the second coming of VR with the launch of the Oculus Rift headset.

In February, VR took centre stage at Mobile World Congress in Barcelona. The Samsung Electronics Unpacked press conference attendees were handed Gear VR headsets to view the event before Facebook founder Mark Zuckerberg, whose Oculus unit helped develop the Gear VR, discussed VR’s effect on social networking and mobile phones. Facebook is heavily investing in VR to deliver enhanced social experiences as Zuckerberg believes VR is “the next platform, where anyone can create and experience anything they want.”

Since Facebook bought Oculus in 2014 for $2 billion many of the key players have heavily invested in what they see as the future of communication. As well as the release of Samsung Gear VR – a headset powered by Oculus technology and a Samsung smart phone – the HTC Vive was released in April which comprises a headset accompanied by two controllers tracking the user’s movement.

Elsewhere, PlayStation VR – the VR gaming system for PlayStation 4 – will be available from October and Microsoft’s HoloLens device which embraces virtual and augmented reality to create a mixed reality, is now shipping. VR headset manufacturer Fibrum also offers special lenses, presenting users with poor eyesight with a clear picture in virtual reality, from -5 to +5 variance.

To help increase VR’s widespread, mainstream adoption, Google developed a more affordable option in the form of Google Cardboard – a simple viewer anyone can build or buy to transform a smart phone into a VR device.

“The ability to place smart phones into simple cardboard headsets is amazingly cost effective, making them ideal for large scale brand activations,” says Adam Hopkins, senior account manager at experiential technology production company, Phygital.

When choosing playback technology, parameters such as panel resolution, physical comfort, whether delivery is wired or wireless and the user’s potential range of movement needs to be considered.

Content type also determines the viewing equipment chosen. “3D 360 video plays easily on a smartphone and Google Cardboard, but a detailed, fully immersive interactive experience needs the latest graphics card in a high-end machine and is better viewed with something like Oculus Rift,” says Jason Higgins, md at digital reality studio Harmony.

Frame rate and motion-based screen blur are common research areas for hardware companies trying to counteract sickness. The mind is susceptible to inconsistencies between what the eye sees and what the head is sensing which, as noted with body motion, creates a sense of confusion and potentially illness.

Alx Klive, ceo at 360 Designs – a VR camera and production company that has captured VR footage at events such as The Oscars and Coachella using Blackmagic cameras in a rig – believes headsets are currently the weak link. “You can capture 360 degree video in 12K without much difficulty, but displays are limited to around 3K right now. Resolution therefore is secondary to things like great image quality and wide dynamic range.”

But VR hardware is redundant without the software helping create realistic, interactive 3D virtual environments. For many embarking on their first VR journey, the natural starting point when developing a VR software environment is to look at gaming engines.

“However, whilst using gaming engines can get you partially along the road to success, they are not focussed on handling large CAD models, point cloud data, multi-user collaboration and developing immersive VR environments,” says Andrew Connell, technical director at Virtalis which built Visionary Render software for commercial VR users looking for an alternative to generating virtual worlds using gaming engines.

But VR isn’t only about headsets. Projection systems also allow people to collaborate in groups on designs and walk-throughs. “Tracking in some of these design walk-throughs is warehouse-scale which is very important,” says Peter Schlueer, president, WorldViz. “Many headsets tout room-scale, but for enterprise use, warehouse-scale tracking of one or multiple people in 3D space is critical.”

Immersion through audio

Realistic sound through positional audio capture and delivery is an essential component of immersive VR experiences. When sound is perceived to come from the same direction as a visual stimulus – known as binaural audio – the virtual experience’s credibility is increased.

“Great VR is when you can close your eyes and still understand the environment around you,” says Harmony’s Higgins.

Ambisonics, a tried and tested technique to capture a sound field in all directions, is keeping pace with VR. At this year’s CES, Sennheiser announced a strategic focus on immersive audio with the unveiling of 3D audio platform, AMBEO. One of the products showcased launching later this year was a microphone developed in conjunction with VR content producers that captures high-quality audio in three dimensions, translating the positions of sound sources around the viewer in VR.

Elsewhere in the 3D sound sphere, Barco acquired immersive audio specialist IOSONO to better equip it to offer immersive VR experiences. One example of its application is at Bauhaus University in Germany where an IOSONO sound system has been combined with a projection-based display to conduct VR research.

OSSIC – developers of the OSSIC X headphones for VR, gaming and music, – recently closed its Kickstarter campaign, capturing over $2.7 million from backers. As this surpasses Oculus’s earlier record of $2.4m for a VR product it is further testament to a growing support to help evolve VR technology.

“3D audio is the most natural way to direct the user and is key to immersion and presence. But while VR audio content is moving to full 3D, the visual field of view is still limited to 10 to 15 per cent of the full sphere at any given time,” says Jason Riggs, ceo, OSSIC.

Elsewhere, Scott Gershin, media and entertainment company Technicolor’s director of sound editorial, has established a new department at the company to meld technology with storytelling to create soundscape experiences for VR projects, video games, theme parks and art installations.

“No longer are we passively experiencing stories through a window, we’re witnessing the evolution of being there and participating in experiences,” he says. “As an audio artist, I’m excited to have the opportunity to create new sonic landscapes that enhance the story and often help drive the experience.”

VR’s presence in AV

Some feel VR has a way to go until it becomes mainstream due to limitations such as bulk, cables, equipment, set-up, production costs and a feeling of nausea. Harmony’s immersive experiences use motion seating to offset the sensation of movement and avoid sickness.

“The sickness is partly down to producing poor VR content, usually through inexperience or the inherent nature of the way in which people experience VR differently. However, graphics quality is improving and will continue to do so as hardware improves,” says Inition’s Rankin.

Realtime 3D artist at Tale of Tales, Michaël Samyn, whose project Cathedral in the Clouds, includes a VR component – a realtime 3D environment – believes most headsets are still inaccessible in terms of cost and because the hardware is inelegant.

“Hardware speed is also a limitation,” he says. “Not so much to make applications run faster – conventional design, a big budget and a team of engineers can do that too – but to allow for more fluid creation.”

Realtime rendering can be tough in VR as the rendered resolution and frame rate must be consistently high. This requires advanced graphics, making this hard to achieve without bulky and expensive hardware.

However, according to research from Futuresource Consulting, consumer video and games VR content is expected to reach $8.3 billion by 2020. In the pro AV world, Futuresource Consulting’s associate director, Chris Mcintyre Brown believes VR as a content format provides an opportunity for both B2B and B2C focused businesses: “It gives content producers the option to use richer more interactive video formats to that of 2D content, helping convey product and service propositions in a more compelling and engaging way.”

Barco’s Doornaert expects VR will have most impact on pro AV by improving design, engineering, manufacturing and construction whilst helping reduce the number of errors and improving communication.

MPC’s Dillon sees VR will have a large impact on communications: “We’re yet to see apps such as Skype for VR launch, but this will be a game changer in terms of why someone would want a headset in their home.”

Content is key to VR’s future success as, like most mediums, the power is in the specialist creation of content. “This means AV companies will need to partner with content providers to offer turnkey solutions,” says Inition’s Rankin.

According to Figment’s Reveley, the VR industry believes 2016 is the start line for the next generation of virtual reality technology. “However, it’s likely to take some time before it becomes ubiquitous like PCs, tablets or mobile phones – perhaps five to 10 years.”

By which time it’s likely to have evolved into a mixed reality format with a blend of augmented reality and virtual reality. Stuart Hetherington, ceo at Holovis agrees mixed reality will create a step-change in experience design, allowing the user to move freely and safely in the space whilst being exposed to effects and extending the boundaries of storytelling.

“What’s real and what’s imagined is now blended using the core VR and AR technology alongside projection mapping, show sets, virtual assets, spatial audio and SFX,” he says. “This takes immersion to a whole new level with the added benefit that the gamification layer introduces almost limitless possibilities for individual and group interaction.”

In the next decade, VR might be viewed as a necessary advanced communication and understanding tool rather than a specialised technology, accessible to all to work solo or collaboratively, enabling businesses and organisations to improve decision-making and to respond quicker to dynamic changes. “The role of VR in pro AV will be huge – VR is the ultimate medium, creating an incredibly compelling sense of presence,” concludes WorldViz’s Schlueer. “VR naturally grabs people’s attention and draws them in which makes it a powerful marketing tool to engage customers interactively, communicate product value, and build brand awareness.”

Have your say

or a new account to join the discussion.