Does authenticity matter in digital avatars and virtual characters?

The idea of projecting yourself into a virtual space has been around for a long time. At the most basic level this is achieved through a unique user name or picture in a game. Over the years, new games have found different ways to place yourself into their world – for example, character creation in sports games like FIFA – but now with the increasing popularity of virtual reality and the rise of the Metaverse, the importance of these avatars has increased tenfold. Our virtual selves are the medium through which we will interact, and will be the only representation others will see of us.

This opens up questions around representation, resources, and realism. Where do you draw the line between authenticity and creativity? Which will people value in the virtual world of the Metaverse? And what tools exist to help you achieve your ultimate form?

Let’s explore some of the options currently available:

The simplest and maybe most common is avatar-based systems like Nintendo’s Mii’s or Apple’s Memojis which allow you to create a custom avatar by picking from all sorts of styles or facial features, hair and accessories. Memojis allow you to send that avatar in a series of pre-assigned poses. Devices from the latest release down to the iPhone 10 can even make of use built-in face tracking software to let users change the expression of their Memoji or even record a voice note and sync the Memoji’s lips to it. like Nintendo’s Mii’s or Apple’s Memojis which allow you to create a custom avatar by picking from all sorts of styles or facial features, hair and accessories. Memojis allow you to send that avatar in a series of pre-assigned poses. Devices from the latest release down to the iPhone 10 can even make of use built-in face tracking software to let users change the expression of their Memoji or even record a voice note and sync the Memoji’s lips to it.

Alternatively, there are direct capture methods like photogrammetry, volumetric or stereoscopic video. 3D scanning and direct capturing maintain the authenticity of a subject, making it a better option for capturing historic costumes, accents and gestures. These methods capture more than just the surface, giving you a more embodied and individual scan.

We use photogrammetry to create 3D scanned models which we can convert into fully rigged characters or avatars. Photogrammetry involves taking hundreds of photos of a subject, moving in a 360 degree circle and snapping at specific intervals so that your resulting photos create a mesh. By combining all of the photos, you can create an incredibly realistic 3D scan.

By their very nature, photogrammetry models and 3D scans are full of high-fidelity detail that deserves to be explored in 3D. It follows that 3D scans look amazing when viewed in augmented and virtual reality. Some of the technology we use includes micro, macro, aerial and video-based photogrammetry, aided by focus stacking, multi-camera arrays and marker tracking. We also use LiDAR/depth-sensing devices.

The 3D scans captured through photogrammetry are stationary but can be rigged for animation. We used photogrammetry scans of artists Declan McKenna and Aphex Twin for a series of AR social media filters used for music promo events.

‘In 3d’ is a lower tier version of this which gives you a less high fidelity model, but outputs a game-ready character, whilst Unreal’s new Metahuman system can use a full 3D scan of a face and develop a high tier, realistic, digital human avatar.

However, photogrammetry and 3D scanning are not good techniques for capturing moving subjects. Chris managed to 3D scan his family dog Frank but the resulting model wasn’t hugely detailed as Frank didn’t want to stay still!  To overcome this you can use hugely expensive multi camera rigs, which obviously isn’t an option for most people! Photogrammetry also takes much more time to shoot and process, and there is often plenty of post-production work to prevent holes in the model and to patch textures.

So if 3D scanning is great for stationary objects such as buildings, fossils, vehicles, and people who can stay still, what do we use for moving subjects? That’s where volumetric capture comes in: making use of depth-sensing cameras to capture the 3D geometry of a moving subject.

Volumetric video is the true-3D equivalent to conventional video. It captures the visual likeness of the subject just like ordinary video but also grabs its geometric shape, per frame. That means that viewers will see a 3D hologram conveying the authentic shape and movement of a living subject, without intervention from digital artists or developers.

Volumetric film captures a person’s exact actions making it a great option for personalising the movement of an avatar, but it’s not a versatile tool because the animations are baked versus giving you a rigged model, meaning you’re limited to the actions you’ve recorded and can’t respond dynamically. This is the technique we used to capture Aisha Devi and SCALPING.

Another capture system (see left), stereoscopic video is based on using two cameras set up next to each other and at the same distance to the subject to emulate the left and right eye. This creates plasticity and depth when the separate videos are composed next to each other in post-production, resulting in a “3D effect”. Zubr uses this technique when filming characters on greenscreen to be able to freely place captures of humans in augmented or virtual environments. This enables us to tell stories in VR in a human way and avoids the need to create 3D modelled characters which is time-consuming and often ends in an uncanny interaction.

Zubr recently used the stereoscopic technique to capture costumed reenactors for a historic VR experience. As this project was for a museum, we wanted to keep the experience as historically accurate and authentic as possible. This included the passengers you interact with during the VR experience so rather than 3D modelling people, which often takes an uncanny turn, we worked with a local director to cast Derry actors in each role. It was important to have genuine Derry accents and dialects, and we wanted to work with the local community as much as possible as it was their history we were bringing to life. We sourced appropriate 19th century costumes and flew to Derry to conduct a stereoscopic green screen shoot at Ulster University Magee Campus. Working with our expert photographer Misha, our creative director Jack coordinated the shoot, envisaging where each actor needed to be positioned so we could comp them into our 3D modelled scenes.