Thanks to Chris Price, Art Director at Zubr, for this guest blog detailing the different ways in which you can present yourself as a virtual character. 

Over the years, as Zubr delves into the R&D of all these methods we’re about to examine, I have captured myself in many different ways. I enjoy the challenge of maintaining the most authenticity in my 3D representation. When I was younger playing video games with friends, the character creation tools were always a good laugh. My friends would joke that I was often found adjusting my real life appearance to try and match the avatar. Like parting my hair in a different way or changing clothes to match more closely. As the techniques of creating and capturing evolve it gives evermore flexibility and realism to represent ourselves in the metaverse.

The idea of projecting yourself into a virtual space has been around for a long time. At the most basic level this is achieved through a unique user name or picture in a game. Over the years, new games have found different ways to place yourself into their world – for example, character creation in sports games like FIFA – but now with the increasing popularity of virtual reality and the rise of the Metaverse, the importance of these avatars has increased tenfold. Our virtual selves are the medium through which we will interact, and will be the only representation others will see of us.

This opens up questions around representation, resources, and realism. Where do you draw the line between authenticity and creativity? Which will people value in the virtual world of the Metaverse? And what tools exist to help you achieve your ultimate form?

As a studio dedicated solely to the creation of immersive 3D content for AR, VR, and XR, Zubr is inherently interested in these questions and exploring their answers. Having successfully delivered over 250 projects, Zubr has experimented with a range of platforms and software to discover different methods of making virtual characters and different ways of representing people. Through our extensive R&D over the last 7 years, we’ve proven that each technique is suited to different output and skill levels. This means we’re now able to analsye which current solution for virtual character creation would most benefit a project, and high-detail authenticity isn’t always the answer!

Let’s explore some of the options currently available:

The simplest and maybe most common is avatar-based systems like Nintendo’s Mii’s or Apple’s Memojis which allow you to create a custom avatar by picking from all sorts of styles or facial features, hair and accessories. Memojis allow you to send that avatar in a series of pre-assigned poses. Devices from the latest release down to the iPhone 10 can even make of use built-in face tracking software to let users change the expression of their Memoji or even record a voice note and sync the Memoji’s lips to it. like Nintendo’s Mii’s or Apple’s Memojis which allow you to create a custom avatar by picking from all sorts of styles or facial features, hair and accessories. Memojis allow you to send that avatar in a series of pre-assigned poses. Devices from the latest release down to the iPhone 10 can even make of use built-in face tracking software to let users change the expression of their Memoji or even record a voice note and sync the Memoji’s lips to it.

Alternatively, there are direct capture methods like photogrammetry, volumetric or stereoscopic video. 3D scanning and direct capturing maintain the authenticity of a subject, making it a better option for capturing historic costumes, accents and gestures. These methods capture more than just the surface, giving you a more embodied and individual scan.

We use photogrammetry to create 3D scanned models which we can convert into fully rigged characters or avatars. Photogrammetry involves taking hundreds of photos of a subject, moving in a 360 degree circle and snapping at specific intervals so that your resulting photos create a mesh. By combining all of the photos, you can create an incredibly realistic 3D scan.

By their very nature, photogrammetry models and 3D scans are full of high-fidelity detail that deserves to be explored in 3D. It follows that 3D scans look amazing when viewed in augmented and virtual reality. Some of the technology we use includes micro, macro, aerial and video-based photogrammetry, aided by focus stacking, multi-camera arrays and marker tracking. We also use LiDAR/depth-sensing devices.

The 3D scans captured through photogrammetry are stationary but can be rigged for animation. We used photogrammetry scans of artists Declan McKenna and Aphex Twin for a series of AR social media filters used for music promo events.

‘In 3d’ is a lower tier version of this which gives you a less high fidelity model, but outputs a game-ready character, whilst Unreal’s new Metahuman system can use a full 3D scan of a face and develop a high tier, realistic, digital human avatar.

However, photogrammetry and 3D scanning are not good techniques for capturing moving subjects. Chris managed to 3D scan his family dog Frank but the resulting model wasn’t hugely detailed as Frank didn’t want to stay still!  To overcome this you can use hugely expensive multi camera rigs, which obviously isn’t an option for most people! Photogrammetry also takes much more time to shoot and process, and there is often plenty of post-production work to prevent holes in the model and to patch textures.

So if 3D scanning is great for stationary objects such as buildings, fossils, vehicles, and people who can stay still, what do we use for moving subjects? That’s where volumetric capture comes in: making use of depth-sensing cameras to capture the 3D geometry of a moving subject.

Volumetric video is the true-3D equivalent to conventional video. It captures the visual likeness of the subject just like ordinary video but also grabs its geometric shape, per frame. That means that viewers will see a 3D hologram conveying the authentic shape and movement of a living subject, without intervention from digital artists or developers.

Volumetric film captures a person’s exact actions making it a great option for personalising the movement of an avatar, but it’s not a versatile tool because the animations are baked versus giving you a rigged model, meaning you’re limited to the actions you’ve recorded and can’t respond dynamically. This is the technique we used to capture Aisha Devi and SCALPING.

Another capture system (see left), stereoscopic video is based on using two cameras set up next to each other and at the same distance to the subject to emulate the left and right eye. This creates plasticity and depth when the separate videos are composed next to each other in post-production, resulting in a “3D effect”. Zubr uses this technique when filming characters on greenscreen to be able to freely place captures of humans in augmented or virtual environments. This enables us to tell stories in VR in a human way and avoids the need to create 3D modelled characters which is time-consuming and often ends in an uncanny interaction.

Zubr recently used the stereoscopic technique to capture costumed reenactors for a historic VR experience. As this project was for a museum, we wanted to keep the experience as historically accurate and authentic as possible. This included the passengers you interact with during the VR experience so rather than 3D modelling people, which often takes an uncanny turn, we worked with a local director to cast Derry actors in each role. It was important to have genuine Derry accents and dialects, and we wanted to work with the local community as much as possible as it was their history we were bringing to life. We sourced appropriate 19th century costumes and flew to Derry to conduct a stereoscopic green screen shoot at Ulster University Magee Campus. Working with our expert photographer Misha, our creative director Jack coordinated the shoot, envisaging where each actor needed to be positioned so we could comp them into our 3D modelled scenes.

However, stereoscopic capture is somewhat limited too. In the project explained above the characters aren’t actually 3D; they are 2D and billboard toward the player so whilst they appear 3D this is an illusion created by carefully combining the two videos and positioning them within the VR scene to mimic/trick our eyes into seeing them as 3D.

It’s interesting to weigh up the pros and cons of each technique and think how they could be used in the development of the Metaverse. It doesn’t take long before you start to think about the ethics of representation. In real life there are many conversations about who gets to do what in different spaces, and this is no different in Vr or the Metaverse. Immersive media specialist and Zubr collaborator Catherine Allen, and Verity McIntosh, Researcher and Senior Lecturer for Virtual and Extended Realities at UWE, have conducted extensive research into safety in the Metaverse.

Meanwhile, many games have come under criticism lately for not including a wide enough range of character customisation options. For example, Nintendo’s highly successful Animal Crossing: New Horizons faced backlash for its lack of inclusive hairstyles, resulting in the developers updating the game and adding new customisation options. However, some felt this resulted in stereotyping. Equally, in most games players are limited to existing stock animations, body types, and mobility levels, leaving many disabled people feeling unrepresented or unwelcome in virtual spaces.

This leads to our final point: if you had the choice to look however you wanted, would you want to look like you, or completely different? Does authenticity matter in the Metaverse? And what does it mean for our social interactions online, in VR, and in the Metaverse when we don’t really know who we’re talking to?