Technologies behind the Social Shopping VR Experience in High Fidelity

Several weeks ago, the Content and Prototyping team at High Fidelity was tasked with the mission of exploring what it means to go shopping in a social environment within virtual reality. While 2D avatar customization systems have existed for some time, and various VR platforms handle their avatar creation systems differently, the functionality to support user-generated content both from a wearable and avatar perspective was a new challenge to tackle.

As we’ve spent time experimenting with what it means to explore tactile avatar and identity customization within a virtual world, we wanted to break down our approach for creating the first version of our avatar shopping experience within the High Fidelity platform.

Right off the bat, there were several assumptions that we wanted to explore and validate as we designed a VR shopping experience. Early implementations of avatar shopping experiences involved the idea of a shelf entity that could be cloned by a user when it was grabbed. This item only existed while it was in the users hand, and when it was brought close to a specific joint, specified in the object’s uniquely defined user data properties, it would create an attachment and apply that transform. There was no user customization built into the try-on process at that stage, but we were able to verify that the ability to try on and customize your avatar in a spatial way was something to continue exploring.

For a fully featured store, we wanted to improve the existing design. We needed to build a more complex and robust interaction system that addressed the limitations of the previous design in three key ways:

1. Shopping needed to work without users having permissions to create, or rez new items within the domain.
2. We wanted to experiment with how it felt to use our hands in VR to scale and adjust items on our avatars, so they needed to be persistent between users.
3. The store should support the idea that social shopping and being able to help friends adjust items on their avatars was a fun and engaging way to explore and customize a virtual identity.

The Concept of a ‘Wearable’

High Fidelity supports the ability to customize your avatar with different clothes and accessories in several ways. Your avatar usually will have a base look to it with the mesh and textures built-in to the model itself. But for on-the-fly customization, there are two options for adding items to your avatar. The simpler of these is what we define as an attachment, a the structure in the engine for defining a model file and a transform that becomes a part of your avatar. To create a hat in this manner, you would specify a URL to an FBX or OBJ formatted hat (3D object), and the position, rotation, and scale of that hat on your avatar’s head.

The more complex and powerful option for attaching items to your avatar takes advantage of one of the core features of in-world components of the High Fidelity engine, the entity. Often times, entities are objects that live in a single domain server in High Fidelity, and these can contain a robust number of properties ranging from physical behaviors to custom scripts with compound logic. A second type of entity exists, called an avatar entity, which is tied to a user’s client rather than the server. These wearables can travel with a user across domains, and enable worn objects to contain more complex and rich features. 

We decided to focus on a store that created wearable objects as avatar entities, giving us flexibility to add scripted behaviors down the road to objects that could be part of an outfit or avatar, and enable users to have the flexibility to interact with the items once purchased while remaining in VR.

Today’s avatar store consists of a set of store and trial entities, which are spawned as avatar entities after purchasing.

The Store Copy

In High Fidelity, there are currently three levels of permissions that can be granted to users that relate to their ability to create new content in the world:

• The user has no permissions to create anything in the domain.
• The user has permissions to create anything they’d like in the domain, and what they create has a finite lifetime.
• The user has permissions to create anything they’d like in a domain, with no restriction on its lifetime.

In the case of the last two permission levels, domain operators can apply finer grained permissions via domain edit filters. But being cautious about granting rez permissions to unknown users remains the most secure. So, we made the decision to lock down the store domain and offload the ability to create the trial copies of the items in the store to the entity script server, a component of the domain that runs scripted behaviors for an item on the server side, rather than the client side. For our store, this store copy would be responsible for making sure that there was at least one trial version available for someone to adjust and try on (and ultimately purchase) and handle spawning new wearables if there were no available try-on copies. The store copy of each item contains user data that specifies the joints that the wearable can be attached to, as well as a reference to the purchasable entity’s marketplaceID, a unique identifier that is used to bring up the store page within the High Fidelity Marketplace. The store items are then locked so that the originals could not be moved or manipulated by anyone in the domain, and generate trial copies as children.

The Trial Copy

At any given time in the store, there is one trial copy available on the shelf for a user to grab and try on. We made a few design decisions for the trial copies based on the available feature set for entities in High Fidelity:

• The trial items are almost identical copies of their parent store copy with regards to their physical properties, but have different scripted behaviors that enable them to be tried on. AttachmentItemScript.js defines all of the logic required to handle removal from the shelf, being tried on, and being removed from an avatar.
• Trial copies are spawned in the exact same location as their parent, and are created as invisible copies to avoid z-index fighting with their parent model. Because entity server scripts do not have any understanding of the physics engine, the store copy does not respond to grab events from user hand controllers, so startNearGrab events in the engine are detected instead by the invisible trial entity. When the attachmentItemScript code detects that a user has grabbed the store copy, it makes the child visible and the user seamlessly experiences the ability to grab something they want to try on.
• Each trial entity, when created, has the parentID property set to the store item entityID that spawned it. These entityIDs are unique identifiers that reference a specific instantiation of an entity within a domain. Within the attachment item script on the child entity, we can reference the parentID to determine the state of a given item,and whether it’s still “on the shelf”, being worn by someone, or that it has been abandoned in the store. We use this state to know when to use the Messages API in High Fidelity to tell the store copy to create a new child to replace the one being tried on.
• Trial copies have a lifetime property set as soon as they’re spawned that defines how long they’ll exist in the domain. When the trial copy is attached to an avatar and being worn, the lifetime property is set to -1, which prevents it from being removed. Once an entity is removed from the shelf, it has a lifetime of 60 seconds before it is automatically cleaned up if no one wears it within that time frame.

attachmentItemScript.js

The High Fidelity engine automatically handles the core infrastructure of grabbing and manipulating objects, and we took advantage of that within attachmentItemScript.js. The ability to grab, scale, rotate, and move items are already covered, so we just needed to do a bit of setup, then hook into the grab events generated from a user in-world. In attachmentItemScript.js, we created a JavaScript prototype for an AttachableItem and connected to several world events fired for each client in the space with visibility into the entity.

Functions

preload: This is run on every client once the client becomes aware of the entity. The script gets information from the entity using the Entities API to determine how the entity should behave. In preload, the client becomes aware of which joints on an avatar an item can be attached to, whether or not the item is attached to someone, and if the item is still on the shelf. If the item is still on the shelf (determined by Entities.getNetableType, which returns whether or not the ID passed in is an avatar or entity), the client subscribes to a message channel to prepare to tell the store copy that the item has been removed if that user picks up the trial version.

starNearGrab: This is run on the client responsible for picking up the item. The function checks to see if it is the first time the user has grabbed that item and handles the logic for informing the parent of the trial item’s removal. We initially tried putting in the attachment logic into starNearGrab, but because the controller methods temporarily change an object’s parent to the holder, we moved that logic into releaseGrab.

releaseGrab: This is where the bulk of the try-on logic lives for a wearable, and is fired when a user lets go of the item. The function checks what hand a user has picked up the item with, and excludes that arm or hand from being automatically attached to. The behavior that executes is determined by whether or not the item is already attached. In the event that the item was previously attached to a joint, it confirms that the item is still within range of the joint (and if not, detaches the item from its parent). The function then cycles through the possible joints that the item can be attached to, and checks for the position of the joint using the MyAvatar API with MyAvatar.getJointPosition to identify if the item is within range of a suitable joint (and if so, attaches it to the new parent). Because this is running in a client entity script, it works for all users within the space to enable them to adjust each others worn items.

The Store

Within the store, there are several other scripted components that facilitate the try-on process. A number of mirrors, with trigger Zone Entities around them, turn on when a user approaches and utilize the functionality of a secondary camera to reflect back the to the user. These mirrors generate an overlay, which is a client-only item that renders just for the person within the space, and allow users to see in real-time how their items look on their avatars. Auxiliary shopping aids exist in the store as well, including recycle bins for individual objects and instant cleanup zones that remove all non-purchased attachments. Store security also exists — if you try to leave without paying for an item, it will vanish upon exiting the store .

The checkout process makes use of the new commerce functionality built within High Fidelity to allow items to be purchased and certified by the High Fidelity blockchain. Users set up wallets and purchase their desired wearables using HFC, and these transactions are written to the blockchain to generate a unique certificateID that validates the item as a genuine, purchased copy of the item. Users can inspect each other’s items to ensure that they’re valid, and domain owners will be able to grant specific, granular permissions for users based on the validity of items.

To implement the checkout process to make it easy to use in VR, we designed the user flow to feel familiar, yet delightful. Upon entering the checkout zones, overlay copies of all of the unpurchased items on your avatar appear in front of you. These overlays can be grabbed and swiped across a scanner, which brings up the corresponding marketplace page when it detects that an overlay for an item is in range. The scripts that power the checkout zone store the physical properties of the trial items, so when you purchase and wear the genuine copy of your new wearable, the adjustments you’ve made immediately apply to the item that you now have as a part of your avatar. 

Looking Forward

As we continue to iterate on the requirements through the avatar shopping experience, we’ll be running user tests and gathering feedback about the overall process of shopping socially within virtual reality and explore different design strategies as we look ahead to the full rollout of the High Fidelity blockchain and HFC. We’ve seen endless creativity and amusement from different configurations of wearable items and we’re excited to share what comes next. Stay tuned for more!

(PS: Are you a 3D artist with a style you’d like to get into your own virtual shelves? We’re hiring content creators!)

Liv Erickson is the engineering team lead for the content and prototyping team at High Fidelity. You can find the source code behind the avatar store on GitHub here and for the High Fidelity platform here.