Innovative avatars conversations in virtual reality

Learn how we talk to virtual avatars at XpertVR

If you have played any video game with characters you are likely familiar with at least one type of virtual avatar conversation. The most common being a conversation with another person. In this situation you have an avatar that represents who you are and so does the other person so you either communicate through voice, text chat or sometimes emoji’s. In professional training environments these avatars are usually paid actors or co-workers that are adding to the realism of your virtual training. The second type of avatar conversation are based off of a conversation tree. In these conversation you select what you want to say from options presented to you (i.e. “Hello”, “Don’t talk to me”, “How is your day going”, etc) and then the conversation follows the path or branch you select. The third type of avatar conversations are AI based. In these conversation an Artificial Intelligence is able to understand what you say/type and gives a response based off of what it has learned is likely to be the right answer.

The problem with virtual conversations

To recap the three avatar communication options are:

Person to Person
Conversation Tree
AI Based

Unfortunately all of three of these options are less than perfect. Person to person is great for roleplaying games but in a learning environment it can add co-worker biases and get’s expensive quickly.

Conversation tree’s are time consuming to design and implement. Plus, no matter how much work you put in selecting options will always feel limiting.

Lastly, AI based sounds great but the technology is not currently ready to have full on conversations. Instead different AI or chatbots specialize in specific areas so they can answer specific types of questions.

XpertVR’s solution

While we wait for the future where fully Artificial Intelligent avatars exist, sound natural and communicate like a regular person, XpertVR decided to build a solution for today. What we call it is the Conversation Matrix Tool (CMT). To put it simply CMT is a conversation tree creation tool but with some exciting functionality and whole lot of reusability to build on.

At the moment CMT is an internal tool we have built for Unity Engine so that our team can easily create avatar conversations in virtual reality. And by easy we mean so simple that someone with no Unity skills can jump in and start making conversations from the get go. But that’s not all, because we built CMT on Unity we are able to automatically attach it different 3D avatars, events, character animations and more.

This means instead of just having a character respond to you, it could respond to something you do during your training. As an example let’s pretend you’re a police officer in one of our police training simulations. In the simulation you need to investigate a crime scene and then question witnesses. With the CMT if you collect the right evidence the CMT will unlock new options for you to choose, plus possibly change the witnesses reactions, animations and responses. Not to mention you can swap out different avatars (currently using Ready Player Me avatars) with the click of a button, while transferring facial animations and other information.

Having a dynamic conversation tree like this, that is directly tied into a greater simulation means that organizations looking at VR training options can now mix their hard and soft skills training so that it feels more natural compared to the real world.

Next Steps

We have working on our Conversation Matrix Tool for almost a year now but we know there is so much further we can take it. In the short-term we are integrating saving functionality so that users can save where they were in a conversation. Plus we are adding data collection features and improving animations. The data collection allows teachers to help give direction to students in class and helps students learn by themselves while practicing at home. As for the animations, this is an area we will always be improving because it leads to higher realism and sense of immersion, especially when it comes to facial animations.

In the long-term we are looking into how we can integrate AI into CMT, possibly starting with Amazon Lex or something similar. That being said we won’t stop there and are always looking for suggestions for our partners and anyone with new ideas!

Tagged avatar conversation virtual reality vr/ar